Neural invasion prediction system and method based on hybrid model, and electronic equipment

Abstract

The invention provides a nerve invasion prediction system and method based on a hybrid model, and electronic equipment. The method comprises the following steps: acquiring initial medical information; preprocessing the initial medical information to obtain target medical information; based on the target medical information, acquiring deep learning image features through multi-scale convolution and an attention mechanism of a deep learning model, investigating a tumor region and a peritumor region through a radiomics model, and acquiring radiomics image features; carrying out feature splicing on the radiomics image features and the deep learning image features to obtain classification model input information; and inputting the classification model input information into the classification network to obtain a category prediction result of the initial medical information. According to the feature extraction method combining deep learning and radiomics, the features of the tumor can be captured from different angles, the accuracy of nerve infringement prediction is improved, and the precision requirement of nerve infringement prediction of the prostate part is met.

Description

Technical Field

[0001] This application relates to the technical field of medical image processing, and particularly to a nerve invasion prediction system, method, and electronic device based on a hybrid model. Background Art

[0002] The prostate is an important organ of the male reproductive system and is also a site prone to cancer. Prostate cancer is the third most common cancer globally, and nerve invasion is one of the potential metastasis pathways of the cancer. Confirming nerve invasion helps to evaluate the invasiveness of the disease and the complexity of preoperative surgery. Clinically, if nerve invasion can be clearly identified before surgery, doctors can more precisely handle nerve-related tissues when removing tumors, thereby improving the thoroughness of the surgery and reducing the risk of recurrence.

[0003] In related technologies, an auxiliary system can be used to classify prostate cancer images, making the doctor's diagnosis process more objective. Taking a prostate cancer image classification system disclosed in Application No. CN202210175854.8 as an example, to avoid subjective judgments by doctors based on a single indicator or excessive reliance on personal experience, various parameters in the multi-parameter MRI image are extracted through feature extraction, feature fusion is performed according to the attention mechanism to obtain a fused feature map, classification is performed based on the feature map to obtain the MRI image category, for doctors to work based on the classified images, saving time costs, enhancing the efficiency of image classification, improving the accuracy of image classification by adopting the attention mechanism, and improving the efficiency of feature extraction by performing segmentation processing on the image. Although it uses multi-parameter MRI data, it fails to fully fuse the multi-scale, multi-level, and multi-region features of multi-modal data and cannot meet the higher-precision nerve invasion prediction requirements for the prostate.

[0004] Based on this, this application provides a nerve invasion prediction system, method, and electronic device based on a hybrid model. Summary of the Invention

[0005] Aiming at the problem in the related technology that the precision requirements for nerve invasion prediction of the prostate cannot be met, this application provides a nerve invasion prediction system, method, and electronic device based on a hybrid model.

[0006] The objectives of this application are achieved by the following technical solutions: In the first aspect, this application provides a nerve invasion prediction system based on a hybrid model, and the system includes: An information acquisition module, which is used to acquire initial medical information. The initial medical information includes medical images and data labels. The medical images are multi-parametric MRI images obtained by prostate magnetic resonance scanning of an examinee, and the data labels are used to indicate the tumor region of interest in the medical images. A feature extraction module, which is used to obtain deep learning image features through multi-scale convolution and attention mechanism of a deep learning model based on the initial medical information, and to examine the tumor region and the peritumoral region through a radiomics model and obtain radiomics image features. A result acquisition module, which is used to splice the deep learning image features and the radiomics image features to obtain classification model input information; and input the classification model input information into a classification network to obtain a class prediction result of the initial medical information.

[0007] In a second aspect, the present application further provides a method for predicting nerve invasion based on a hybrid model, which is implemented based on the system for predicting nerve invasion based on a hybrid model described in the first aspect. The method includes the steps of: S101, acquiring initial medical information, where the initial medical information includes medical images and data labels. The medical images are multi-parametric MRI images obtained by prostate magnetic resonance scanning of an examinee, and the data labels are used to indicate the tumor region of interest in the medical images. S102, based on the initial medical information, obtaining deep learning image features through multi-scale convolution and attention mechanism of a deep learning model, and examining the tumor region and the peritumoral region through a radiomics model and obtaining radiomics image features. S103, splicing the deep learning image features and the radiomics image features to obtain classification model input information; and inputting the classification model input information into a classification network to obtain a class prediction result of the initial medical information.

[0008] In a third aspect, the present application further provides an electronic device, including one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors to perform the method described in the second aspect.

[0009] Combined with the above technical solutions and the technical problems to be solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows: A nerve invasion prediction system, method and electronic device based on a hybrid model are provided. The use of multi-scale convolution and attention mechanism enables the model to capture richer image features including local details and global context information; the application of the radiomics model to analyze the tumor region and its surrounding regions (peritumoral regions) enables the utilization of high-throughput features (such as morphology, texture, gray-scale distribution, etc.) extracted from medical images, providing more information for the analysis of the biological characteristics of tumors. Among them, the deep learning feature extraction part adopts a multi-level graph convolutional network based on multi-scale attention, which focuses on extracting multi-level information of multi-modal images. The omics part adopts region-based feature extraction and screening, which can extract multi-region features. Combining the improved deep learning and radiomics can fully integrate high-throughput semantic features of multi-modal, multi-scale, multi-level and multi-region, realize the complementary advantages of the two, capture the features of tumors from different angles, improve the accuracy of nerve invasion prediction, and meet the accuracy requirements for nerve invasion prediction of the prostate site. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present application will be further described below with reference to the drawings and embodiments.

[0011] Figure 1 It is a schematic flowchart of a nerve invasion prediction method provided by an embodiment of the present application.

[0012] Figure 2 It is a schematic flowchart of a hybrid model based on radiomics and deep learning provided by an embodiment of the present application.

[0013] Figure 3 It is a schematic diagram of a deep learning model provided by an embodiment of the present application.

[0014] Figure 4 It is a schematic diagram of a feature extraction module of a deep learning model provided by an embodiment of the present application.

[0015] Figure 5 It is a schematic diagram of a radiomics model provided by an embodiment of the present application.

[0016] Figure 6 It is a schematic diagram of a feature screening module of a radiomics model provided by an embodiment of the present application.

[0017] Figure 7 It is a schematic diagram of modules of a nerve invasion prediction system provided by an embodiment of the present application.

[0018] Figure 8 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] Next, in combination with the accompanying drawings and specific embodiments, the present application will be further described. It should be noted that, on the premise of no conflict, the following-described embodiments or technical features can be arbitrarily combined to form new embodiments. The following will illustrate the implementation procedures of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation procedures. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, rather than for limiting the protection scope of the present application.

[0020] The following briefly explains the technical field and related terms of the embodiments of the present application to facilitate the understanding of those skilled in the art.

[0021] Deep learning is a branch of machine learning. Its core idea is to simulate the working mode of human brain neurons by constructing a multi-layer neural network model, enabling the computer to autonomously learn and extract high-level features from data.

[0022] Radiomics is an auxiliary diagnosis technology that uses automated algorithms to extract and analyze a large amount of information in medical images in a high-throughput manner, and combines statistical analysis and data mining technologies to assist in disease diagnosis, classification, and prognosis assessment.

[0023] The accuracy of nerve invasion prediction in related technologies is insufficient and cannot meet the accuracy requirements of the prostate region. On the one hand, it is because the feature extraction method is single and cannot capture multi-scale and multi-level image features; on the other hand, it is because the analysis of the tumor region and its surrounding regions is insufficient and cannot make full use of the high-throughput features in medical images; on the other hand, it is because the processing ability of multi-modal data is limited and cannot achieve feature fusion of multi-modal, multi-scale, multi-level, and multi-region.

[0024] The present application provides a nerve invasion prediction system, method and electronic device based on a hybrid model, which adopts multi-scale convolution and an attention mechanism. Through multi-scale convolution, features of different scales can be captured, and the attention mechanism helps the model focus on more important features, thereby improving the recognition ability and accuracy of the model. By analyzing the tumor region and its surrounding regions (peritumoral regions) through the radiomics model, high-throughput features such as morphology, texture, and gray-scale distribution are extracted from medical images, making full use of the multi-dimensional information in medical images and providing more basis for the analysis of the biological characteristics of tumors. By combining the improved deep learning with radiomics, high-throughput semantic features of multi-modal, multi-scale, multi-level and multi-region are fully integrated to achieve the complementary advantages of the two, capture the features of tumors from different angles, and improve the accuracy of nerve invasion prediction through the application of the hybrid model. Among them, the hybrid model refers to a model that combines two or more different models, technologies or methods to utilize their respective advantages and make up for the deficiencies of a single method, thereby improving the overall performance or prediction accuracy. In the present application, the hybrid model specifically refers to the combination of the statistical analysis of the deep learning model and the radiomics model and the classification model for prediction. The method will be described first below, and then the system and others will be described.

[0025] Method embodiment.

[0026] See Figure 1 , Figure 1 is a schematic flowchart of a nerve invasion prediction method provided by an embodiment of the present application.

[0027] An embodiment of the present application provides a nerve invasion prediction method, and the method includes: S101, obtaining initial medical information, where the initial medical information includes a medical image and a data label, the medical image is a multi-parametric MRI image obtained by prostate magnetic resonance scanning of an examination object, and the data label is used to indicate the tumor region of interest in the medical image.

[0028] First, initial medical information is obtained. The initial medical information includes a medical image of the prostate of an examination object (such as a patient) and the corresponding data label. The data label can be marked by a professional doctor or imaging expert and is used to clearly indicate the region of interest (ROI, that is, the original region of interest mentioned below) of the tumor in the image, that is, the specific location of the tumor. Since there may be multiple lesions, the corresponding regions of interest may also be multiple.

[0029] S102, based on the initial medical information, obtaining deep learning image features through multi-scale convolution and an attention mechanism of a deep learning model, and examining the tumor region and the peritumoral region through a radiomics model to obtain radiomics image features.

[0030] Using the multi-scale convolution and attention mechanism of a deep learning model to perform deep feature extraction on medical images. The multi-scale convolution can capture the detailed information of different scales in the image, while the attention mechanism can focus on the key regions in the image, thereby extracting more representative deep learning image features.

[0031] At the same time, an imaging genomics model is used to analyze the tumor region and its surrounding regions (peritumoral regions), and imaging genomics image features reflecting the characteristics of the tumor such as morphology, texture, and intensity are extracted. It can be considered that the imaging genomics image features can more comprehensively describe the tumor and its relationship with the surrounding tissues.

[0032] Furthermore, referring to Figure 6 , considering that the standard prostate region is similar to axial symmetry, in the method of omics feature screening, the imaging genomics image features of the original ROI region (original region of interest) can be compared with the imaging genomics image features of the mirrored ROI (mirrored region of interest) after being inverted along the axis of symmetry, so as to achieve personalized feature screening for the prostate region.

[0033] S103, splice the deep learning image features and the imaging genomics image features to obtain the input information of the classification model; input the input information of the classification model into the classification network to obtain the category prediction result of the initial medical information.

[0034] Splice the deep learning image features and the imaging genomics image features to form the input information of the classification model containing rich information. It can be considered that feature fusion can make full use of the advantages of the two models and improve the accuracy of prediction. Input the spliced features into the classification network, and the trained model classifies the input information to finally obtain the category prediction result of the initial medical information. The category prediction result is used to generate a report containing the classification result to assist users (doctors) in making diagnosis and treatment decisions. In particular, for the methods proposed in the embodiments of the present application, the category prediction result is only used as an intermediate result, that is, it is not directly aimed at obtaining a disease diagnosis result or a health status.

[0035] Compared with the related art where only a deep learning model or a radiomics model is used for prediction, mainly considering the simplicity and ease of implementation of the diagnostic model, the technical solution provided in this embodiment enables the model to capture richer image features including local details and global context information through the use of multi-scale convolution and attention mechanism; the application of the radiomics model to analyze the tumor region and its surrounding region (peritumoral region) enables the utilization of high-throughput features (such as morphology, texture, gray-scale distribution, etc.) extracted from medical images, providing more information for the analysis of the biological characteristics of tumors. By combining the feature extraction methods of deep learning and radiomics, the characteristics of tumors can be captured from different perspectives, improving the accuracy of predicting nerve invasion and meeting the precision requirements for predicting nerve invasion in the prostate region.

[0036] That is to say, the technical solution provided in this application is not simply splicing deep learning and radiomics technologies. Instead, aiming at the need of predicting nerve invasion in prostate cancer, by extracting multi-level information from multi-modal images, the application of multi-scale convolution and attention mechanism enables the deep learning model to better adapt to tumor images of different scales and morphologies, enhancing the robustness of the hybrid model. At the same time, to examine the tumor region and the peritumoral region and obtain radiomics image features, feature extraction and screening are carried out in sub-regions (tumor region and peritumoral region), and multi-region features can be extracted. The specific implementation will be described below.

[0037] In specific applications, this technical solution improves the accuracy of the prediction results, can provide more reliable reference for clinicians, helps to optimize the surgical plan, reduce unnecessary surgical trauma, and improve the quality of life and prognosis of patients.

[0038] The medical image in this embodiment is a multi-parametric MRI image, and the multi-parametric MRI image includes a T2WI sequence and an ADC map. The data labels include the tumor region of interest labels of the T2WI sequence and the ADC map respectively; the tumor region of interest labels indicate the tumor regions of interest (ROIs) of the T2WI sequence and the ADC map respectively.

[0039] Taking the method and device for training a diagnostic model disclosed in the application number CN202410265980.1 as an example, a deep learning method is used to segment blood vessels and finally realize the diagnosis of nerve invasion, and its technical solution relies on CT images. However, the technical solution of this embodiment is applied to the prostate region, and the medical image used is a multi-parametric MRI image. Considering that the multi-parametric MRI image of the prostate involves multiple modalities and the image information is complex, the technical solution provided in this application can fully extract the features associated with nerve invasion in the image, thereby meeting the precision requirements for predicting nerve invasion in the prostate region.

[0040] In some embodiments, the obtaining of the deep learning image features through multi-scale convolution and attention mechanism of the deep learning model based on the initial medical information, and the investigation of the tumor region and the peritumoral region through the radiomics model to obtain radiomics image features (S102) includes: Preprocess the initial medical information to obtain target medical information; Input the target medical information into the deep learning model to obtain deep learning image features; it is also used to input the target medical information into the radiomics model to obtain radiomics image features.

[0041] Among them, the preprocessing includes: Resample the multi-parameter MRI images and their corresponding data labels to make the voxel sizes of all images consistent, and complete the normalization of the voxels; Perform voxel value homogenization processing on the resampled MRI images, and adjust the voxel values of each layer of images to have a mean of 0 and a standard deviation of 1 through calculation; Based on the preset size requirements of the model input, adjust the resampled and voxel value homogenized MRI images and their image layers of the region of interest according to the preset two-dimensional size to obtain standardized target medical information.

[0042] The preprocessing includes voxel normalization, voxel value homogenization, and size adjustment. Voxel normalization is used to resample the MRI images and their data labels to make the voxel sizes of all images consistent, so as to eliminate size differences caused by different scanning devices or parameters. Voxel value homogenization is used to perform voxel value homogenization processing on the resampled MRI images, and adjust the voxel values of each layer of images through calculation to make their mean 0 and standard deviation 1, so as to eliminate intensity differences between different images. Size adjustment is used to adjust the processed MRI images and their image layers of the region of interest to the corresponding preset two-dimensional size according to the preset size requirements of the model input, so as to standardize the target medical information.

[0043] The advantage of this embodiment is that the preprocessing step enables the hybrid model to adapt to MRI images with different resolutions and parameter settings, enhancing the adaptability and generalization ability of the model. Specifically, through voxel normalization and voxel value homogenization processing, it is ensured that the obtained MRI images are consistent before and after processing, providing a standardized basis for subsequent feature extraction. First, the initial medical information is preprocessed, and then the graph network features and key regions in the image are captured through multi-scale convolution and attention mechanism. The radiomics model provides the morphological and texture features of the tumor. The combination of the two enhances the model's ability to express prostate cancer features. Feature splicing integrates the advantages of deep learning and radiomics, improving the accuracy and robustness of nerve invasion prediction.

[0044] It should be noted that the deep learning model can be trained through a training set. As an example, the training set includes a plurality of first training data. The first training data includes MRI images of prostate cancer from different patients, different scanning devices, and different scanning parameters, and serves as medical sample image data; it also includes sample annotation information corresponding to the medical sample image data. It can be considered that the sample annotation information of the image data of each medical image includes detailed annotations on tumor regions, nerve invasion, and surrounding normal tissues, etc.

[0045] In some embodiments, the inputting the target medical information into the deep learning model to obtain deep learning image features includes: Inputting the target medical information into the trained deep learning model, and processing the original multi-parametric MRI image and its region of interest through the feature extraction network of the deep learning model to extract the graph network features of the image; further processing and integrating the graph network features by using a graph convolutional network to obtain deep learning image features.

[0046] It can be understood that the original MRI image and its ROI are processed by using the feature extraction network in the deep learning model, and the deep features of the image are automatically learned and extracted through the convolutional layer of the deep learning model. The features extracted by the deep learning model are input into the graph convolutional network. The graph convolutional network is designed to process graph-structured data and can further integrate and process the image features obtained from the deep learning model, strengthening the correlation and hierarchical structure information between the features. Through the processing of the graph convolutional network, the finally obtained deep learning image features are a rich representation that fuses multi-scale, multi-modal, and multi-region information, which not only contains local image details but also contains the relationships between image regions and overall context information.

[0047] The advantage of this embodiment is that the features automatically extracted by the deep learning model are combined with the structured features processed by the graph convolutional network, enhancing the model's ability to express medical image features.

[0048] In some embodiments, the inputting the target medical information into the radiomics model to obtain radiomics image features includes: Performing radiomics feature extraction and radiomics feature screening on the tumor region and the peritumoral region of the image of the target medical information; obtaining radiomics image features based on the finally screened radiomics features of the two; the peritumoral region is an annular region within a preset distance around the tumor region, and the tumor region and the peritumoral region are used as a combined region; the value range of the preset distance is not less than 4 mm and not more than 6 mm. The preset distance is preferably 5 mm.

[0049] In a specific application, the process of obtaining radiomics image features includes inputting target medical information into a radiomics model, performing feature extraction and feature screening, and then inputting the results into a classifier to output the results (radiomics image features).

[0050] In a specific application, radiomics image features can be obtained based on the omics features finally selected from both of them. It can be that the omics features finally selected from both of them are concatenated as radiomics image features; it can also be that based on the omics features finally selected from both of them, the omics features of the region of interest and the omics features of the peritumoral region are obtained and combined as radiomics image features, and the specific implementation manner will be described below.

[0051] It can be considered that when processing target medical information (i.e., preprocessed MRI images), through a delicate feature extraction and screening process, key information is provided for subsequent prediction. The preprocessed MRI images are input into the radiomics model as target medical information. Since the target medical information has undergone preprocessing steps such as resampling, voxel value normalization, and size adjustment, the consistency and standardization of the data are ensured. In the input MRI images, the tumor region and the peritumoral region are clearly demarcated. The tumor region is the known region of interest (ROI) of the tumor, while the peritumoral region is an annular region preset at a certain distance around the tumor region to examine the potential impact of the tumor on the surrounding tissues.

[0052] Omics feature extraction is performed on the tumor region and the peritumoral region respectively. Morphological features (such as size, shape, edge, etc.) and texture features (such as gray-level co-occurrence matrix, gray-level run-length matrix, etc.) are extracted from the images, which together constitute a comprehensive description of the tumor and its surrounding environment. Morphological features are used to correlate with the physical properties of the tumor and can help identify the invasiveness of the tumor; texture features can describe the pattern of gray-level changes in the image and reflect the microscopic structure and tissue characteristics of the tumor.

[0053] Then, the extracted omics features are screened to remove redundant and irrelevant features and retain the features that make significant contributions to the prediction of nerve invasion. The omics features finally selected from the tumor region and the peritumoral region are concatenated.

[0054] It should be noted that related solutions often only focus on the characteristics of the tumor region itself, while ignoring the potential impact of the tumor on the surrounding tissues. For the application scenario of prostate cancer, this technical solution takes into account that prostate cancer is prone to occur in the peripheral region of the prostate. In this region, the cell division rate is relatively fast, abnormal cell hyperplasia is likely to occur, and the structure is complex, with many branched ducts and lymph nodes, providing opportunities for cancer cell metastasis. At the same time, the growth and invasion of prostate cancer are not only affected by the characteristics of the tumor itself, but also significantly affected by the surrounding microenvironment (such as blood vessels, lymphatic vessels, nerves, and interstitial tissues). As the transition zone between the tumor and normal tissues, the biological characteristics of the peritumoral region can reflect the interaction between the tumor and the surrounding tissues. Therefore, by analyzing the tumor region and the peritumoral region simultaneously, a more comprehensive description of the tumor and its microenvironment is provided, which helps to more accurately understand the biological behaviors such as tumor growth, invasion, and metastasis.

[0055] The advantage of this embodiment is that it provides a process for extracting and screening omics features in sub-regions (regions of interest and peritumoral regions), and multi-region features can be extracted. By comprehensively considering the omics features of the tumor region and the peritumoral region, the radiomics model can more comprehensively capture the biological information of the tumor and its surrounding environment, thereby improving the accuracy of predicting nerve invasion.

[0056] In some embodiments, the feature extraction network includes a pyramid convolutional network, a residual connection module, and a channel attention module; The pyramid convolutional network includes 3D convolutional kernels of 9×9×9, 5×5×5, and 3×3×3. The pyramid convolutional network is used to extract multi-scale information from the feature map output by the CNN layer of the deep learning model, and then perform pipelined layer-by-layer fusion on the extracted multi-scale information to obtain intermediate features; The residual connection module includes two different types of anisotropic convolutions, which are used to capture the inter-layer relationship and the intra-layer detail information, and are fused with the intermediate features by means of vector dot addition; The channel attention module is used to receive the feature map after vector dot addition and abstract the global information at the feature channel level, and finally outputs the graph network features with a dimension of 2048.

[0057] It can be considered that the pyramid convolution network uses three-dimensional convolution kernels of different sizes to extract multi-scale information from the feature maps output by the CNN layer of the deep learning model. The extracted multi-scale features are integrated through a pipelined layer-by-layer fusion process to form rich intermediate feature representations. The layer-by-layer fusion helps the model understand the image content at different levels and enhances the feature expression ability. The residual connection module contains two different types of anisotropic convolutions, and the anisotropic convolutions focus on capturing inter-layer relationships and intra-layer detail information. Then, through the way of vector dot addition, the residual connection module fuses the obtained information with the intermediate features, which helps the model learn more complex feature representations.

[0058] The advantage of this embodiment is that, because this embodiment is a multi-level graph convolution network based on multi-scale attention and focuses on extracting multi-level information of multi-modal images, more rich and discriminative features can be learned through multi-scale convolution and residual connection, improving the feature expression ability. The features that fuse inter-layer relationships and intra-layer details enable the model to perform better when processing complex images of the prostate, thus improving the overall performance.

[0059] In some embodiments, the graph convolution network is used to obtain graph network features from the feature extraction network, and both the tumor region of interest in the medical image of the target medical information and the complete image layer containing the tumor region of interest are defined as nodes of the graph, obtaining a node set; For each group of adjacent nodes in the node set, the edge distance between the adjacent nodes is calculated through a distance formula; When the edge distance is less than or equal to a predefined threshold, it is considered that there is a valid edge between the adjacent nodes corresponding to the edge distance; Among them, the predefined threshold is the sum of the mean and standard deviation of the defined node distances, and the distance formula is as follows:

[0060] d(i,j) is the edge distance between adjacent nodes i and j, V ik is the value of node i in the k-th feature dimension, V jk is the value of node j in the k-th feature dimension, represents the L1 norm, which can be understood that the distance between nodes is given by the L1 norm, reflecting the overall difference of nodes in the feature space.

[0061] For each set of adjacent nodes in the node set, the graph convolutional network calculates the edge distance between them through a distance formula. The calculated edge distance is compared with a predefined threshold. If the edge distance is less than or equal to the threshold, the network considers that there is a valid edge between the two nodes, indicating that they are sufficiently similar or relevant in the feature space. The advantage of this embodiment is that by calculating the edge distance between nodes and determining valid edges, the graph convolutional network can better capture and utilize the correlation between different regions in the image.

[0062] In some embodiments, The methods for performing omics feature extraction and omics feature screening on the tumor region and the peritumoral region of the image of the target medical information include: For the image in the target medical information, the tumor region and the peritumoral region are augmented through Log transformation and eight-direction wavelet transformation; Omics features are extracted from the augmented image according to the preset feature quantity types to obtain an overall feature set; the preset feature quantity types include shape features, first-order statistical features, gray-level co-occurrence matrix features, gray-level dependence matrix features, gray-level run-length matrix features, gray-level size zone matrix features, and neighborhood gray-level difference matrix features; LASSO regression is used to screen the features in the overall feature set to obtain the omics features of the tumor region of interest and the omics features of the peritumoral region; Taking the tumor region of interest of the medical image as the original region of interest, the overall features of the original region of interest are compared with the overall features of the mirror region of interest using the mirror comparison method to obtain a set of omics features with statistical differences, and the union is taken with the omics features of the tumor region of interest obtained by LASSO regression to obtain the omics features of the region of interest; The omics features of the region of interest and the omics features of the peritumoral region are combined and output as the radiomics image features.

[0063] It can be considered that during the image augmentation process, for the MRI image of prostate cancer, the Log transformation is performed on the tumor region and the peritumoral region respectively. The Log transformation can enhance the details of the regions with low contrast in the image, making the difference between the tumor region and the surrounding tissues more obvious, which is helpful for subsequent feature extraction. On the basis of the Log transformation, the eight-direction wavelet transformation is performed on the image. The wavelet transformation can decompose the image from multiple directions and scales, capturing the high-frequency and low-frequency information in the image, thereby further enriching the feature space of the image.

[0064] During the omics feature extraction process, the preset feature quantity types include shape features, first-order statistical features, gray-level co-occurrence matrix features, gray-level dependence matrix features, gray-level run-length matrix features, gray-level size zone matrix features, and neighborhood gray-level difference matrix features, etc., which can comprehensively describe information such as the texture, structure, and intensity distribution of the image.

[0065] During the omics feature screening process, LASSO regression is used to screen the overall feature set. LASSO regression can introduce an L1 regularization term to compress the coefficients of irrelevant or redundant features to zero while retaining important features, thereby achieving feature selection and dimensionality reduction.

[0066] Taking the tumor region of interest in the medical image as the original region of interest, the overall features of the original region of interest and the mirrored region of interest are compared using the mirror comparison method to obtain a set of omics features with statistical differences. The union of the set of omics features with statistical differences and the omics features of the region of interest obtained by LASSO regression is taken to obtain the omics features of the region of interest. The omics features of the region of interest and the omics features of the peritumoral region are spliced to obtain the final radiomics image features.

[0067] The advantages of this embodiment are that by performing image augmentation through Log transformation and wavelet transformation, and extracting features of multiple feature quantity types, more details and tiny lesions in the image can be captured, thereby improving the accuracy of diagnosis. The screening effect of LASSO regression can remove irrelevant or redundant features and retain the features most valuable for the diagnosis of prostate cancer, thereby enhancing the representation ability of the features, ensuring that the selected features are highly relevant to the prediction target, and thus improving the prediction performance of the model. The omics feature screening targets the shape features of the prostate region and uses the mirror comparison strategy to achieve personalized feature screening, removing redundant and irrelevant features and reducing noise interference, making the model more focused on the features that have a substantial impact on the prediction results.

[0068] In some embodiments, the feature combination is to splice the omics features of the region of interest and the omics features of the peritumoral region; the initial medical information further includes the original gland label, and using the mirror comparison method to compare the overall features of the original region of interest and the mirrored region of interest to obtain a set of omics features with statistical differences, including: Obtaining the boundary information of the prostate and the central axis of the prostate in the medical image; Quantifying the degree of axial symmetry by calculating the Dice coefficient between the original gland label and the mirrored gland label along the central axis; If the dice coefficient is greater than 0.8, it is considered that the prostate region meets the requirement of axial symmetry; the label of the tumor region of interest is mirror-flipped along the central axis of the prostate to generate a mirrored tumor region of interest label, and the omics image features of the mirrored region of interest are extracted; The omics image features of the mirrored region of interest and the omics features of the original region of interest are compared and screened by mirroring using statistical methods such as T-test, U-test, Cohen's d effect size analysis, and mean comparison. The intersection of the features selected in each case is used as the set of omics features with statistical differences.

[0069] It can be considered that by obtaining the prostate boundary information and the central axis, that is, by extracting the prostate boundary information from medical images, the scope and shape of the prostate are clarified, and the central axis of the prostate is determined, providing a basis for subsequent mirror operations and symmetry analysis. Then, the dice coefficient between the original gland label and the gland label mirrored along the central axis is calculated. The dice coefficient is used to measure the similarity between the two label regions. If the coefficient is greater than 0.8, it indicates that the prostate region has high axial symmetry on both sides of the central axis and meets the biological symmetry requirements. In this case, the label of the tumor region of interest (ROI) is mirror-flipped along the central axis of the prostate to obtain the mirrored tumor ROI label. If the coefficient is not greater than 0.8, the subsequent steps are stopped. The omics features of the mirrored tumor ROI are extracted, and features such as its morphology, texture, and gray-scale distribution are obtained. The specific method is the same as the extraction of the omics features of the original tumor ROI, and will not be elaborated in this application.

[0070] During the process of mirror comparison and screening, statistical methods such as T-test, U-test, Cohen's d effect size analysis, and mean comparison are used in combination to compare the omics features of the original tumor ROI and the mirrored tumor ROI. Finally, the features with statistical differences in the mirrored region are selected, that is, the features that show significant differences in the symmetry background. The intersection of the features selected in each case (i.e., the significant feature set) is used as the set of omics features with statistical differences.

[0071] The advantage of this embodiment is that by comparing features in the mirrored region, features with significant differences in the symmetry background can be identified. These features are more likely to be related to the biological characteristics of the tumor, thereby improving the accuracy of feature screening. The selected significant feature set can more comprehensively describe the features of the tumor and its surrounding environment, providing a high-quality feature basis for subsequent prediction models, and helping to improve the performance of the prediction model and the accuracy of the prediction results. Multiple statistical methods are used for feature screening, and the intersection of the screening results is used as the set of omics features with statistical differences, ensuring the robustness of feature screening and reducing the risk of misjudgment caused by the limitations of a single method.

[0072] In some embodiments, the classification network is a convolutional neural network model, and inputting the input information of the classification model into the classification network to obtain the class prediction result of the initial medical information includes: Using the convolutional kernels of the convolutional neural network model to extract the temporal features in the input information of the classification model, obtaining a feature map and expanding it; classifying the expanded feature map through two fully connected layers and outputting the prediction result of nerve invasion.

[0073] It can be considered that the convolutional kernels of the convolutional neural network model are used to extract the temporal features in the input information. The convolutional kernels slide on the image and perform convolutional operations on local regions to extract the local features of the image. After the convolutional operation, a feature map is generated, and the feature map contains the high-level feature representation of the input image. Each feature map corresponds to a specific feature detected by the convolutional kernel. The feature map is expanded into a one-dimensional vector so that it can be input into the fully connected layer, converting the multi-dimensional feature information into a linear sequence to provide input for classification. The expanded feature map is input into two fully connected layers, which will learn the complex relationships between the features and perform classification. Finally, the network outputs the class prediction result of the initial medical information. For example, whether nerve invasion has occurred (which can be represented by 0 and 1).

[0074] As an example, a method for predicting nerve invasion based on a hybrid model is provided. The hybrid model includes a deep learning model, a radiomics model, and a classification model; the method includes: R101, obtaining initial medical information, where the initial medical information includes medical images and data labels. The medical images are obtained by prostate magnetic resonance scanning of the examination object, and the data labels are used to indicate the tumor regions of interest in the medical images; the medical images are multi-parametric MRI images, the multi-parametric MRI images include T2WI sequences and ADC maps, and the data labels include the tumor region of interest labels of the T2WI sequences and the ADC maps respectively.

[0075] In a specific application, the collected prostate multi-parametric MRI images are examined, and the tumor regions of prostate cancer are manually outlined on the T2WI sequences and ADC maps respectively. The regions of interest are extracted, and the regions containing prostate cancer tumors are extracted and saved according to the outlined labels to obtain the initial medical information.

[0076] R102, preprocessing the initial medical information to obtain the target medical information; Input the target medical information into the trained deep learning model, and process the original multi-parameter MRI image and its region of interest through the feature extraction network of the deep learning model to extract the graph network features of the image; further process and integrate the graph network features by using a graph convolutional network to obtain deep learning image features; that is, obtain deep learning image features through the multi-scale convolution and attention mechanism of the deep learning model. Perform omics feature extraction and omics feature screening on the tumor region and the peritumoral region of the image of the target medical information; obtain radiomics image features based on the finally screened omics features of the two; the peritumoral region is an annular region within a preset distance around the tumor region, and the tumor region and the peritumoral region are used as a combined region; the value range of the preset distance is 5 mm; that is, examine the tumor region and the peritumoral region through a radiomics model and obtain radiomics image features.

[0077] The preprocessing includes: Resample the multi-parameter MRI image and its corresponding data label to make the voxel sizes of all images consistent and complete the normalization of the voxels. Perform voxel value homogenization processing on the resampled MRI image, and adjust the voxel values of each layer of the image to have a mean of 0 and a standard deviation of 1 through calculation. Based on the preset size requirements of the model input, adjust the resampled and voxel value homogenized MRI image and its region of interest image layer according to the preset two-dimensional size to obtain the standardized target medical information.

[0078] Among them, the feature extraction network of the deep learning model includes a pyramid convolutional network, a residual connection module, and a channel attention module. The pyramid convolutional network includes 3D convolutional kernels of 9×9×9, 5×5×5, and 3×3×3. The pyramid convolutional network is used to perform multi-scale information extraction on the feature map output by the CNN layer of the deep learning model, and then perform pipeline-style layer-by-layer fusion on the extracted multi-scale information to obtain intermediate features. The residual connection module includes two different types of anisotropic convolutions, which are used to capture inter-layer relationships and intra-layer detail information, and are fused with the intermediate features in a vector dot-adding manner. The channel attention module is used to receive the feature map after vector dot-adding and abstract global information at the feature channel level, and finally outputs graph network features with a dimension of 2048.

[0079] The graph convolutional network of the deep learning model is used to obtain graph network features from the feature extraction network, define both the tumor region of interest in the medical image of the target medical information and the complete image layer containing the tumor region of interest as nodes of the graph, and obtain a node set. For each set of adjacent nodes in the node set, calculate the edge distance between the adjacent nodes through a distance formula; When the edge distance is less than or equal to a predefined threshold, it is considered that there is a valid edge between the adjacent nodes corresponding to the edge distance; Among them, the predefined threshold is the sum of the mean and standard deviation of the defined inter-node distance, and the distance formula is as follows:

[0080] d(i,j) is the edge distance between adjacent nodes i and j, V ik is the value of node i in the k-th feature dimension, V jk is the value of node j in the k-th feature dimension.

[0081] Among them, in the radiomics model, the methods for extracting and screening radiomics features for the tumor region and the peritumoral region of the image of the target medical information include: For the image in the target medical information, the tumor region and the peritumoral region are augmented by Log transformation and wavelet transformation in eight directions; Extract radiomics features from the augmented image according to the preset feature quantity types to obtain an overall feature set; the preset feature quantity types include shape features, first-order statistical features, gray-level co-occurrence matrix features, gray-level dependence matrix features, gray-level run-length matrix features, gray-level size zone matrix features, and neighborhood gray-level difference matrix features; Use LASSO regression to screen the features in the overall feature set to obtain the radiomics features of the tumor region of interest and the peritumoral region radiomics features; Take the tumor region of interest of the medical image as the original region of interest, use the mirror comparison method to compare the overall features of the original region of interest with the mirror region of interest, obtain a set of radiomics features with statistical differences, and take the union with the radiomics features of the tumor region of interest obtained by LASSO regression to obtain the radiomics features of the region of interest; Output the combined radiomics features of the region of interest and the peritumoral region radiomics features as radiomics image features.

[0082] See Figure 2 , Figure 2 is a schematic flowchart of a hybrid model based on radiomics and deep learning provided by an embodiment of the present application. In specific applications, the content of network construction may include: The model inputs are the original MRI and data labels (multi-parametric T2WI sequence and ADC map, and their tumor region of interest labels), which are used as the image inputs for the deep learning model and the radiomics model respectively. In the deep learning model (deep learning feature extraction - graph convolutional network - deep learning image features) part, the model uses multi-scale convolution and attention mechanism to improve the ability to mine MRI features. In the radiomics model (omics feature extraction - omics feature screening - radiomics image features) part, the model examines the tumor region and the peritumoral region respectively, realizes differential omics feature extraction, and enriches the inclusion of omics features. In the feature fusion and classification part, after splicing the deep learning image features and the radiomics model features, the fully connected network in the final classification network of the model includes two fully connected layers, with a corresponding probability score of 0.33 for invasion and a corresponding probability score of 0.79 for non-invasion, and the output is 1 (non-invasion).

[0083] See Figure 3 , Figure 3 is a schematic diagram of the deep learning model provided by the embodiment of the present application. Among them, the method of deep learning feature extraction based on graph convolution can include: The graph convolutional network connects and integrates data from different modalities and types (original MRI and original ADC, T2WI region of interest and ADC region of interest) by associating image features, performs deep learning feature extraction to obtain a deep learning image feature group, and then inputs it into the graph convolutional network to obtain deep learning image features for classification output.

[0084] After the input of its multi-parametric MRI image and its corresponding data label image, 2048-dimensional information (deep learning image feature group) is output from the graph network feature through deep learning feature extraction to the graph convolutional network. In the graph convolutional network, first, both the tumor region of interest and the complete image layer containing these regions of interest are defined as the nodes V of the graph. Second, the calculation of the edge distance d between adjacent nodes i and j is given by the difference of the image in the feature space, as shown in the following formula 1:

[0085] In the graph convolutional network, the screening of edges is a key step in constructing the graph structure, which is used to define whether there is an effective connection relationship between nodes. Specifically, calculate the mean value and the standard deviation of the defined node distances. When the distance between adjacent nodes is less than or equal to the predefined threshold , it is considered that there is an effective edge between these two adjacent nodes. Specifically, as shown in the following formula 2:

[0086] The value of Adjacency(i,j) indicates whether nodes i and j are directly connected (there is a valid edge). The role of this screening mechanism is to adjust the structure of the graph according to the similarity in the feature space, only retaining the edges between nodes that are close enough in distance, thereby reducing the sparse noise of the graph and enhancing the efficiency of the model.

[0087] Among them, the feature extraction module of deep learning that combines multi-scale convolution and attention mechanism is as follows: To better construct the graph convolutional network, it is necessary to fully exploit the features of different scales and levels of MRI. The feature extraction network design in the deep learning model includes pyramid convolution, residual connection, and channel attention, and has been trained on the prostate cancer Gleason score risk classification task.

[0088] See Figure 4 , Figure 4 is a schematic diagram of the feature extraction module of the deep learning model provided by the embodiments of the present application. The depth of this network is 6, the input (data) is the original multi-parametric MRI and the region of interest (N × number of Slices), and the output is the image features of 2048 dimensions.

[0089] The sizes of prostate tumors vary significantly, and a single fixed convolution kernel is difficult to comprehensively extract heterogeneous tumor image features. Therefore, multi-scale convolution is used in the deep learning feature network to achieve multi-scale information extraction, including three groups of isotropic three-dimensional convolution kernels with different receptive fields, whose scales are 3, 5, and 9 respectively, and a pipeline-style layer-by-layer fusion is adopted. The process is shown in the following formula 3:

[0090] Among them, is the output value after the input data passes through three groups of isotropic three-dimensional convolution kernels with different receptive fields. is the value of the output of the i-th layer convolution, where i is a positive integer from 1 to 3, corresponding to the three-dimensional convolution kernels with scales of 3, 5, and 9 respectively. During the image data processing, three convolution operations are performed, and the output part of each convolution will be used as one of the inputs for the subsequent convolution operation. By introducing the multi-receptive field convolution mechanism and combining the gradient fusion strategy, it aims to strengthen the accurate extraction and efficient fusion of the features of heterogeneous tumor regions, thereby enhancing the expression ability of the model.

[0091] Considering that MRI data has a three-dimensional structure, both the inter-layer and intra-layer information are crucial. Therefore, the traditional residual connection is improved, and two types of anisotropic convolutions (3×3×1 and 1×1×3) are set up to focus on processing the context between layers and the information within layers. Channel attention abstracts the global information of the feature channels and uses it to weight and guide the overall features. The feature abstraction and guidance processes are shown in the following Formulas 4 and 5 respectively:

[0092]

[0093] Among them, is the output feature of the first layer processed by the channel attention mechanism; is the channel attention function; is the input feature; D is the depth of the feature map, H is the height of the feature map, and W is the width of the feature map; is the feature value at the position of a specific depth d, height h, and width w.

[0094] is the output feature of the third layer processed by the channel attention mechanism; is the non-linear transformation function; is the activation function, g is the gain function used for non-linear transformation of the features, and are the weight data, represents element-wise multiplication.

[0095] First, through the channel attention function performs global average pooling on the input feature map to generate the global statistical features of each channel, so as to capture the correlation between channels; then, through the non-linear transformation generates the channel attention weights to weight and adjust the input features and enhance the response to important channels. This application improves the selectivity and expression ability of the model for features through the channel attention mechanism. Especially when processing heterogeneous lesion regions, it can more accurately extract and fuse key features, thereby improving the model performance and expression ability. Among them, SE-Net (Squeeze-and-Excitation Networks) is a deep learning architecture used to enhance the feature representation ability in convolutional neural networks (CNNs). The deep learning model of this application can include a module integrating the SE-Net architecture. By introducing the channel attention mechanism, it dynamically adjusts the importance of feature channels, thereby enhancing the model's attention ability to key features.

[0096] Among them, the differential feature extraction part based on radiomics is as follows: See Figure 5 , Figure 5 which is a schematic diagram of the radiomics model provided by the embodiments of the present application. Images including the region of interest and the 5-mm peripheral region of the tumor are respectively subjected to omics feature extraction to obtain the overall omics features of the region of interest and the overall omics features of the 5-mm peripheral zone of the tumor. Then, omics feature screening is respectively performed to obtain the omics feature screening data of the region of interest and the omics feature screening data of the 5-mm peripheral zone of the tumor. Then, through joint omics feature determination, the omics features of the region of interest and the omics features of the 5-mm peripheral zone of the tumor are spliced (to obtain MRI omics features), which are output as radiomics image features.

[0097] That is to say, considering that some prostate cancer tumors tend to invade the surrounding nerves and capsules, an additional 5-mm annular region of the peripheral zone of interest is included as a joint region during omics feature extraction to achieve differential MRI feature extraction, and the omics features finally screened from the two (images including the region of interest and the 5-mm peripheral region of the tumor) are spliced as the radiomics image features of MRI.

[0098] In specific applications, first, log5mm and wavelet transforms in eight directions are used to expand the original images. In each category, seven types of feature quantities are included: shape features, first-order statistical features, gray-level co-occurrence matrix (GLCM), gray-level run-length matrix (GLRLM), gray-level size zone matrix (GLSZM), and neighborhood gray-tone difference matrix (NGTDM). Secondly, the overall feature set of LASSO regression is used for screening to obtain the feature items closely related to predicting nerve invasion.

[0099] The initial medical information further includes an original gland label, and the boundary information of the prostate is determined based on the original gland label.

[0100] Among them, the differential feature extraction part based on radiomics is as follows: The radiomics model includes a feature screening module. See Figure 6 , Figure 6Schematic diagram of the feature screening module of the radiomics model provided in this embodiment. The feature screening module aims to screen and optimize the omics features of the region of interest (ROI) and the 5-mm region around the tumor. Through symmetry and position discrimination (calculating the Dice coefficient and the distance to the gland boundary), the original ROI (original region of interest) and the mirrored ROI (mirrored region of interest) are obtained. Then, the omics image features of the original ROI and the mirrored ROI are obtained. After feature comparison, the omics features are compared using four statistical methods (individual screening process) to obtain the screened omics features, and then comprehensive screening is performed to obtain the omics features (omics feature set) screened by the mirrored comparison method.

[0101] Specifically, first, the boundary information (prostate position box) of the prostate is determined through data labels, and the central axis of the prostate (the dashed line segment within the prostate position box) is further extracted. It can be seen that there is a unilateral and isolated tumor region (region of interest). On this basis, to ensure the axial symmetry of the prostate region, a constraint strategy is adopted. The degree of axial symmetry is quantified by calculating the Dice coefficient between the original gland label and the mirrored gland label along the central axis. If the Dice is greater than 0.8, the prostate region can be considered to meet the requirements of axial symmetry.

[0102] Subsequently, the ROI label is mirrored along the central axis of the prostate to generate the mirrored ROI label, and the same strategy is used to extract the omics image features of the mirrored ROI. In the mirrored comparison process of radiomics features, the aim is to identify features with significant differences in the mirrored region. When the results screened out all meet the respective screening conditions of the t-test, u-test, Cohen's d effect size analysis, and mean comparison, the screened (significantly different) results are used as the set of omics features with statistical differences. It can be understood that four common statistical methods, namely the t-test, u-test, Cohen's d effect size analysis, and mean comparison, are adopted. Among them, features with a p-value less than 0.05 in the t-test and u-test are considered to have significant differences in the mirrored region. Features with an effect size d greater than 0.8 in Cohen's d indicate significant differences in the mirrored region. Features with a change rate t of the feature value greater than 2.0 are also considered to have significant differences. To ensure the robustness of the screening results, the intersection of the features screened by the above four methods is used as the final set of omics features with statistical differences, and the union of the omics features obtained by LASSO regression screening is taken to obtain the omics features of the region of interest.

[0103] Finally, the omics features of the region of interest and the omics features of the peritumoral region are spliced as the radiomics image features. The above screening process not only enhances the biological interpretability of the radiomics model but also provides a high-quality feature basis for subsequent model construction.

[0104] R103 splices the deep learning image features and the radiomics image features to obtain the input information for the classification model; inputs the input information for the classification model into a classification network to obtain the class prediction result of the initial medical information.

[0105] Among them, the classification network is a convolutional neural network model, and the step of inputting the input information for the classification model into the classification network to obtain the class prediction result of the initial medical information includes: Using the convolutional kernels of the convolutional neural network model to extract the temporal features in the input information for the classification model, obtaining a feature map and unfolding it; classifying the unfolded feature map through two fully connected layers and outputting the prediction result of nerve invasion.

[0106] In a specific application, for the T2WI sequence and ADC map in the multi-parametric MRI images of prostate cancer, after ensuring the integrity and scanning accuracy of the images, the region of interest of the tumor is manually delineated and saved. Subsequently, four types of image files (including T2WI images, ADC images, region of interest images on T2WI, and region of interest images on ADC) are subjected to voxel normalization processing and input into the network of the above hybrid model, and the discrimination of whether there is nerve invasion can be obtained.

[0107] Multi-modal means using multiple types of medical image data for feature extraction and analysis. In the prediction of nerve invasion in prostate cancer, the multi-parametric MRI images include the T2WI sequence and the ADC map. T2-weighted imaging (T2WI) is an image used to reflect the T2 relaxation time difference between tissues, showing the structure and lesion area of the prostate, while the ADC map (apparent diffusion coefficient map) is an image calculated based on diffusion-weighted imaging (DWI) and is used to evaluate the microscopic diffusion characteristics of tissues. Images of different modalities provide complementary information about prostate lesions, helping to more comprehensively understand the characteristics of the disease and predict nerve invasion.

[0108] Multi-scale means considering image features of different sizes and ranges during the feature extraction process. Specifically in this application, the technical solution takes into account local features and global features. Local features such as the shape, size, and edge of the tumor are extracted from the region of interest (ROI) through image segmentation technology. Global features reflect the features of the entire prostate or surrounding tissues, including tissue density, texture, contrast, etc. Multi-scale feature extraction helps to capture the subtle differences between the tumor and its surrounding tissues, thereby improving the accuracy of prediction.

[0109] Multi-level means that in a deep learning model, different levels of features are extracted through different network levels. Specifically in this application, low-level features such as edges and corners extracted by shallower network layers are considered, and high-level features such as semantic features and context information extracted by deeper network layers are also considered. Through multi-level feature extraction, the prediction performance can be improved.

[0110] Multi-region means that when extracting image features in radiomics, not only the tumor region itself is considered, but also the surrounding tissue regions are considered. This application takes into account the tumor region and the peritumoral region. By considering the tumor region and the peritumoral region simultaneously, the invasiveness of the tumor and the risk of perineural invasion can be evaluated more comprehensively.

[0111] Obviously, the above-mentioned method for predicting perineural invasion based on a hybrid model, including the multi-level graph convolutional deep learning feature extraction method with a multi-scale attention mechanism and the radiomics image feature extraction method based on the combination of the region of interest and its surrounding regions, can fully integrate high-throughput semantic features of multi-modal, multi-scale, multi-level, and multi-region, achieve the complementary advantages of the two, and improve the prediction accuracy of perineural invasion in prostate cancer; at the same time, fully integrate high-throughput semantic features of multi-modal, multi-scale, multi-level, and multi-region, thereby improving the prediction performance.

[0112] In specific applications, the above-mentioned hybrid model is used to predict perineural invasion in prostate cancer on a dataset to prove the influence of the MRI modality on the classification performance (see Table 1), and the effect of the hybrid model proposed in this application compared with other methods (radiomics model, GCN model). The dataset is sourced from hospital data and a total of 711 patients are included. The detailed experimental results are shown in Table 2 (where the Hybrid model is the hybrid model protected by this application). In terms of the four evaluation indicators of accuracy, precision, recall, and F1 score, it is proved that the hybrid model proposed in this application can meet the expected requirements. The detailed experimental results are as follows: First of all, after testing, the hybrid model proposed in this application performs excellently in the four classification indicators: the accuracy is 0.830, the precision is 0.809, the recall is 0.826, and the F1 score is 0.817. The F1 score is an indicator used to evaluate the performance of a classification model and is the harmonic mean of precision and recall, which can comprehensively consider the accuracy and integrity of the model in predicting classification.

[0113] Compared with using a single model (such as a radiomics model and a GCN model), the hybrid model proposed in this application has a significant improvement in all evaluation metrics. For example, compared with only using the radiomics model (accuracy 0.760, precision 0.783, recall 0.720, F1-score 0.750), the hybrid model performs better; compared with the single GCN model (accuracy 0.810, precision 0.804, recall 0.787, F1-score 0.796), the hybrid model also further improves the F1-score. Secondly, referring to Table 1, two MRI modalities: T2WI and ADC are fused in the model. Compared with single-modal input (such as the accuracy of the T2WI and ADC modalities being 0.740 and 0.802 respectively), it can be observed that by combining the T2WI and ADC modalities, the performance of the model is further improved.

[0114]

[0115]

[0116] Obviously, the method for predicting perineural invasion based on the hybrid model provided in this example, through the deep fusion of the deep learning feature extraction technology of multi-scale convolution and attention mechanism, and the combination of radiomics feature analysis of tumor regions and peritumoral regions, can fully mine the high-throughput semantic features of multiple levels, multiple scales, and multiple regions in multi-modal MRI images, achieve complementary advantages, not only comprehensively capture the features of tumors from different angles, but also significantly improve the accuracy and robustness of perineural invasion prediction, meet the high-precision requirements of perineural invasion prediction in prostate cancer, and provide a more reliable basis for clinical diagnosis and treatment decisions during auxiliary diagnosis.

[0117] System embodiment.

[0118] An embodiment of this application provides a system for predicting perineural invasion based on a hybrid model, and its specific implementation manner is consistent with the implementation manner and the achieved technical effects recorded in the above method implementation manner, and some contents will not be repeated.

[0119] See Figure 7 , Figure 7 is a schematic diagram of the modules of a system for predicting perineural invasion provided by an embodiment of this application.

[0120] The system for predicting perineural invasion includes: An information acquisition module, which is used to acquire initial medical information, where the initial medical information includes medical images and data labels, the medical images are multi-parametric MRI images obtained by prostate magnetic resonance scanning of an examination object, and the data labels are used to indicate the tumor region of interest in the medical images; A feature extraction module, which is used to obtain deep learning image features based on the initial medical information through multi-scale convolution and attention mechanism of a deep learning model, and to examine the tumor region and the peritumoral region through a radiomics model to obtain radiomics image features; A result acquisition module, which is used to splice the deep learning image features and the radiomics image features to obtain classification model input information; and input the classification model input information into a classification network to obtain a category prediction result of the initial medical information.

[0121] In some embodiments, the feature extraction module includes: A preprocessing unit, which is used to preprocess the initial medical information to obtain target medical information; A feature extraction unit, which is used to input the target medical information into the deep learning model to obtain deep learning image features; and is also used to input the target medical information into the radiomics model to obtain radiomics image features; Wherein, the preprocessing includes: Resampling the multi-parametric MRI image and its corresponding data label to make the voxel sizes of all images consistent, and completing voxel normalization; Performing voxel value homogenization processing on the resampled MRI image, and adjusting the voxel values of each layer of the image to have a mean of 0 and a standard deviation of 1 through calculation; Based on the preset size requirements of the model input, adjusting the resampled and voxel value homogenized MRI image and its image layer of the region of interest according to the preset two-dimensional size to obtain standardized target medical information.

[0122] In some embodiments, The feature extraction unit includes: A first extraction subunit, which is used to input the target medical information into a trained deep learning model, and process the original multi-parametric MRI image and its region of interest through the feature extraction network of the deep learning model to extract the graph network features of the image; and further process and integrate the graph network features by using a graph convolutional network to obtain deep learning image features; A second extraction subunit, which is used to perform omics feature extraction and omics feature screening on the tumor region and the peritumoral region of the image of the target medical information; and obtain radiomics image features based on the finally screened omics features of the two; the peritumoral region is an annular region within a preset distance around the tumor region, and the tumor region and the peritumoral region are used as a combined region; the value range of the preset distance is not less than 4 mm and not more than 6 mm.

[0123] In some embodiments, the feature extraction network includes a pyramid convolution network, a residual connection module, and a channel attention module; The pyramid convolution network includes 3D convolution kernels of 9×9×9, 5×5×5, and 3×3×3. The pyramid convolution network is used to perform multi-scale information extraction on the feature map output by the CNN layer of the deep learning model, and then perform pipelined layer-by-layer fusion on the extracted multi-scale information to obtain intermediate features; The residual connection module includes two different types of anisotropic convolutions, which are used to capture inter-layer relationships and intra-layer detail information, and are fused with the intermediate features by means of vector dot addition; The channel attention module is used to receive the feature map after vector dot addition and abstract global information at the feature channel level, and finally outputs graph network features with a dimension of 2048.

[0124] In some embodiments, the graph convolution network is used to obtain graph network features from the feature extraction network. The tumor region of interest in the medical image of the target medical information and the complete image layer containing the tumor region of interest are both defined as nodes of the graph to obtain a node set; For each group of adjacent nodes in the node set, the edge distance between the adjacent nodes is calculated through a distance formula; When the edge distance is less than or equal to a predefined threshold, it is considered that there is a valid edge between the adjacent nodes corresponding to the edge distance; Among them, the predefined threshold is the sum of the mean and standard deviation of the defined node distances, and the distance formula is as follows:

[0125] d(i,j) is the edge distance between adjacent nodes i and j, V ik is the value of node i in the k-th feature dimension, V jk is the value of node j in the k-th feature dimension.

[0126] In some embodiments, the methods for performing omics feature extraction and omics feature screening on the tumor region and the peritumoral region of the image of the target medical information include: For the images in the target medical information, the tumor region and the peritumoral region are augmented through Log transformation and eight-direction wavelet transformation; Perform omics feature extraction on the augmented image according to the preset feature quantity types to obtain an overall feature set; the preset feature quantity types include shape features, first-order statistical features, gray-level co-occurrence matrix features, gray-level dependence matrix features, gray-level run-length matrix features, gray-level size zone matrix features, and neighborhood gray-level difference matrix features; Use LASSO regression to screen the features in the overall feature set to obtain the omics features of the tumor region of interest and the omics features of the peritumoral region; Take the tumor region of interest of the medical image as the original region of interest, and use the mirror comparison method to compare the overall features of the original region of interest with those of the mirror region of interest, obtain an omics feature set with statistical differences, and take the union with the omics features of the tumor region of interest obtained by LASSO regression to obtain the omics features of the region of interest; Output the combined omics features of the region of interest and the omics features of the peritumoral region as the radiomics image features.

[0127] In some embodiments, the initial medical information further includes an original gland label, and the using the mirror comparison method to compare the overall features of the original region of interest with those of the mirror region of interest to obtain an omics feature set with statistical differences includes: Obtain the boundary information of the prostate and the central axis of the prostate in the medical image; Quantify the degree of axial symmetry by calculating the Dice coefficient between the original gland label and the mirror gland label along the central axis; If the Dice coefficient is greater than 0.8, it is considered that the prostate region meets the requirement of axial symmetry; mirror-flip the tumor region of interest label along the central axis of the prostate to generate a mirror tumor region of interest label, and extract the radiomics image features of the mirror region of interest; Use statistical methods such as T-test, U-test, Cohen's d effect size analysis, and mean comparison to perform mirror comparison and screening on the radiomics image features of the mirror region of interest and the radiomics features of the original region of interest, and take the intersection of the selected features as the omics feature set with statistical differences.

[0128] In some embodiments, the classification network is a convolutional neural network model, and the result acquisition module includes: A feature extraction and unfolding unit, which is used to use the convolution kernels of the convolutional neural network model to extract the temporal features in the input information of the classification model, obtain a feature map and unfold it; A fully-connected classification unit, which is used to receive the feature map expanded by the feature extraction and expansion unit and output the prediction result of nerve invasion after classification through two fully-connected layers.

[0129] Device embodiment.

[0130] An embodiment of the present application provides an electronic device, the specific implementation manner of which is consistent with the implementation manners and the achieved technical effects described in the above method and system implementation manners, and some contents will not be elaborated here. The electronic device includes one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors to perform any of the above methods.

[0131] See Figure 8 , Figure 8 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

[0132] The electronic device may include, for example, at least one memory 11, at least one processor 12, and a bus 13 connecting different platform systems.

[0133] The memory 11 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 111 and / or a cache memory 112, and may further include a read-only memory (ROM) 113.

[0134] Wherein, the memory 11 further stores a computer program, and the computer program can be executed by the processor 12 so that the processor 12 implements the steps of any of the above methods.

[0135] The memory 11 may further include a utility 114 having at least one program module 115. Such program modules 115 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. The implementation of a network environment may be included in each or some combination of these examples.

[0136] Correspondingly, the processor 12 may execute the above computer program and may also execute the utility 114.

[0137] The processor 12 may be implemented using one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0138] The bus 13 may represent one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor bus, or any bus architecture using multiple bus architectures.

[0139] The electronic device may also communicate with one or more external devices 14, such as a keyboard, a pointing device, a Bluetooth device, etc., and may also communicate with one or more devices capable of interacting with the electronic device, and / or communicate with any device (such as a router, a modem, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through the input / output interface 15. Further, the electronic device may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 16. The network adapter 16 may communicate with other modules of the electronic device through the bus 13. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device in practical applications, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0140] It should be noted that in the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. " / ", describing the association relationship of associated objects, indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c may represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c may be single or multiple. It is worth noting that "at least one (item)" may also be interpreted as "one (item) or more items (items)".

[0141] The terms "first", "second", etc. in the description, claims and the above-mentioned drawings of this application are configured to distinguish similar objects and do not necessarily need to be configured to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0142] This application is described from the perspectives of purpose of use, efficacy, progress and novelty, etc., and has met the functional improvement and use requirements emphasized by the patent law. The above description and the accompanying drawings of this application are only preferred embodiments of this application and do not limit this application thereby. Therefore, all those that are similar or identical to the structure, device, features, etc. of this application, that is, all equivalent substitutions or modifications made according to the scope of the patent application of this application, shall fall within the scope of protection of the patent application of this application.

Claims

1. A neural invasion prediction system based on a hybrid model, characterized in that, The system includes: An information acquisition module, configured to acquire initial medical information, where the initial medical information includes medical images and data tags, the medical images are multi-parametric MRI images obtained by prostate magnetic resonance scanning of an examination object, and the data tags are used to indicate the tumor region of interest in the medical images; A feature extraction module, configured to obtain deep learning image features through multi-scale convolution and attention mechanism of a deep learning model based on the initial medical information, and to examine the tumor region and the peritumoral region through a radiomics model and obtain radiomics image features; A result acquisition module, configured to splice the deep learning image features and the radiomics image features to obtain classification model input information; and to input the classification model input information into a classification network to obtain a class prediction result of the initial medical information.

2. The neural infringement prediction system based on a hybrid model according to claim 1, wherein The feature extraction module includes: A preprocessing unit, configured to preprocess the initial medical information to obtain target medical information; A feature extraction unit, configured to input the target medical information into the deep learning model to obtain deep learning image features; and also configured to input the target medical information into the radiomics model to obtain radiomics image features; Wherein, the preprocessing includes: Resampling the multi-parametric MRI images and their corresponding data tags to make the voxel sizes of all images consistent, and completing the normalization of the voxels; Performing voxel value normalization processing on the resampled MRI images, and adjusting the voxel values of each layer of images to have a mean of 0 and a standard deviation of 1 through calculation; Based on the preset size requirements of the model input, adjusting the resampled and voxel value-normalized MRI images and their region-of-interest image layers according to a preset two-dimensional size to obtain standardized target medical information.

3. The neural infringement prediction system based on the hybrid model according to claim 2, characterized in that The feature extraction unit includes: A first extraction subunit, configured to input the target medical information into a trained deep learning model, and process the original multi-parametric MRI images and their regions of interest through the feature extraction network of the deep learning model to extract the graph network features of the images; and further process and integrate the graph network features by using a graph convolutional network to obtain deep learning image features; A second extraction subunit, configured to perform omics feature extraction and omics feature screening on the tumor region and the peritumoral region of the target medical information image; and obtain radiomics image features based on the finally screened omics features of the two; the peritumoral region is an annular region within a preset distance around the tumor region, and the tumor region and the peritumoral region are used as a combined region; the value range of the preset distance is not less than 4 mm and not more than 6 mm.

4. The neural infringement prediction system based on a hybrid model according to claim 3, wherein The feature extraction network includes a pyramid convolutional network, a residual connection module, and a channel attention module; The pyramid convolution network includes 3D convolution kernels of 9×9×9, 5×5×5, and 3×3×3. The pyramid convolution network is used to perform multi-scale information extraction on the feature map output by the CNN layer of the deep learning model, and then perform pipelined layer-by-layer fusion on the extracted multi-scale information to obtain intermediate features; The residual connection module includes two different types of anisotropic convolutions, which are used to capture inter-layer relationships and intra-layer detail information, and are fused with the intermediate features in a vector dot-adding manner; The channel attention module is used to receive the feature map after vector dot-adding and abstract global information at the feature channel level, and finally outputs a graph network feature with a dimension of 2048; 5. The neural infringement prediction system based on a hybrid model according to claim 4, wherein The graph convolution network is used to obtain the graph network feature from the feature extraction network, and define both the tumor region of interest and the complete image layer containing the tumor region of interest in the medical image of the target medical information as nodes of the graph to obtain a node set; For each group of adjacent nodes in the node set, the edge distance between the adjacent nodes is calculated through a distance formula; When the edge distance is less than or equal to a predefined threshold, it is considered that there is a valid edge between the adjacent nodes corresponding to the edge distance; Among them, the predefined threshold is the sum of the mean and standard deviation of the defined inter-node distance, and the distance formula is as follows: d(i,j) is the edge distance between adjacent nodes i and j, V ik is the value of node i on the k-th feature dimension, V jk is the value of node j on the k-th feature dimension.

6. The neural infringement prediction system based on the hybrid model according to claim 3, wherein For the tumor region and the peritumoral region of the image of the target medical information, the methods for extracting and screening omics features include: For the image in the target medical information, the tumor region and the peritumoral region are augmented through Log transformation and wavelet transformation in eight directions; Omics features are extracted from the augmented image according to the preset feature quantity types to obtain an overall feature set; the preset feature quantity types include shape features, first-order statistical features, gray-level co-occurrence matrix features, gray-level dependence matrix features, gray-level run-length matrix features, gray-level size zone matrix features, and neighborhood gray-level difference matrix features; LASSO regression is used to screen the features in the overall feature set to obtain the omics features of the tumor region of interest and the omics features of the peritumoral region; Taking the tumor region of interest of the medical image as the original region of interest, the overall features of the original region of interest and the mirror region of interest are compared using the mirror comparison method to obtain a set of omics features with statistical differences, and the union is taken with the omics features of the tumor region of interest obtained by LASSO regression to obtain the omics features of the region of interest; The omics features of the region of interest and the omics features of the peritumoral region are jointly output as radiomics image features; 7. The neural infringement prediction system based on the hybrid model according to claim 6, characterized in that The feature combination is to splice the omics features of the region of interest and the omics features of the peritumoral region; The initial medical information also includes the original gland label. The method of comparing the overall features of the original region of interest and the mirror region of interest using the mirror comparison method to obtain a set of omics features with statistical differences includes: Obtaining the boundary information of the prostate and the central axis of the prostate in the medical image; Quantify the degree of axial symmetry by calculating the Dice coefficient between the original gland label and the mirrored gland label along the central axis; If the Dice coefficient is greater than 0.8, it is considered that the prostate region meets the requirements of axial symmetry; flip the tumor region of interest label along the central axis of the prostate to generate a mirrored tumor region of interest label, and extract the omics image features of the mirrored region of interest; Use statistical methods such as T-test, U-test, Cohen's d effect size analysis, and mean comparison to perform mirrored comparison and screening on the omics image features of the mirrored region of interest and the omics features of the original region of interest, and take the intersection of the selected features as the set of omics features with statistical differences.

8. The neural infringement prediction system based on a hybrid model according to claim 1, wherein The classification network is a convolutional neural network model, and the result acquisition module includes: A feature extraction and expansion unit, which is used to extract the temporal features in the input information of the classification model using the convolutional kernels of the convolutional neural network model, obtain a feature map and expand it; A fully connected classification unit, which is used to receive the feature map expanded by the feature extraction and expansion unit, and output the prediction result of nerve invasion after classification through two layers of fully connected layers.

9. A method for predicting nerve invasion based on a hybrid model, implemented based on a nerve invasion prediction system based on a hybrid model according to any one of claims 1-8, characterized in that, The method includes the steps of: S101, obtain initial medical information, where the initial medical information includes a medical image and a data label, the medical image is a multi-parameter MRI image obtained by prostate magnetic resonance scanning of an examination object, and the data label is used to indicate the tumor region of interest in the medical image; S102, based on the initial medical information, obtain deep learning image features through multi-scale convolution and attention mechanism of a deep learning model, and examine the tumor region and the peritumoral region through an imaging omics model to obtain imaging omics image features; S103, splice the deep learning image features and the imaging omics image features to obtain the input information of the classification model; input the input information of the classification model into the classification network to obtain the category prediction result of the initial medical information.

10. An electronic device, characterized in that, Includes one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors to perform the method according to claim 9.

Images