Ovarian tumor cross malignancy classification system based on hybrid radiomics model

Abstract

The invention discloses an ovarian tumor cross malignancy classification system based on a hybrid radiomics model. On one hand, a backbone network in the Med3D network is used as a depth feature extractor to construct a target network, and multi-modal 3D magnetic resonance image data is used to carry out fine tuning training on the target network to obtain a corresponding ovarian tumor classification network for obtaining depth features and tumor classification results. On the other hand, omics characteristics of the tumor region of interest are extracted, and a tumor classification result corresponding to the omics prediction model is obtained. In addition, the depth features and the omics features are fused, and a corresponding classification result is obtained through a KNN classification algorithm. And finally, obtaining a final ovarian tumor cross malignancy classification result according to weighted average of the three classification results. According to the invention, by integrating various models and algorithms, more comprehensive feature mining and analysis are carried out on the multi-modal MRI image data of the ovarian tumor patient, so that the final ovarian tumor classification effect is improved.

Description

Technical Field

[0001] The present invention relates to the fields of radiomics and deep learning, and in particular to an ovarian tumor cross-malignancy classification system based on a hybrid radiomics model. Background Art

[0002] Borderline Ovarian Tumors (BOTs) and Malignant Ovarian Tumors (MOTs) are two major types of ovarian tumors, which have significant differences in biological behavior, clinical characteristics, treatment strategies, and prognosis. BOTs are intermediate entities different from benign and malignant ovarian tumors. Most BOTs do not invade ovarian tissue, and conservative surgery to preserve fertility can be considered for BOT patients in appropriate cases. Malignant ovarian tumors are highly malignant ovarian epithelial tumors with a mortality rate much higher than that of BOTs; for MOT patients, the preferred treatment is surgical resection combined with chemotherapy. Therefore, accurate preoperative diagnosis of borderline ovarian tumors and malignant ovarian tumors is crucial for optimal patient management. However, due to the great similarities between BOT and MOT in biology, histopathology, clinical and imaging features (reference: Borrelli GM, de Mattos LA, de Paula Andres M, et al. Role of imaging tools for the diagnosis of borderline ovarian tumors: a systematic review and meta-analysis[J]. Journal of minimally invasive gynecology, 2017, 24(3): 353-363.), accurate preoperative differential diagnosis between the two remains a challenge.

[0003] Magnetic resonance imaging (MRI) is a non-invasive imaging technology commonly used in the clinical diagnosis of ovarian tumors. It is currently also used for standardized examinations for the diagnosis of ovarian tumor types. After obtaining MRI image data, radiologists usually obtain disease-related information contained in the MRI through visual recognition. This method has great limitations, such as high subjective dependence, low time efficiency, and insufficient image information mining. In recent years, the radiomics method, which has developed rapidly, can achieve fast and efficient ovarian tumor classification by combining MRI image data with artificial intelligence algorithms. According to different feature extraction strategies, radiomics methods can be roughly divided into traditional radiomics methods based on manual features and methods based on deep learning. Traditional imaging genomics methods first extract tumor feature information such as shape and texture in a predefined manner, and then use traditional machine learning algorithms to screen features and build prediction models (Reference: Li Y, Jian J, Pickhardt PJ, et al. MRI-based machine learning for differentiatingborderline from malignant epithelial ovarian tumors: A multicenter study[J]. Journal of Magnetic Resonance Imaging, 2020, 52(3):897-904.). Unlike traditional imaging genomics methods, deep learning methods can implement the entire analysis process through a deep neural network, including automatic extraction of imaging features, feature screening, and output of prediction results. Current studies have shown that data-driven deep learning methods can achieve a high prediction accuracy in ovarian tumor classification (Reference: Wang Y, Zhang H, Wang T, et al. Deep learning for the ovarian lesion localization and discrimination between borderline and malignant ovarian tumors based on routine MR imaging[J]. Scientific Reports, 2023,13(1):2770.).

[0004] It should be pointed out that although deep learning-based methods can simplify the analysis process and improve prediction performance, they also have some disadvantages, such as high demand for training data and poor interpretability. In contrast, the feature extraction method, feature selection algorithm, model construction and other parts of traditional radiomics methods all need to rely on the design experience of the algorithm designer to make subjective choices, but this type of method usually has good interpretability. In addition, the image features extracted by deep learning methods often focus on local features, while the features extracted by traditional radiomics methods focus more on the global features of tumors (reference: Ning Z, Luo J, Li Y, et al. Pattern classification for gastrointestinal stromal tumors by integration of radiomics and deepconvolutional features[J]. IEEE journal of biomedical and health informatics, 2018, 23(3):1181-1191.). In summary, traditional radiomics methods and deep learning methods have their own advantages and disadvantages. How to develop a hybrid radiomics model that combines the advantages of both has been a research hotspot in recent years. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art. Based on the above research status, an ovarian tumor borderline classification system based on a hybrid imaging omics model is proposed for the classification of ovarian borderline tumors and malignant tumors. The hybrid imaging omics model proposed in the present invention can provide a certain degree of interpretability while improving the tumor classification performance. The present invention first constructs an ovarian tumor classification model through deep learning and traditional imaging omics methods, and obtains their respective prediction results; then the deep features and omics features are fused, and then the corresponding prediction results are obtained based on the fused features and the K nearest neighbor (KNN) algorithm; finally, the above prediction results are combined to obtain the final tumor classification prediction results. In addition, the present invention adopts a training scheme based on three-dimensional transfer learning to solve the problem of insufficient training data when training deep learning models, as well as the common data type differences and dimensional differences in transfer learning.

[0006] The technical solution of the present invention is: an ovarian tumor cross-malignancy classification system based on a hybrid imaging omics model, the system specifically comprising: The data acquisition module is used to acquire multimodal 3D magnetic resonance imaging data of ovaries of patients with different ovarian tumors and construct a data set; Tumor segmentation module, used to segment the tumor region of interest in the dataset; The network construction module is used to use the backbone network in the pre-trained Med3D model as a feature extraction network and jointly construct a tumor classification network based on deep learning with a classifier network based on a fully connected layer; The network training module is used to extract the tumor part in the image data based on the tumor region of interest segmentation result, and then cut it into the input of the tumor classification network based on deep learning, and train it based on the tumor classification label data; Feature extraction module, which is used to extract tumor imaging features and screen out key features based on the patient's 3D magnetic resonance imaging data and tumor region of interest segmentation results; Omics classification module, which is used to build an omics classification model based on machine learning algorithms and train the omics classification model using key feature data and tumor classification labels; The feature fusion module is used to use the output of the feature extraction network in the trained tumor classification network as the deep feature, combine it with the tumor imaging features to obtain the fusion feature, and build a KNN classification model based on the fusion feature of the training data; The tumor classification module is used to process the multimodal 3D MRI image data of the ovaries of any ovarian tumor patient using a hybrid model composed of a deep learning-based tumor classification network, an omics classification model, and a KNN classification model to obtain three independent tumor classification results. The final tumor classification result of the hybrid model is obtained by weighted averaging these three classification results.

[0007] Furthermore, the ovarian multimodal 3D magnetic resonance imaging data includes two modalities, T2 and DWI, and the data are pre-processed by denoising and resolution adjustment.

[0008] Furthermore, in the construction of a tumor classification network based on deep learning, in order to process multimodal imaging data, a feature extraction network with the same network structure is used for each modality. After the deep features of each modality are extracted, they are spliced ​​as the final deep features and input into the subsequent classifier network based on the fully connected layer.

[0009] Furthermore, the training of the deep learning-based tumor classification network is achieved through transfer learning, that is, the backbone network in the pre-trained Med3D model is first used to initialize the feature extraction network part of each modality in the deep learning-based tumor classification network, and then the 3D magnetic resonance imaging data of ovarian tumor patients are used to fine-tune the classifier network based on the fully connected layer and some convolutional layers of the feature extraction network.

[0010] Furthermore, when extracting the omics features of multimodal imaging data, the omics features of each modality are extracted using the imaging data of each modality and the tumor region of interest data, and spliced ​​to obtain the final omics features, and then subsequent feature screening processing is performed.

[0011] Furthermore, in the construction of the omics classification model, the selection of feature selection models and prediction models based on machine learning algorithms is to select the model combination with the best classification performance on the validation set from multiple feature selection models and binary classification models through the traversal method.

[0012] Furthermore, the process of acquiring fusion features in the feature fusion module is to first screen the deep features through a feature selection algorithm, and then splice the deep features obtained after the screening with the key features screened out by the tumor imaging genomics features to obtain the fusion features.

[0013] Furthermore, the imaging data of an ovarian tumor patient is input, and the prediction process of the KNN model is to find the k cases in the training set that are most similar to the fusion features of the patient's imaging data, and take the mean of the tumor classification results of these k cases as the patient's survival prediction result. The specific k value is determined by the cross-validation method.

[0014] Furthermore, the final result of tumor classification of the hybrid model is the weighted average result of the output values ​​of the three independent prediction models, where the weight coefficients are empirically set to a set of fixed values ​​or determined by performing multivariate linear regression analysis on the training data.

[0015] Furthermore, the hybrid model simultaneously outputs interpretable results, specifically, outputting several cases similar to the patient in the fusion feature space based on the patient's imaging data for the doctor's reference or comparative analysis.

[0016] Compared with the background technology, the beneficial effect of the present invention is that the present invention realizes more comprehensive feature mining and analysis of multimodal MRI image data of ovarian tumor patients by comprehensively utilizing deep learning models, traditional imaging genomics analysis models and KNN algorithms, thereby improving the final ovarian tumor classification effect. And by adopting a three-dimensional transfer learning training scheme and data enhancement technology, the present invention solves the problem of insufficient training data in the deep neural network training process to a certain extent, as well as the common data type differences and dimensional differences in transfer learning, thereby improving the prediction performance of the deep learning model and the effectiveness of deep features. In addition, by combining the deep learning model with the traditional imaging genomics analysis method and the KNN algorithm, the hybrid imaging genomics model proposed in the present invention has a certain degree of interpretability while achieving higher tumor classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 Schematic diagram of the hybrid imaging omics model proposed in the present invention.

[0019] Figure 2 It is a schematic diagram of the three-dimensional transfer learning solution in the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific implementation examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0021] The present invention provides an ovarian tumor cross-malignancy classification system based on a hybrid imaging omics model. The schematic diagram of the model used by the system of the present invention is as follows: Figure 1 As shown in the figure, the backbone network in the Med3D model trained based on a large number of 3D medical image data sets is used as the deep feature extraction network, and several fully connected layers are used as the classifier network to jointly construct the ovarian tumor classification network. After collecting and collating the T2 modality and DWI modality MRI data of patients with ovarian tumors, they are first preprocessed and the tumor region ROI is segmented. The multimodal MRI image data and tumor classification label data of patients with ovarian tumors are used as training data sets to fine-tune the fully connected layers and some convolutional layers of the tumor classification network. Then, on the one hand, the deep features of the MRI data are extracted by the trained multimodal deep learning model, and the first corresponding deep learning model classification prediction value is obtained; on the other hand, the omics features of the multimodal MRI image data are extracted by the traditional imaging omics method, and the second corresponding omics model classification prediction value is obtained by the traditional machine learning algorithm. In addition, the deep features and omics features are fused and analyzed, and the third corresponding classification prediction value is obtained based on the fusion features and the KNN model. Finally, the weighted average result of the above three prediction values ​​is taken as the final ovarian tumor classification result. Next, the processing flow of each module in the ovarian tumor cross-malignancy classification system of the present invention will be described in detail according to a specific embodiment of the present invention, as follows: The data acquisition module is used for data collection and preprocessing. First, multimodal (T2, DWI) ovarian 3D MRI image data of 419 patients with ovarian tumors (107 borderline cases and 312 malignant cases) and corresponding tumor cross-malignant category label data were collected from a hospital. In order to ensure sample balance and improve data quality as much as possible, 150 cases with good image quality and all borderline ovarian tumor data were selected from the malignant ovarian tumor data for subsequent experiments. The spatial resolution of these 3D MRI image data in the x, y, and z directions was uniformly adjusted to 1.5 ´ 1.5 ´ 7mm by resampling, and the image pixel resolution was 512 ´ 512 ´ 24. In addition, the image data of each modality were aligned. 80% of all patient data were randomly selected as training sets, and the remaining 20% ​​were used as test sets.

[0022] Tumor segmentation module is used for tumor ROI region segmentation. The ovarian tumor ROI region is segmented through the semi-automatic segmentation function in the medical image segmentation tool ITK-SNAP. After obtaining the semi-automatic segmentation result, the segmentation result is fine-tuned to achieve the accuracy and reliability of the segmentation result.

[0023] Network building module for building multimodal deep learning tumor classification models. Figure 2 As shown in the figure, the backbone network (implemented based on 3D-ResNet18) in the Med3D model pre-trained based on the 3DSeg dataset (a summary dataset consisting of 23 3D medical imaging public datasets) is used as the feature extraction network for each modality imaging data. The pre-trained Med3D model can be obtained from the following link: https: / / github.com / Tencent / MedicalNet. The output of the above feature extraction network is used as the deep feature, and the deep features extracted from the T2 modality and DWI modality imaging data are spliced ​​to obtain the deep features of the multi-modality imaging data, which are then used as the input of the fully connected layer classifier. The fully connected layer classifier includes 3 layers in total, with the number of nodes in the 1st, 2nd, and 3rd layers being 1153, 128, and 2, respectively. The outputs of the 2 nodes in the last layer are used as the predicted probabilities of the tumor types of borderline ovarian tumors and malignant ovarian tumors, respectively.

[0024] The network training module is used to train the deep learning model. First, the tumor part of the multimodal image is extracted using the mask data of the tumor ROI. That is, the tumor part with a size of 256 ´ 256 ´ 12 is cropped out with the tumor as the center as the input of the deep learning model, and the real label of the tumor category is used as the training label. After the above preprocessing of the training data set, the deep learning model is fine-tuned and trained. The training part includes the last module of the feature extraction network (including 4 convolutional layers) and all fully connected layers, such as Figure 2 As shown. The training parameter batch size is set to 16, the optimizer is Adam, the learning rate is 0.001, and the maximum number of iteration epochs is 50. After the training is completed, the feature extraction network corresponding to each modality is obtained for the extraction of deep features. That is, when a multimodal 3D MRI image data is input, the deep features of each modality can be extracted through the trained feature extraction network, and then the final deep feature data can be obtained by splicing.

[0025] Feature extraction module, used for multimodal omics feature extraction. Based on the 3D MRI image data in the data acquisition module and the tumor ROI segmentation result mask data in the tumor segmentation module, the pyradiomics library is used to extract the omics feature data of the tumor ROI area of ​​each modality image data. The same type and number of omics features are extracted for each modality. The categories of omics features mainly include first-order statistical features, texture features, and high-order transformation features. The omics features of each modality are spliced ​​to obtain the final multimodal omics feature data.

[0026] The omics classification module is used for the construction and training of traditional omics classification models. First, the above omics feature data are pre-screened, that is, the features with low variance (variance < 0.05) and the features with no significant difference between groups (t-test, p > 0.05) are deleted. Then, the optimal model combination of "feature selection + classification prediction" with the best classification performance on the validation set (20% of the training set is randomly selected as the validation set) is selected by the traversal method as the final omics classification model, and the small number of omics features screened by the feature selection model are called key features. The three traversal dimensions are: feature selection model (Lasso, HSICLasso, ANOVA, MRMRe, AdaBoost), binary classification model (LogisticRegression, SVM, RandomForest, GradientBoosting), and the number of key features (10, 20, 30, 40, 50).

[0027] Feature fusion module is used for feature fusion and construction of KNN prediction model. Figure 1As shown in the figure, the multimodal deep features and multimodal omics features are respectively subjected to feature screening and then spliced ​​and fused to obtain fusion features, wherein the deep feature screening is implemented by the HSICLasso algorithm. Then, a KNN prediction model is constructed based on the fusion features of the training data and the KNN algorithm. After the model is constructed, given the multimodal imaging data of a new patient, its fusion features can be extracted and input into the KNN prediction model. The KNN model calculates the 15 training data set cases that are most similar to the patient in the fusion feature space, and finally obtains the prediction result of the patient's tumor category by counting the true labels of the tumor categories of these 15 cases.

[0028] The tumor classification module is used for the construction and performance evaluation of the hybrid imaging omics model. Based on the test data set in the data acquisition module, the tumor ROI in the imaging data is first segmented using the method in the tumor segmentation module. Then, the test set data is processed with the deep learning prediction model trained by the network training module to obtain the tumor category prediction probability (P1) of the deep learning model and the deep feature data; at the same time, the omics feature data is extracted using the method in the feature extraction module and input into the omics prediction model trained in the omics classification module to obtain the tumor category prediction probability (P2) of the omics model. In addition, the fusion feature data of the test data set is obtained using the method in the feature fusion module and input into the KNN model therein to obtain the tumor category prediction probability (P3) of the KNN model. The weighted average result of the above three prediction probabilities is used as the final prediction result of the hybrid imaging omics model. In the application of the present invention, the prediction results of the three models are considered to be equally important, so the average value (P1+P2+P3) / 3 is directly taken as the final prediction result. By comparing the final prediction results of the test set with its true survival category label, the classification performance of the hybrid model on the test set can be calculated (the classification performance is characterized by the AUC indicator commonly used in the medical field). The experimental results show that the test set AUC of the hybrid imaging omics model proposed in the present invention in the ovarian tumor cross-malignancy classification task can reach about 0.92. Compared with some existing methods in recent years, this indicator shows that the hybrid model of the present invention has a significant improvement in the cross-malignancy classification performance of ovarian tumors.

[0029] The above is a specific embodiment of an ovarian tumor cross-malignancy classification system based on a hybrid imaging genomics model provided by the present invention. Through the above 8 main modules, an ovarian tumor cross-malignancy classification model based on multimodal (T2 and DWI) 3D MRI image data and the performance evaluation results of the model can be obtained. Then, given any ovarian tumor patient's ovarian multimodal 3D MRI image data, the above prediction model can be used to process and obtain the corresponding cross-malignant tumor category prediction results.

[0030] The present invention has good scalability. The present invention provides a mode combining a deep learning model with a traditional imaging genomics model, and the selection of ovarian tumor subdivision categories, the selection of modalities in multimodal imaging data, the definition of tumor ROI areas, and the feature extraction network structure of the deep learning model, the selection of tumor ROI segmentation algorithms, and feature selection algorithms, etc., in the above embodiments can all be appropriately adjusted according to the specific circumstances of other embodiments.

[0031] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A cross-malignancy classification system for ovarian tumors based on a hybrid radiomics model, characterized in that: The system specifically includes: The data acquisition module is used to acquire multimodal 3D magnetic resonance imaging data of ovaries of patients with different ovarian tumors and construct a data set; Tumor segmentation module, used to segment the tumor region of interest in the dataset; The network construction module is used to use the backbone network in the pre-trained Med3D model as a feature extraction network and jointly construct a tumor classification network based on deep learning with a classifier network based on a fully connected layer; The network training module is used to extract the tumor part in the image data based on the tumor region of interest segmentation result, and then cut it into the input of the tumor classification network based on deep learning, and train it based on the tumor classification label data; Feature extraction module, which is used to extract tumor imaging features and screen out key features based on the patient's 3D magnetic resonance imaging data and tumor region of interest segmentation results; Omics classification module, which is used to build an omics classification model based on machine learning algorithms and train the omics classification model using key feature data and tumor classification labels; The feature fusion module is used to use the output of the feature extraction network in the trained tumor classification network as the deep feature, combine it with the tumor imaging features to obtain the fusion feature, and build a KNN classification model based on the fusion feature of the training data; The tumor classification module is used to process the multimodal 3D MRI image data of the ovaries of any ovarian tumor patient using a hybrid model composed of a deep learning-based tumor classification network, an omics classification model, and a KNN classification model to obtain three independent tumor classification results. The final tumor classification result of the hybrid model is obtained by weighted averaging these three classification results.

2. The ovarian tumor cross-malignancy classification system based on hybrid radiomics model according to claim 1, characterized in that: The multimodal 3D MRI data of ovaries include T2 and DWI, and the data are preprocessed by denoising and resolution adjustment.

3. The ovarian tumor cross-malignancy classification system based on hybrid radiomics model according to claim 1, characterized in that: In the construction of a tumor classification network based on deep learning, in order to process multimodal imaging data, a feature extraction network with the same network structure is used for each modality. After the deep features of each modality are extracted, they are spliced ​​as the final deep features and input into the subsequent classifier network based on the fully connected layer.

4. The ovarian tumor cross-malignancy classification system based on hybrid radiomics model according to claim 1, characterized in that: The training of the deep learning-based tumor classification network is achieved through transfer learning, that is, the backbone network in the pre-trained Med3D model is first used to initialize the feature extraction network part of each modality in the deep learning-based tumor classification network, and then the 3D magnetic resonance imaging data of ovarian tumor patients are used to fine-tune the classifier network based on the fully connected layer and some convolutional layers of the feature extraction network.

5. The ovarian tumor cross-malignancy classification system based on hybrid radiomics model according to claim 1, characterized in that: When extracting omics features of multimodal imaging data, the omics features of each modality are extracted using the imaging data of each modality and the tumor region of interest data respectively, and then spliced ​​to obtain the final omics features, and then subsequent feature screening processing is performed.

6. The ovarian tumor cross-malignancy classification system based on hybrid radiomics model according to claim 1, characterized in that: In the construction of omics classification models, the selection of feature selection models and prediction models based on machine learning algorithms is to select the model combination with the best classification performance on the validation set from multiple feature selection models and binary classification models through the traversal method.

7. The ovarian tumor cross-malignancy classification system based on hybrid radiomics model according to claim 1, characterized in that: The process of acquiring fusion features in the feature fusion module is to first screen the deep features through the feature selection algorithm, and then splice the deep features obtained after the screening with the key features screened out by the tumor imaging genomics features to obtain the fusion features.

8. The ovarian tumor cross-malignancy classification system based on hybrid radiomics model according to claim 1, characterized in that: Input the imaging data of an ovarian tumor patient. The prediction process of the KNN model is to find the k cases in the training set that are most similar to the fusion features of the patient's imaging data, and use the mean of the tumor classification results of these k cases as the patient's survival prediction result. The specific k value is determined by the cross-validation method.

9. The ovarian tumor cross-malignancy classification system based on hybrid radiomics model according to claim 1, characterized in that: The final result of tumor classification of the hybrid model is the weighted average result of the output values ​​of the three independent prediction models, where the weight coefficients are empirically set to a set of fixed values ​​or determined by performing multivariate linear regression analysis on the training data.

10. The ovarian tumor cross-malignancy classification system based on hybrid radiomics model according to claim 1, characterized in that: The hybrid model also outputs interpretable results, specifically, outputting several cases similar to the patient's image data in the fusion feature space for the doctor's reference or comparative analysis.

Images