A malignant liver tumor classification method based on multimodal data fusion
Through multimodal data fusion technology, convolutional neural networks and recurrent neural networks are used to extract and encode CT image features, combined with clinical information, the problem of low diagnostic accuracy of liver tumors in the existing technology is solved, and the malignant liver tumors is achieved. High accuracy classification.
Patent Information
- Application Number
- CN202110946370.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-18
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-08-18
AI Technical Summary
The prior art is difficult to accurately distinguish different categories of malignant tumors in liver tumor diagnosis, especially in rural or remote areas, and the classification method based on deep learning models fails to effectively utilize the timing relationship between clinical information and multi-phase CT images, resulting in classification. Low accuracy.
The spatial feature extraction algorithm based on convolutional neural network and the timing feature encoding method of recurrent neural networks are adopted, and the classification accuracy of malignant liver tumors is improved through the multimodal data fusion module.
It significantly improved the classification accuracy of malignant liver tumors, especially in the classification of primary liver cancer by 25.9%, and performed better than traditional methods in the differential diagnosis of multi-stage liver malignant tumors.
Smart Images

Figure CN113657503B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of liver tumor image analysis, and in particular to a malignant liver tumor classification method based on multimodal data fusion. Background Art
[0002] According to global cancer statistics, liver cancer is the sixth most commonly diagnosed cancer and the third leading cause of cancer death worldwide. The five-year relative survival rate is only 18.3%, and a large number of liver malignancies are primary tumors, including hepatocellular carcinoma (HCC) and intrahepatic cholangiocarcinoma (ICC). Different types of liver malignancies have different treatment options, so it is crucial to diagnose their specific types before surgery. However, it is easy to misdiagnose clinically. Moreover, long-term follow-up often brings panic and anxiety to patients. There is an urgent need to develop a reliable and practical non-invasive imaging method to accurately distinguish different types of malignant liver tumors to guide clinical management strategies.
[0003] Multi-phase contrast-enhanced CT (CECT) provides a complete image description of the patient's liver, reflecting the changes in blood supply of different liver tumors, and is the main tool for diagnosing liver tumors. However, the diagnosis method of using the complete liver image provided by CT to distinguish malignant tumors is often time-consuming and laborious, and is even more difficult for hospitals in rural or remote areas. In recent years, classification methods based on deep learning models have also been applied. The technology used mainly uses CT or MRI information to classify images of benign and malignant tumors. However, there is no classification model that combines clinical information with CT images, nor is there a classification technology that considers the temporal relationship between multi-phase images when classifying multi-phase CT images, resulting in low classification accuracy. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a malignant liver tumor classification method based on multimodal data fusion. The spatial feature extraction algorithm based on convolutional neural network is used to obtain the single-phase image features, and the temporal feature encoding method based on recurrent neural network is used to mine the spatial features of multi-phase images. The multimodal fusion module is used to combine structured and unstructured information, which can significantly improve the accuracy of malignant liver tumor classification.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] A malignant liver tumor classification method based on multimodal data fusion, the method comprising the following steps:
[0007] S1. Dataset acquisition:
[0008] Collect preoperative clinical information and CECT images of patients with pathologically confirmed HCC, ICC, and metastatic liver cancer, and divide the collected sample data into training set samples and test set samples;
[0009] The CECT images were scanned using a 64-channel MDCT scanner, with a slice thickness of 5 mm and a reconstruction interval of 1 mm. Each image included a plain scan phase, an arterial phase, and a portal venous phase.
[0010] S2. CECT image preprocessing:
[0011] Export CECT images including plain scan phase, arterial phase and portal venous phase from the image archiving and communication system, and pre-process the CECT images to form a training image library;
[0012] The 2D slice with the largest tumor area along the z-axis in the 3D CECT image was selected as the input of the subsequent model; the HU value range of the CECT image in the original DICOM format image was limited to a window width of 400 and a window level of 40 to reduce the noise from irrelevant organs.
[0013] Furthermore, the plain scan period was selected as a reference, and the CECT images of other periods were registered using the affine registration algorithm, and all images were resized to 224 × 224 pixels by linear interpolation to fit the input of the model.
[0014] S3. Clinical data coding:
[0015] Data were coded for age, sex, platelets, total bilirubin, alpha-fetoprotein, carbohydrate antigen 19-9, carcinoembryonic antigen, carbohydrate antigen 125, and hepatitis B surface antigen of the samples;
[0016] Specifically, the age of the samples was divided into five intervals and then encoded into dummy variables using four binary variables; the gender of the samples was represented by a binary variable; platelet, total bilirubin, alpha-fetoprotein, carbohydrate antigen 19-9, carcinoembryonic antigen, carbohydrate antigen 125, and hepatitis B surface antigen data were used as preoperative blood biochemical indicators of patients and converted into categorical variables and encoded as dummy variables based on whether they were normal or missing.
[0017] S4. Model architecture:
[0018] Based on the modular design concept of spatial feature extraction-temporal feature encoding-feature integration-classifier, a deep learning model is constructed; the deep learning model includes:
[0019] Spatial feature extraction module, which is a fully convolutional network that extracts detailed spatial features of CECT images by using convolutional layers;
[0020] The temporal feature encoding module uses RNN to mine the change patterns between CECTs in different periods;
[0021] The feature integration module fuses the output of the temporal feature encoding module with the encoded clinical data;
[0022] The classifier module uses the output of the feature integration module through a softmax activation function to classify liver malignant tumors.
[0023] The deep learning model takes the preprocessed multi-phase CECT images and the corresponding encoded clinical data as input, and finally outputs the possibility score of each type of liver malignant tumor; the spatial feature extraction module uses the VGG16 convolutional layer pre-trained on ImageNet to extract the spatial features of the CECT image, and uses the trainable convolutional layer with a kernel size of 7×7 to output features with a length of 128 dimensions; then, the temporal feature encoding module uses the RNN to mine the change pattern of liver tumors in different phases of CECT, and uses the GRU to capture the dependencies in the temporal multi-phase images, and constructs an RNN with a GRU with a feature dimension of 32; then, in the feature integration module, the output of the temporal feature encoding module is connected with the encoded clinical data dummy variable to be fused into a 52-dimensional feature; finally, in the classifier module, the output of the feature integration module is activated by the softmax function to obtain the probability value of each category, and the category with the largest probability value is the predicted category, thereby achieving the classification task of liver malignant tumors.
[0024] S5. Model training:
[0025] Dynamic data enhancement is performed on the preprocessed CECT images to ensure the consistency of tumor location in the images; model training is performed based on the training image library and encoded clinical information;
[0026] The specific contents of dynamic data enhancement for the preprocessed CECT images are as follows:
[0027] The preprocessed CECT images were randomly rotated within 10 degrees, translated within 10% up and down, and translated within 10% left and right. The entire image was also randomly scaled within 10%. Multiple CECT phases of the same patient used the same enhancement mode to ensure the consistency of the tumor position in the image and keep the clinical data unchanged.
[0028] S6. Model processing and estimation:
[0029] The preprocessed multi-phase CECECT images and the corresponding encoded clinical data were input into the trained deep learning model to output the likelihood score for each type of liver malignancy, which was evaluated based on accuracy, micro-average AUC, and macro-average AUC.
[0030] This step uses accuracy, sensitivity, specificity, positive predictive value, negative predictive value, F1 score and area under the ROC curve to comprehensively evaluate the output results of the model.
[0031] Furthermore, the 95% confidence intervals of accuracy, sensitivity, specificity, PPV, and NPV were calculated using the Clopper-Pearson method, and the 95% CI of the area under the ROC curve was calculated using the DeLong method.
[0032] The malignant liver tumor classification method based on multimodal data fusion provided by the present invention has at least the following beneficial effects compared with the prior art:
[0033] The present invention adopts multimodal data fusion of structured information and unstructured image information, that is, multimodal data fusion is used to combine image features with clinical data, CNN is used to extract image spatial features, and RNN is used to encode multi-phase image features in time series. On the basis of convolutional neural network, gated recurrent unit (GRU) is used to capture the dependency relationship in multi-phase images in time series, and the classification score of malignant liver tumor is output through feature integration module, which can significantly improve the classification accuracy. Compared with the traditional method that only uses convolutional neural network to extract image spatial features, ignores the temporal relationship between multi-phase images, and does not consider multimodal data fusion, the present invention improves the accuracy of primary liver cancer classification by 25.9%. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 Schematic diagram of the preprocessing process of CT images in the embodiment;
[0035] Figure 2 Schematic diagram of the architecture of the STIC model in the embodiment;
[0036] Figure 3 Schematic diagram of the construction process of the data set in the embodiment;
[0037] Figure 4 The results of the classification performance of the STIC model for primary liver malignancies in the embodiment, wherein sub-figure (a) is a comparison of the classification performance of different models for HCC and ICC on the test set using the ROC curve, sub-figure (b) is the distribution of the prediction scores of HCC and ICC by different models, and sub-figure (c) is the comparison results of the accuracy, sensitivity and specificity of the three models;
[0038] Figure 5: The figure shows the classification performance results of the SITC model for three liver malignancies in the embodiment, wherein sub-figure (a) is the micro-average and macro-average ROC curves of the STIC model for distinguishing HCC, ICC and metastatic liver cancer, sub-figure (b) is the ROC curve of the STIC model for HCC diagnosis and the corresponding diagnostic points of the "doctor-consensus" and three STIC assistant doctors, sub-figure (c) is the ROC curve of the STIC model for ICC diagnosis and the corresponding diagnostic points of the "doctor-consensus" and three STIC assistant doctors, and sub-figure (d) is the ROC curve of the STIC model for metastatic liver cancer diagnosis and the corresponding diagnostic points of the "doctor-consensus" and three STIC assistant doctors. DETAILED DESCRIPTION
[0039] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of the present invention.
[0040] Example
[0041] The present invention relates to a malignant liver tumor classification method based on multimodal data fusion, comprising the following steps:
[0042] Step 1: Dataset preparation:
[0043] Preoperative clinical information and CECT of patients with pathologically confirmed HCC, ICC, and metastatic liver cancer were collected, and the samples were divided into training and test sets. CT images should be scanned using a 64-channel MDCT (multidetector CT; Somatom Definition, Siemens, Forchheim, Germany) scanner with slice thickness (5 mm) and reconstruction interval (1 mm), including the plain scan phase (NC phase), arterial phase (ART phase), and portal venous phase (PV phase).
[0044] Step 2: CT image preprocessing:
[0045] CECT images of the plain, arterial, and portal phases were exported from the Picture Archiving and Communication Systems (PACS), and the 2D slices with the largest tumor area along the z-axis in the 3D CT images were selected as the input of the subsequent model. The CT values of the original DICOM format images ranged from -1024 Hounsfield units (HU) to 3071 HU. In order to focus on the liver part in the abdominal CT scan to reduce the noise from irrelevant organs, the window width (WW) was set to 400 and the window level (WL) was set to 40. Considering that there was a certain positional deviation between CT images of different phases, the plain phase (NC phase) was selected as the reference, and the CT images of other phases were registered using the affine transformation registration algorithm. The mean square error (MSE) was used as the loss function to be optimized, the initial learning rate of the Adam optimizer was set to 0.01, and the number of optimization iterations was set to 100. HU value conversion and multi-phase CT registration were implemented using Python 3.6 software packages Pydicom 2.0.0 and Airlab, respectively. All images were resized to 224 × 224 pixels by linear interpolation using OpenCV-python 3.4.0 to fit the input of the model. Figure 1 A flowchart of CT image preprocessing is shown.
[0046] Step 3: Clinical data coding:
[0047] Data encoding was performed on age, sex, platelets (PLT), total bilirubin (TBIL), alpha-fetoprotein (AFP), carbohydrate antigen 19-9 (CA19-9), carcinoembryonic antigen (CEA), carbohydrate antigen 125 (CA125), and hepatitis B surface antigen (HBsAg) of the samples. Specifically, age was divided into five intervals and then encoded into dummy variables using four binary variables. Sex was represented by a binary variable. The remaining data were the patient's preoperative blood biochemical indicators, which were converted into categorical variables and encoded as dummy variables based on whether they were normal or missing.
[0048] Step 4: Model architecture:
[0049] A deep learning model is constructed using the Spatial Extractor-Temporal Encoder-Integration-Classifier (STIC) modular design idea. Figure 2The detailed architecture of the STIC model is shown. The model consists of four different blocks: The spatial feature extraction module is a fully convolutional network that uses convolutional layers to extract detailed spatial features of CECT images. The temporal feature encoding module uses a recurrent neural network (RNN) to mine the change patterns between different phases of CECT. In the feature integration module, the output of the temporal feature encoding module is fused with the encoded clinical data. Finally, in the classifier module, the output of the feature integration module is activated by a softmax function to achieve the classification task of liver malignancies. Specifically:
[0050] First, the model takes the preprocessed multi-phase CECT images and the corresponding encoded clinical data as input, and finally outputs the probability score of each type of liver malignancy; then, the spatial feature extraction module (SpatialExtractor) uses the VGG16 convolutional layer pre-trained on ImageNet to extract the spatial features of the CECT images, and uses the trainable convolutional layer with a kernel size of 7×7 to output features with a length of 128 dimensions; then, the temporal feature encoding module (TemporalEncoder) uses a recurrent neural network (RNN) to mine the change pattern of liver tumors in different phases of CECT, and uses a gated recurrent unit (GRU) to capture the dependencies in the temporal multi-phase images, and constructs an RNN with a layer of GRU with a feature dimension of 32; then, in the feature integration module (Integration), the output of the temporal feature encoding module is connected with the encoded clinical data dummy variable to fuse into a 52-dimensional feature. Finally, in the classifier module (Classifier), the output of the feature integration module is activated by the softmax function to obtain the probability value of each class. The largest probability value is the predicted class, thus achieving the classification task of liver malignancy.
[0051] Step 5: Model training:
[0052] First, the input image was dynamically enhanced. Each time the image was randomly rotated within 10 degrees, the vertical position was translated within 10%, the horizontal position was translated within 10%, and the entire image was randomly scaled within 10%. The same enhancement mode was used for multiple CECT phases of the same patient to ensure the consistency of the tumor position in the image. The data enhancement strategy was only for the image, and the clinical data remained unchanged.
[0053] Secondly, the STIC model and the two baseline models were trained for 50 epochs using the cross entropy loss function. Considering the training speed and computer memory, the mini-batch size was set to 32. In order to avoid overfitting, early stopping was performed based on the performance of the validation set, that is, when the loss function of the validation set did not decrease within 20 epochs, the training process was interrupted. The initial learning rate of the Adam optimizer was set to 0.001. If the validation set loss function did not decrease for 10 epochs, the learning rate was reduced to 0.8 times the previous one.
[0054] Step 6: Model evaluation:
[0055] The accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1 score and area under the ROC curve (AUC) were used to comprehensively evaluate the diagnostic performance of the model. Micro-average AUC (considering the imbalance between groups) and macro-average AUC (not considering the imbalance between groups) were calculated to evaluate the performance of the multinomial classification model. The 95% confidence intervals (CI) of accuracy, sensitivity, specificity, PPV and NPV were calculated using the Clopper-Pearson method. The 95% CI of AUC was calculated using the DeLong method.
[0056] To prove the high accuracy of the method proposed in the present invention, this example is experimentally verified using actual malignant liver tumor patient data:
[0057] 1) Dataset: The data comes from 612 patients diagnosed from August 2012 to September 2020. All included patients were pathologically confirmed to have one of the following malignant liver tumors: HCC, ICC, or metastatic liver cancer. The corresponding CECT images and clinical data are preoperative information. The training set includes 499 patients diagnosed from August 2012 to September 2019, and the test set includes 113 patients diagnosed from October 2019 to September 2020. This example randomly selects 20% of the training set as the validation set to guide the model training process. Figure 3 The process of dataset construction is shown.
[0058] 2) Benchmark model: In order to compare the performance with the existing technology, this embodiment establishes two benchmark models, namely the benchmark imaging model and the benchmark joint model. The benchmark imaging model puts the CECT of three stages (NC stage, ART stage and PV stage) into the red, green and blue channels respectively, and uses the VGG16 convolution layer preprocessed on ImageNet to extract features, followed by three fully connected layers (with dimensions of 512, 128 and 32 respectively), and finally uses the softmax activation function to implement the classification task. The benchmark joint model uses the same strategy as the benchmark imaging model to process multi-stage images, transforms the structure of the fully connected layer of the neural network, and integrates clinical data.
[0059] 3) Expert evaluation: Under the premise of unknown pathological diagnosis results, two radiologists with more than five years of experience reached a consensus diagnosis based on the multi-phase CECT and clinical data of the test set samples. Their diagnosis results were named "doctor consensus" and compared with the results of the STIC model.
[0060] In order to evaluate the performance of model-assisted diagnosis, this embodiment also conducted a study on doctors' diagnosis assisted by STIC. Under the premise of unknown pathological results, three professional radiologists with more than five years of experience independently made the final diagnosis based on multi-phase CECT, clinical data and STIC model prediction scores.
[0061] 4) Statistical analysis: For the collected data, three models were used for training, and then their accuracy was compared: To compare the STIC model with the two benchmark models, the present embodiment uses the McNemar chi-square test with continuity correction to calculate the p-values of the accuracy, sensitivity, and specificity differences of the two models. The Wald Z method was used with continuity correction to calculate the 95% confidence interval (CI) of the accuracy, sensitivity, and specificity differences. To compare the diagnostic levels of the three doctors assisted by the STIC model, the Cochran Q test was used to calculate the p-values of the diagnostic accuracy, sensitivity, and specificity differences among multiple doctors. All statistical analyses were performed using R software.
[0062] 5) Results:
[0063] 5.1 Differentiation of primary liver malignancies
[0064] This example first trained the STIC model to distinguish between two primary liver malignancies (HCC and ICC), and then compared it with the baseline model. Table 1 summarizes the performance of the STIC model and the two baseline models on the test set, where sensitivity and specificity are defined by considering HCC as positive and ICC as negative. Among the three models, the STIC model performed best in distinguishing between the two primary liver malignancies, with an accuracy of 86.2% (95% CI, 74.6%-93.9%), a sensitivity of 0.892 (95% CI, 0.746-0.970), a specificity of 0.810 (95% CI, 0.581-0.946), and an F1 score of 0.892. The receiver operating characteristic (ROC) curves of the three models are shown in Figure 1. Figure 4 The AUC of the STIC model is 0.893 (95% CI, 0.803-0.982), which is much higher than the 0.709 (95% CI, 0.573-0.845) of the baseline imaging model and the 0.766 (95% CI, 0.644-0.888) of the baseline joint model. The distribution of the prediction scores of HCC and ICC by the three models is shown in Figure 2. Figure 4 As shown in sub-figure b in . The distribution of the prediction scores of the two baseline models is wider, making the prediction results unstable. The scores output by the STIC model proposed in the present invention are more concentrated in HCC and ICC, showing its excellent performance in this binary classification task. Using the McNemar chi-square test, the STIC model outperformed the baseline imaging model, with an accuracy increase of 25.9% (95% CI 11.0%-40.7%, p-value=0.001) and a sensitivity increase of 0.270 (95% CI 0.082-0.459, p-value=0.009). At the same time, the STIC model also outperformed the baseline joint model, with an accuracy increase of 17.2% (95% CI 3.7%-30.8%, p-value=0.016) and a sensitivity increase of 0.189 (95% CI0.015-0.363, p-value=0.046) ( Figure 4 ).
[0065] Table 1: Performance of the STIC model and two baseline models for identifying primary liver malignancies
[0066]
[0067] 5.2 Differential diagnosis of three types of liver malignancies
[0068] On the basis of the above, the STIC model was extended to the differential diagnosis of three types of liver malignancies. As shown in Table 2, the diagnostic accuracy of the STIC model and "doctor-consensus" for each type of tumor was analyzed in detail. The overall accuracy of the STIC model was 72.6% (95% CI, 63.4%-80.5%), and its AUC values for diagnosing HCC, ICC, and metastatic liver cancer were 0.937 (95% CI, 0.892-0.982), 0.727 (95% CI, 0.599-0.855), and 0.878 (95% CI, 0.816-0.940), respectively. The micro-average and macro-average ROC curves of the STIC model are shown in Table 2. Figure 5 As shown in the neutron graph (a), the micro-average AUC is 0.868 and the macro-average AUC is 0.852. The overall accuracy of the "doctor-consensus" is 70.8% (95% CI, 61.5%-79.0%). There is no significant difference between the STIC model and the "doctor-consensus" in the sensitivity and specificity of diagnosing each type of tumor (Table 2), which indicates that the STIC model proposed in the present invention performs comparable to that of professional radiologists.
[0069] Table 2. Comparison of the diagnostic performance of liver malignancies on the test set by the STIC model, “doctor-consensus”, and three doctors assisted by the STIC model
[0070]
[0071]
[0072] 5.3 Performance of STIC-assisted diagnosis
[0073] This example evaluates the diagnostic performance of doctors assisted by the STIC model. The total accuracy of the three doctors assisted by the STIC model on the test set was 77.0% (95% CI, 68.1%-84.4%), 78.8% (95% CI, 70.1%-85.9%), and 81.4% (95% CI, 73.0%-88.1%), respectively. Table 2 shows their sensitivity, specificity, PPV, NPV, and F1 score for diagnosing HCC, ICC, and metastatic liver cancer. The ROC curve for HCC diagnosis shows that the performance of the "doctor-consensus" and the three STIC-assisted doctors is basically the same as the diagnostic level of the STIC model ( Figure 5 Neutron graph (b). For ICC diagnosis, the performance of the "doctor-consensus" diagnosis was lower than the ROC curve of the STIC model, while the diagnostic performance of the three STIC-assisted doctors was higher than the model ROC curve ( Figure 5 Neutron image (c). For the diagnosis of metastatic liver cancer, the performance of "doctor-consensus" was lower than the ROC curve, while the performance of the three STIC-assisted doctors was higher than the ROC curve ( Figure 5Neutron graph (d). When comparing the diagnostic levels of the three STIC-assisted doctors and the "doctor-consensus", there was a significant difference in their sensitivity in diagnosing ICC (p-value = 0.038). The sensitivity of "doctor-consensus" in diagnosing ICC was 0.429 (95% CI, 0.218-0.660), and the average sensitivity of the three STIC-assisted doctors in diagnosing ICC was 0.698. The confusion matrix on the test set also showed that the number of correctly predicted ICC samples was higher in the doctor's diagnosis results assisted by STIC.
[0074] The malignant liver tumor classification method proposed in the present invention effectively utilizes CECT images and clinical data to distinguish HCC, ICC and metastatic liver cancer. With the assistance of the STIC model, the classification accuracy can be significantly improved, especially the sensitivity of ICC recognition is greatly improved.
[0075] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A malignant liver tumor classification method based on multimodal data fusion, characterized in that: The following steps are involved: 1) Dataset acquisition: Collect preoperative clinical information and CECT images of patients with pathologically confirmed HCC, ICC, and metastatic liver cancer, and divide the collected sample data into training set samples and test set samples; 2) CECT image preprocessing: Export CECT images including plain scan phase, arterial phase and portal venous phase from the image archiving and communication system, and pre-process the CECT images to form a training image library; 3) Clinical data coding: Data were coded for age, sex, platelets, total bilirubin, alpha-fetoprotein, carbohydrate antigen 19-9, carcinoembryonic antigen, carbohydrate antigen 125, and hepatitis B surface antigen of the samples; 4) Model Architecture: Based on the modular design concept of spatial feature extraction-temporal feature encoding-feature integration-classifier, a deep learning model is constructed; 5) Model training: Dynamic data enhancement is performed on the preprocessed CECT images to ensure the consistency of tumor location in the images; model training is performed based on the training image library and encoded clinical information; 6) Model processing and estimation: The preprocessed multi-phase CECECT images and the corresponding encoded clinical data were input into the trained deep learning model to output the likelihood score for each type of liver malignancy, and the output results were evaluated based on accuracy, micro-average AUC, and macro-average AUC; The deep learning model includes: Spatial feature extraction module, which is a fully convolutional network that extracts detailed spatial features of CECT images by using convolutional layers; The temporal feature encoding module uses RNN to mine the change patterns between CECTs in different periods; The feature integration module fuses the output of the temporal feature encoding module with the encoded clinical data; The classifier module uses the output of the feature integration module through a softmax activation function to classify liver malignant tumors; The deep learning model takes the preprocessed multi-phase CECT images and the corresponding encoded clinical data as input, and finally outputs the possibility score of each type of liver malignant tumor; the spatial feature extraction module uses the VGG16 convolutional layer pre-trained on ImageNet to extract the spatial features of the CECT image, and uses the trainable convolutional layer with a kernel size of 7×7 to output features with a length of 128 dimensions; then, the temporal feature encoding module uses the RNN to mine the change pattern of liver tumors in different phases of CECT, and uses the GRU to capture the dependencies in the temporal multi-phase images, and constructs an RNN with a GRU with a feature dimension of 32; then, in the feature integration module, the output of the temporal feature encoding module is connected with the encoded clinical data dummy variable to be fused into a 52-dimensional feature; finally, in the classifier module, the output of the feature integration module is activated by the softmax function to obtain the probability value of each category, and the category with the largest probability value is the predicted category, thereby achieving the classification task of liver malignant tumors.
2. The malignant liver tumor classification method based on multimodal data fusion according to claim 1, characterized in that: In step 1), the CECT image is scanned by a 64-channel MDCT scanner, with a slice thickness of 5 mm and a reconstruction interval of 1 mm. Each image includes a plain scan phase, an arterial phase, and a portal venous phase.
3. The malignant liver tumor classification method based on multimodal data fusion according to claim 1, characterized in that: In step 2), the 2D slice with the largest tumor area along the z-axis in the 3D CECT image was selected as the input of the subsequent model; the HU value range of the CECT image in the original DICOM format image was limited to a window width of 400 and a window level of 40 to reduce the noise from irrelevant organs.
4. The malignant liver tumor classification method based on multimodal data fusion according to claim 3, characterized in that: In step 2), the plain scan period was selected as the reference, and the CECT images of other periods were registered using the affine registration algorithm, and all images were resized to 224×224 pixels by linear interpolation to fit the input of the model.
5. The malignant liver tumor classification method based on multimodal data fusion according to claim 1, characterized in that: The specific contents of step 3) are: The age of the samples was divided into five intervals and then encoded into dummy variables using four binary variables; the gender of the samples was represented by a binary variable; platelet, total bilirubin, alpha-fetoprotein, carbohydrate antigen 19-9, carcinoembryonic antigen, carbohydrate antigen 125, and hepatitis B surface antigen data were used as preoperative blood biochemical indicators of patients and converted into categorical variables and encoded as dummy variables based on whether they were normal or missing.
6. The malignant liver tumor classification method based on multimodal data fusion according to claim 1, characterized in that: In step 5), the specific contents of dynamic data enhancement of the CECT image after preprocessing are as follows: After preprocessing, the CECT images were randomly rotated within 10 degrees, translated within 10% up and down, and translated within 10% left and right. The entire image was also randomly scaled within 10%. Multiple CECT phases of the same patient used the same enhancement mode to ensure the consistency of the tumor position in the image and keep the clinical data unchanged.
7. The malignant liver tumor classification method based on multimodal data fusion according to claim 1, characterized in that: In step 6), the output of the model was comprehensively evaluated using accuracy, sensitivity, specificity, positive predictive value, negative predictive value, F1 score, and area under the ROC curve.
8. The method for classifying malignant liver tumors based on multimodal data fusion according to claim 7, characterized in that: In step 6), the 95% confidence intervals of accuracy, sensitivity, specificity, PPV, and NPV were calculated using the Clopper-Pearson method, and the 95% CI of the area under the ROC curve was calculated using the DeLong method.
Citation Information
Patent Citations
Liver tumor segmentation method and device based on multi-stage CT image guidance
CN109961443A
Liver cancer CT image benign and malignant classification method based on deep learning
CN110443268A
Cancer targeted therapy curative effect evaluation method and device, equipment and storage medium
CN110717894A
Automatic liver tumor classification method and device based on physiological indexes and image fusion
CN110910371A
Automatic liver tumor classification method and device based on multi-stage CT image analysis
CN110929789A
