Multimodal image classification method based on evidence-based deep learning

Abstract

The present invention proposes a multimodal image classification method, device and computer-readable storage medium based on evidence deep learning. By making a decision-making fusion of medical images from multiple different modalities, the complementarity of different medical images in describing lesions is fully utilized, thereby obtaining a method that is more reliable and more accurate than relying solely on one modality to obtain lesion category attribute information. We used the BraTS2017 dataset for verification, and compared with traditional feature fusion and classification methods, the multimodal image classification method based on evidence deep learning has better results.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and analysis, and specifically relates to a multimodal image classification method, device and computer-readable storage medium based on evidence deep learning. Background Art

[0002] With the continuous upgrading and widespread popularization of medical equipment, multimodal medical imaging technology has developed rapidly, providing valuable multimodal data for disease research. Since the medical diagnostic system is decision-sensitive and the problems in actual application scenarios are relatively complex, single-modal images are often not comprehensive enough to fully subdivide tumors in related areas. Multimodal imaging can capture more comprehensive pathological information and characteristic manifestations from a variety of different characteristics. By using computer technology to reasonably integrate different types of images and make each modality information complementary, more accurate and reliable diagnostic decisions can be made.

[0003] In recent years, the molecular pathology of central nervous system tumors has made significant progress, and a series of molecular markers that are helpful for the clinical diagnosis and prognosis of brain gliomas have been discovered. In 2016, the World Health Organization (WHO) added the classification of molecular markers for brain tumors for the first time. Different molecular subtypes differ in the tumor's predilection location and treatment sensitivity. The diagnosis and prognosis evaluation of gliomas by combining molecular markers can select corresponding clinical treatment plans, which has become an important milestone in the progress of glioma diagnosis and treatment.

[0004] With the development of computer technology, it has become a major trend to apply machine learning methods to survival prediction and glioma molecular marker detection and classification. Traditional multimodal classification methods usually extract features from images of each modality separately, then concatenate or weightedly add the features to form a new feature vector, and finally send the fused features to the decision model to obtain the final decision result. However, the modalities are often highly correlated, and the early fusion of multimodal data cannot fully demonstrate the complementarity between the modalities and may lead to the input of redundant vectors. Summary of the invention

[0005] Traditional multimodal classification methods usually concatenate or weightedly add the features of each modality, and then use a classifier to obtain the final classification result, ignoring the complementary information between the modalities. To address this problem, the present invention proposes a multimodal image classification method based on evidence-based deep learning, which makes full use of the complementarity of different medical images in describing lesions, thereby obtaining a classification method that is more reliable and accurate than relying solely on one modality for lesion category attribute information.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A multimodal image classification method based on evidence-based deep learning is proposed. After the features proposed by each modality are integrated, an improved cross entropy function is used for optimization training to obtain the final classification result. The specific steps include:

[0008] (I) Data preprocessing: First, label each classification object, and then extract the region of interest from the four modalities of T1, T2, T1Gd, and Flair of the magnetic resonance image;

[0009] (ii) Data augmentation: Using the MONAI framework, data augmentation was performed on each modality and tumor segmentation marker for each patient;

[0010] (III) Feature extraction: Using the tumor marker as a mask, the Pyradiomics toolkit was used to extract features from the four modalities of T1, T2, T1Gd, and Flair, and the features were normalized.

[0011]

[0012] X is the input data, μ is the mean of the input data, σ is the variance of the input data, and X′ is the data after standardization;

[0013] (IV) Multimodal fusion framework: The features extracted from each modality are used as input, and the deep neural network DNN is used as the model to output the evidence vector e. The evidence e is connected to the parameter α of the Dirichlet distribution through subjective logic, and the confidence quality b is calculated. k and uncertainty u as subjective opinions, and finally, based on the Dempster-Shafer evidence synthesis rule, the obtained modal opinions are fused in turn to obtain the final fusion opinion;

[0014] (V) Optimize training: Adjust the traditional cross entropy function to the form of integration over the Dirichlet distribution, incorporate the KL divergence term into the loss function, and normalize the prediction distribution by penalizing evidence generated by incorrect labels. Finally, add the losses of each modality and the loss after modality fusion to obtain the final loss function:

[0015]

[0016] is the loss of each mode, is the loss after modal fusion, N is the number of input samples, and V is the number of input modalities;

[0017] (VI) Classification result prediction: Finally, the loss value obtained through the forward propagation of the neural network is used to measure the degree of match between the network prediction value and the expected result. The loss function is continuously minimized, and the network weights are updated through the Adam optimizer to obtain the classification result.

[0018] In the training of the model, it is necessary to shuffle the multimodal data in order, divide all the data into k parts equally, take one of them as the validation set, and use the remaining k-1 parts as the training set to perform k-fold cross-validation to obtain the final result.

[0019] The present invention also provides an image processing device, which includes a data acquisition component, a memory and a processor, wherein the data preprocessing component can implement the above steps (1) to (3); the processor can be used to execute the above steps (4) to (6); the memory part is used to store the computer programs used in all steps, the processed data, and can store the final training model.

[0020] The present invention also includes a computer-readable storage medium on which a computer program is stored, characterized in that the program implements the steps of the above method when executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the multimodal image classification method based on evidence-based deep learning of the present invention.

[0022] Figure 2 It is a schematic diagram of the multimodal fusion framework of the present invention.

[0023] Figure 3 It is a schematic block diagram of the image processing device of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, the schematic diagram of the multimodal image classification method based on evidence deep learning of the present invention is as follows:

[0026] (I) Data preprocessing: First, mark each classification object with a molecular marker label. For example, if molecular marker classification is performed, the following markings are used: glioma with IDH mutation and 1p / 19q co-deletion is marked as 0, its IDH mutation and 1p / 19q non-combined deletion is marked as 1, and its IDH wild type is marked as 3. Secondly, extract the region of interest. For example, for magnetic resonance images, the tumor segmentation markers of the data set can be corresponded to the four modalities and taken out as the region of interest.

[0027] (II) Data enhancement: Multimodal data enhancement on the training set is performed based on the MONAI framework. Using the MONAI framework, data expansion is performed on each modality and tumor segmentation marker of each patient. Specific enhancement methods include: random affine transformation, random elastic deformation, adding Gaussian noise, flipping, and rotation. By increasing the amount of data in the training set, the generalization ability of the model is improved; and by adding noise data, the robustness of the model is improved.

[0028] (III) Feature extraction: The tumor marker part is used as a mask, and the Pyradiomics toolkit is used to extract features of the four modes of T1, T2, T1Gd, and Flair, and then the features are normalized.

[0029] (IV) Multimodal fusion framework: Figure 2 As shown in the figure, the features extracted from each modality are first taken as input, and the deep neural network DNN is used as the model to output the evidence vector e = [e1, e2, …, e K ], where the softmax function is replaced by an activation function to avoid exaggerating the probability of the predicted category. Then, the evidence e=[e1,e2,…,e K ] is connected to the parameters of the Dirichlet distribution α = [α1, α2, …, α K ],Right now

[0030] α k =e k +1

[0031] The subjective opinion The confidence quality in b k and the uncertainty u is calculated as:

[0032]

[0033]

[0034] in Dirichlet strength. Finally, based on the Dempster-Shafer evidence synthesis rule, each modal opinion is fused in turn, θ is the normalization coefficient, and the fused opinion is obtained. The synthesis rule is as follows:

[0035]

[0036]

[0037] (V) Optimize training: Adjust the traditional cross entropy function to the form of integration over the Dirichlet distribution, incorporate the KL divergence term into the loss function, and normalize the predicted distribution by penalizing evidence generated by incorrect labels. The loss under single mode is:

[0038]

[0039] Where S K is the K-dimensional unit simplex, B(α) is the K-dimensional polynomial beta function, λ t =min(1.0,t / 10)∈[0,1] is the annealing coefficient, D(p i |<1,…,1>) is a uniform Dirichlet distribution, and is the prediction parameter α from sample i i The Dirichlet parameter after removing non-misleading evidence. Finally, the loss of each modality and the loss after modal fusion are added together to obtain the final loss function:

[0040]

[0041] is the loss of each mode, is the loss after modal fusion, N is the number of input samples, and V is the number of input modalities;

[0042] (VI) Classification: Finally, the loss value obtained through the forward propagation of the neural network is used to measure the degree of match between the network prediction value and the expected result. The loss function is continuously minimized, and the network weights are updated through the Adam optimizer to obtain the classification result.

[0043] Figure 3 A schematic block diagram of the image processing device of the present invention is given, which mainly includes three components: a data preprocessing component, a processor and a memory, wherein the data preprocessing component can implement the above steps (1) to (3); the processor can be used to execute the above steps (4) to (6); the memory part is used to store the computer programs used in all steps, the processed data, and can store the final training model.

[0044] Compared with the traditional method, the present invention has higher classification accuracy and better generalization ability.

[0045] The effect of the present invention can be further illustrated by the classification results:

[0046] The experimental data sets are BraTS2017 and TCGA data sets, with a total of 167 multimodal data of objects. During the data preprocessing process of model training, the data are randomly shuffled and the data sets are divided into 5 parts, 4 of which are selected as training sets, and the remaining 1 is selected as a validation set. Feature extraction is performed on the region of interest of the data, and the proposed multimodal fusion method based on evidence deep learning is used for classification. Table 1 is a comparison of the accuracy, AUC value and overall standard deviation after classification with traditional machine learning classification methods SVM, KNN, RF and fully connected neural network (FCNN). In summary, the segmentation effect of the present invention is better than that of the classical method.

[0047] Table 1

[0048] Experimental methods Accuracy ACC AUC KNN 64.02％±7.370％ 0.7827±0.05799 RF 71.19％±8.869％ 0.8716±0.05068 SVM 75.97％±5.979％ 0.8664±0.05141 FCNN 74.75％±6.491％ 0.8544±0.04797 Ours 76.47％±9.628％ 0.9209±0.02968

Claims

1. A multimodal image classification method based on evidence-based deep learning, which fuses the features proposed by each modality and improves the cross entropy function to obtain the final classification result. The specific steps include: (I) Data preprocessing: First, label each classification object, and then extract the region of interest from the four modalities of T1, T2, T1Gd, and Flair of the magnetic resonance image; (ii) Data augmentation: Using the MONAI framework, data augmentation was performed on each modality and tumor segmentation marker for each patient; (III) Feature extraction: Using the tumor marker as a mask, the Pyradiomics toolkit was used to extract features from the four modalities of T1, T2, T1Gd, and Flair, and the features were normalized. X is the input data, μ is the mean of the input data, σ is the variance of the input data, and X′ is the data after standardization; (IV) Multimodal fusion framework: The features extracted from each modality are used as input, and the deep neural network DNN is used as the model to output the evidence vector e. The evidence e is connected to the parameter α of the Dirichlet distribution through subjective logic, and the confidence quality b is calculated. k and uncertainty u as subjective opinions, and finally, based on the Dempster-Shafer evidence synthesis rule, the obtained modal opinions are fused in turn to obtain the final fusion opinion; (V) Optimize training: Adjust the traditional cross entropy function to the form of integration over the Dirichlet distribution, incorporate the KL divergence term into the loss function, and normalize the prediction distribution by penalizing evidence generated by incorrect labels. Finally, add the losses of each modality and the loss after modality fusion to obtain the final loss function: is the loss of each mode, is the loss after modal fusion, N is the number of input samples, and V is the number of input modalities; (VI) Classification result prediction: Finally, the loss value obtained through the forward propagation of the neural network is used to measure the degree of match between the network prediction value and the expected result. The loss function is continuously minimized, and the network weights are updated through the Adam optimizer to obtain the classification result.

2. An image processing device, comprising a data preprocessing component, a memory and a processor, wherein the data preprocessing component can implement steps (1) to (3) described in claim 1; the processor can be used to execute steps (4) to (6) described in claim 1; the memory part is used to store the computer program used in each step described in claim 1, the processed data, and can store the final training model.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the steps of the method according to claim 1 are implemented.

Images