Brain tumor diagnosis method and system based on multimodal supervised contrastive learning

Abstract

The present invention discloses a brain tumor diagnosis method and system based on multimodal supervised contrast learning, which relates to the field of artificial intelligence technology. Patient MRI images are input into a trained brain tumor diagnosis model to output a brain tumor diagnosis result. The training process of the brain tumor diagnosis model is as follows: the patient MRI images are acquired and preprocessed to construct an original MRI image set, and MRI images of different modes of the same patient in the original MRI image set are fused to obtain a fused MRI image set; the original MRI image set and the fused MRI image set are combined to obtain a combined image set, and image diagnosis features are extracted from the combined image set based on an image encoder; cross entropy loss and multimodal supervised contrast loss are constructed, and supervised learning is performed on the image encoder to train the brain tumor diagnosis model. The brain tumor diagnosis method and system realize high-precision brain tumor diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a brain tumor diagnosis method and system based on multimodal supervised contrast learning. Background Art

[0002] Brain tumor is one of the major diseases that seriously threaten human life and health. Its early diagnosis and accurate classification are crucial for developing effective treatment plans and improving patient prognosis. Currently, magnetic resonance imaging (MRI) has become the main technical means for the detection and diagnosis of brain tumors. MRI can provide rich anatomical and pathological information, including tumor location, morphology and tissue characteristics.

[0003] However, due to the heterogeneity of brain tumors and the overlap of features between different categories, traditional imaging analysis methods alone are difficult to meet the needs of clinical accurate diagnosis. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a brain tumor diagnosis method and system based on multimodal supervised contrast learning to achieve high-precision brain tumor diagnosis.

[0005] The brain tumor diagnosis method based on multimodal supervised contrastive learning proposed in the present invention inputs the patient's MRI image into a trained brain tumor diagnosis model to output the brain tumor diagnosis result;

[0006] The training process of the brain tumor diagnosis model is as follows:

[0007] Acquire and preprocess the patient's MRI images to construct an original MRI image set, and fuse the MRI images of the same patient in different modalities in the original MRI image set to obtain a fused MRI image set;

[0008] The original MRI image set and the fused MRI image set are combined to obtain a combined image set, and image diagnostic features are extracted from the combined image set based on an image encoder;

[0009] The cross entropy loss and multimodal supervised contrastive loss are constructed to perform supervised learning on the image encoder to train the brain tumor diagnosis model.

[0010] Furthermore, a brain tumor label is marked for each patient in the original MRI image set and the fused MRI image set.

[0011] Furthermore, in generating the fused MRI image set, MRI images of different modalities of the same patient are fused pairwise.

[0012] Furthermore, the construction process of the cross entropy loss is as follows:

[0013] Combining all extracted imaging diagnostic features into a diagnostic feature set;

[0014] The diagnostic feature set is passed through a layer of fully connected neural network to obtain a predicted category set, and the cross entropy loss between the predicted category set and the true category set is calculated.

[0015] Furthermore, the construction process of the multimodal supervised contrast loss is as follows:

[0016] If two samples in the combined image set have the same category label, they are defined as a positive sample pair; if two samples in the combined image set have different category labels, they are defined as a negative sample pair;

[0017] Construct a label matrix based on positive sample pairs and negative sample pairs;

[0018] Calculate the similarity between samples in the combined image set and construct a similarity matrix;

[0019] Based on the label matrix and the similarity matrix, the log-likelihood function of each sample in the combined image set is calculated to maximize the similarity with the positive sample pair and minimize the similarity with its negative sample pair;

[0020] Compute the multimodal supervised contrastive loss based on the log-likelihood function.

[0021] Furthermore, the positive sample pair is defined as 1, the negative sample pair is defined as 0, and the label matrix is ​​a symmetric matrix.

[0022] Furthermore, the calculation formula of the log-likelihood function is as follows:

[0023] ;

[0024] in, For the combined image set The log-likelihood function of samples is is the number of patients, represents the first Sample and The relationship between samples, if it is a positive sample pair, then Take 1, if it is a negative sample pair, then Take 0, For the combined image set Samples and The similarity between samples, For the combined image set Samples and The similarity between samples.

[0025] 8. The brain tumor diagnosis method based on multimodal supervised contrastive learning according to claim 5 is characterized in that the multimodal supervised contrastive loss The formula is as follows:

[0026] ;

[0027] in, For the The log-likelihood function of samples is is the number of patients.

[0028] The brain tumor diagnosis system based on multimodal supervised contrastive learning inputs the patient's MRI images into the trained brain tumor diagnosis model to output the brain tumor diagnosis results;

[0029] The training process of the brain tumor diagnosis model includes image acquisition module, modality fusion module, image combination module, image extraction module and loss construction module;

[0030] The image acquisition module is used to acquire and preprocess the patient's MRI images to construct an original MRI image set;

[0031] The modality fusion module is used to fuse MRI images of different modalities of the same patient in the original MRI image set to obtain a fused MRI image set;

[0032] The image combination module is used to combine the original MRI image set and the fused MRI image set to obtain a combined image set;

[0033] The image extraction module extracts image diagnostic features from the combined image set based on the image encoder;

[0034] The loss building module is used to construct cross entropy loss and multimodal supervised contrastive loss to perform supervised learning on the image encoder to train the brain tumor diagnosis model.

[0035] Further, a brain tumor label is marked for each patient in the original MRI image set and the fused MRI image set;

[0036] The construction process of multimodal supervised contrast loss is as follows:

[0037] If two samples in the combined image set have the same category label, they are defined as a positive sample pair; if two samples in the combined image set have different category labels, they are defined as a negative sample pair;

[0038] Construct a label matrix based on positive sample pairs and negative sample pairs;

[0039] Calculate the similarity between samples in the combined image set and construct a similarity matrix;

[0040] Based on the label matrix and the similarity matrix, the log-likelihood function of each sample in the combined image set is calculated to maximize the similarity with the positive sample pair and minimize the similarity with its negative sample pair;

[0041] Compute the multimodal supervised contrastive loss based on the log-likelihood function.

[0042] The advantages of the brain tumor diagnosis method and system based on multimodal supervised contrast learning provided by the present invention are that it aims to significantly improve the classification accuracy through two key components while maintaining the generalization ability. The two key components are the modality fusion module and the multimodal supervised contrast loss. First, the modality fusion module integrates complementary MRI imaging modalities to generate information-rich fusion samples. The multimodal supervised contrast loss is designed by verifying whether the predictions of the original MRI image set and the fused MRI image set are consistent. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of the present invention;

[0044] Figure 2 Schematic diagram of the training process of the brain tumor diagnosis model. DETAILED DESCRIPTION

[0045] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0046] like Figure 1 and 2 As shown, the brain tumor diagnosis method based on multimodal supervised contrast learning proposed by the present invention inputs the patient's MRI image into a trained brain tumor diagnosis model to output the brain tumor diagnosis result.

[0047] The key point of this embodiment is the training process of the brain tumor diagnosis model, which is as follows:

[0048] Step 1: Acquire and preprocess the patient's MRI images to construct an original MRI image set, and fuse the MRI images of different modalities of the same patient in the original MRI image set to obtain a fused MRI image set;

[0049] Obtain the patient's MRI images and perform preprocessing, including but not limited to cropping, standardization, etc., to obtain the original MRI image set: ,in is the number of patients, It is The MRI images of patients have a height and width of H and W, and the number of modalities is M. The label set is ,in It is Brain tumor signatures for patients Represents the number of tumor types.

[0050] Perform image fusion on each patient's MRI image in the original MRI image set: that is, prepare an existing pre-trained MRI image fusion network as ,use The MRI images of different modalities of the same patient were fused pairwise to obtain a fused MRI image set. , For the The MRI images of the patients are fused through modality. The fusion process is shown as follows:

[0051] ;

[0052] in, Represents the Patient / patient's modality MRI images, Represents the Patient / patient's modality MRI images, yes and Pre-trained MRI image fusion network Fused images, fused MRI image set The corresponding label is ,in and Exactly the same, because modality fusion does not change the tumor category corresponding to the MRI image.

[0053] Step 2: Combining the original MRI image set and the fused MRI image set to obtain a combined image set, and extracting image diagnostic features from the combined image set based on an image encoder;

[0054] The original MRI image set is combined with the fused MRI image set, and the combined image set is The original MRI label set is then combined with the fused MRI label set, and the combined label is .

[0055] Using Image Encoder Extracting diagnostic imaging features: Feed into encoder In, it is expressed as follows:

[0056] ;

[0057] in represents a set of diagnostic features, Dimensions representing diagnostic features of each patient, The diagnostic feature set imaging diagnostic features.

[0058] Step 3: Construct cross entropy loss and multimodal supervised contrast loss to supervise the image encoder to train the brain tumor diagnosis model;

[0059] The diagnostic feature set obtained in step 2 is passed through a fully connected neural network to obtain a predicted category set, and the cross entropy loss between the predicted category set and the true category set is calculated. .

[0060] The multimodal supervised contrast loss is calculated The details are as follows:

[0061] (1) Construct a set of positive and negative sample pairs;

[0062] For combined image sets As long as the two MRI images are of the same category, they are defined as a positive sample pair; if the two MRI images are of different categories, they are defined as a negative sample pair, which is expressed as follows:

[0063] ;

[0064] ;

[0065] in, It is a set of positive sample pairs, consisting of MRI images belonging to the same category. is a set of negative sample pairs, consisting of MRI images that do not belong to a category; and Respectively imaging diagnostic features and imaging diagnostic features, and Respectively The label corresponding to the image diagnostic feature and the The labels corresponding to the image diagnostic features are , since the image diagnostic features are extracted from the samples in the combined image set by the image encoder, the After the sample passes through the image encoder, it corresponds to For the convenience of description and correspondence, this embodiment uses the same index for samples and image diagnosis features (i.e., the same express).

[0066] It is understandable that if Figure 1As shown in the figure, the fused MRI images and non-fused MRI images of the same category are regarded as positive samples, and the pull-closer operation is performed on the positive samples, that is, the features are pushed forward. The fused MRI images and non-fused MRI images of different categories are regarded as negative samples, and the push-out operation is performed, that is, the features are distinguished. Taking choroid plexus tumor (Choroid PlexusTumor) as an example, this method first fuses its MRI images of different modalities (such as T1w, T2w, T1ce), generates a fused MRI image, and extracts its deep features. In the feature space, the fused MRI and non-fused MRI of the same category are regarded as positive samples, such as the T1ce-T1w fused image of the same patient and its T1w original image, and the pull-closer operation is used to make their features consistent. For MRIs of different categories (such as ependymoma and glioma) regarded as negative samples, such as the fused image of choroid plexus tumor and the T1w-T2w fused image of ependymoma, the category distinction is increased by the push-out operation.

[0067] (2) Construct a label matrix based on positive sample pairs and negative sample pairs :

[0068] ;

[0069] in, express and The relationship between two samples, if it is a positive sample pair, it is 1, and if it is a negative sample pair, it is 0; label matrix is a symmetric matrix.

[0070] (3) Calculate the similarity between samples in the combined image set and construct a similarity matrix;

[0071] ;

[0072] in, For the Samples and The similarity between samples is measured by the temperature parameter Scale to control the sharpness of the distribution.

[0073] (4) Based on the label matrix and the similarity matrix, the log-likelihood function of each sample in the combined image set is calculated to maximize the similarity with the positive sample pair and minimize the similarity with its negative sample pair; the formula is as follows:

[0074] ;

[0075] in, For the combined image set The log-likelihood function of samples is is the number of patients, represents the first Sample and The relationship between samples, if it is a positive sample pair, then Take 1, if it is a negative sample pair, then Take 0, For the combined image set Samples and The similarity between samples, For the combined image set Samples and The similarity between samples.

[0076] (5) Calculate multimodal supervised contrast loss based on log-likelihood function ;

[0077] ;

[0078] Training a brain tumor diagnosis model using gradient descent and calculating the multimodal supervised contrastive loss and cross entropy loss To update the network parameters, the training stops when the number of training iterations reaches the set number.

[0079] According to steps one to three, the brain tumor diagnosis method proposed in this embodiment aims to significantly improve the classification accuracy while maintaining generalization ability through two key components. The two key components are the modality fusion module and the multimodal supervised contrast loss. First, the modality fusion module (multimodal image fusion) integrates complementary MRI imaging modalities to generate information-rich fusion samples. The multimodal supervised contrast loss is designed by verifying whether the predictions of the original MRI image set and the fused MRI image set are consistent. Compared with other existing brain tumor diagnosis methods, the brain tumor diagnosis method of this embodiment can make full use of the cross-modal complementary information of multimodal imaging even in the face of a small number of training samples.

[0080] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A brain tumor diagnosis method based on multimodal supervised contrastive learning, characterized in that: Input the patient's MRI images into the trained brain tumor diagnosis model to output the brain tumor diagnosis results; The training process of the brain tumor diagnosis model is as follows: Acquire and preprocess the patient's MRI images to construct an original MRI image set, and fuse the MRI images of the same patient in different modalities in the original MRI image set to obtain a fused MRI image set; The original MRI image set and the fused MRI image set are combined to obtain a combined image set, and image diagnostic features are extracted from the combined image set based on an image encoder; Construct cross entropy loss and multimodal supervised contrast loss to supervise the image encoder to train the brain tumor diagnosis model; The construction process of multimodal supervised contrast loss is as follows: If two samples in the combined image set have the same category label, they are defined as a positive sample pair; if two samples in the combined image set have different category labels, they are defined as a negative sample pair; Construct a label matrix based on positive sample pairs and negative sample pairs; Calculate the similarity between samples in the combined image set and construct a similarity matrix; Based on the label matrix and the similarity matrix, the log-likelihood function of each sample in the combined image set is calculated to maximize the similarity with the positive sample pair and minimize the similarity with its negative sample pair; Compute the multimodal supervised contrastive loss based on the log-likelihood function.

2. The brain tumor diagnosis method based on multimodal supervised contrastive learning according to claim 1, characterized in that: The brain tumor labels are labeled for each patient in the original MRI image set and the fused MRI image set.

3. The brain tumor diagnosis method based on multimodal supervised contrastive learning according to claim 2, characterized in that: In generating the fused MRI image set, MRI images of the same patient in different modalities are fused pairwise.

4. The brain tumor diagnosis method based on multimodal supervised contrastive learning according to claim 2, characterized in that: The construction process of cross entropy loss is as follows: Combining all extracted imaging diagnostic features into a diagnostic feature set; The diagnostic feature set is passed through a layer of fully connected neural network to obtain a predicted category set, and the cross entropy loss between the predicted category set and the true category set is calculated.

5. The brain tumor diagnosis method based on multimodal supervised contrastive learning according to claim 1, characterized in that: The positive sample pair is defined as 1, the negative sample pair is defined as 0, and the label matrix is ​​a symmetric matrix.

6. The brain tumor diagnosis method based on multimodal supervised contrastive learning according to claim 1, characterized in that: The calculation formula of the log-likelihood function is as follows: ； in, For the combined image set The log-likelihood function of samples is is the number of patients, represents the first Sample and The relationship between samples, if it is a positive sample pair, then Take 1, if it is a negative sample pair, then Take 0, For the combined image set Samples and The similarity between samples, For the combined image set Samples and The similarity between samples.

7. The brain tumor diagnosis method based on multimodal supervised contrastive learning according to claim 1, characterized in that: Multimodal Supervised Contrastive Loss The formula is as follows: ； in, For the The log-likelihood function of samples is is the number of patients.

8. A brain tumor diagnosis system based on multimodal supervised contrastive learning, characterized in that: Input the patient's MRI images into the trained brain tumor diagnosis model to output the brain tumor diagnosis results; The training process of the brain tumor diagnosis model includes image acquisition module, modality fusion module, image combination module, image extraction module and loss construction module; The image acquisition module is used to acquire and preprocess the patient's MRI images to construct an original MRI image set; The modality fusion module is used to fuse MRI images of different modalities of the same patient in the original MRI image set to obtain a fused MRI image set; The image combination module is used to combine the original MRI image set and the fused MRI image set to obtain a combined image set; The image extraction module extracts image diagnostic features from the combined image set based on the image encoder; The loss building module is used to construct cross entropy loss and multimodal supervised contrast loss to perform supervised learning on the image encoder to train the brain tumor diagnosis model; The construction process of multimodal supervised contrast loss is as follows: If two samples in the combined image set have the same category label, they are defined as a positive sample pair; if two samples in the combined image set have different category labels, they are defined as a negative sample pair; Construct a label matrix based on positive sample pairs and negative sample pairs; Calculate the similarity between samples in the combined image set and construct a similarity matrix; Based on the label matrix and the similarity matrix, the log-likelihood function of each sample in the combined image set is calculated to maximize the similarity with the positive sample pair and minimize the similarity with its negative sample pair; Compute the multimodal supervised contrastive loss based on the log-likelihood function.

9. The brain tumor diagnosis system based on multimodal supervised contrastive learning according to claim 8, characterized in that: Label brain tumor labels for each patient in the original MRI image set and the fused MRI image set; The construction process of multimodal supervised contrast loss is as follows: If two samples in the combined image set have the same category label, they are defined as a positive sample pair; if two samples in the combined image set have different category labels, they are defined as a negative sample pair; Construct a label matrix based on positive sample pairs and negative sample pairs; Calculate the similarity between samples in the combined image set and construct a similarity matrix; Based on the label matrix and the similarity matrix, the log-likelihood function of each sample in the combined image set is calculated to maximize the similarity with the positive sample pair and minimize the similarity with its negative sample pair; Compute the multimodal supervised contrastive loss based on the log-likelihood function.

Images