Semantic priori guidance-based incomplete multi-mode brain tumor segmentation method

Abstract

The invention discloses an incomplete multi-mode brain tumor segmentation method based on semantic prior guidance. According to the method, an SMML network model is constructed based on a mutual learning distillation framework, and a MedSAM-3D model is introduced into each branch to provide prior semantic guidance; according to the directional pixel-level distillation algorithm designed by the invention, the dual-branch network can mutually exchange learned valuable knowledge, and network parameters are further optimized; through the combination of the two models, the model achieves the SOTA effect on a brain tumor data set, the model exceeds other existing models at present, the best segmentation effect is obtained, and the current SOTA is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of brain tumor segmentation in incomplete multimodal scenarios, and specifically constructs a method for assisting and enhancing brain tumor segmentation in incomplete multimodal scenarios based on semantic guidance. Background Art

[0002] Accurate brain tumor segmentation is crucial for clinical assessment and surgical planning. In clinical practice, MRI scans of multiple imaging modalities, including T1-weighted (T1), contrast-enhanced T1-weighted (T1ce), T2-weighted (T2), and attenuated fluid-transformed restoration sequence (Flair) images, are usually combined to achieve accurate brain tumor segmentation. CRNP introduces an uncertainty-aware multimodal learning model through cross-modal random network prediction. UNETR++ introduces an innovative efficient paired attention (EPA) block that cleverly captures rich interdependent spatial and channel features by using spatial attention and channel attention simultaneously in two different branches.

[0003] However, in real medical scenarios, it is common for one or more modalities to be missing due to image corruption, artifacts, acquisition protocols, allergies to contrast agents, or cost constraints. Currently, many studies have investigated incomplete multimodal brain tumor segmentation to handle various treatment situations with missing modalities. Some methods address various missing modality settings by training models for specific modalities, which ignore the valuable complementary information that exists across modalities. Other methods attempt to effectively learn representative multimodal features that can flexibly adapt to different missing modality scenarios. RFNet introduces a region-aware fusion module to enhance feature representation by considering the different sensitivities of different modalities to tumor regions. mmFormer proposes a transformer-based method that combines transformers and CNNs to learn modality-invariant representations. Despite significant progress, it remains challenging to extract robust and discriminative multimodal features when dealing with arbitrary missing modalities, which seriously affects segmentation performance.

[0004] Given the practical value of incomplete multimodal brain tumor segmentation research and the above limitations, we find it difficult to study how to extract robust and discriminative multimodal features in scenarios with arbitrary missing modalities. If we want to achieve this effect, we must solve two problems: how to extract robust multimodal features and how to obtain discriminative multimodal features. Summary of the invention

[0005] In order to solve the problems in the background technology, the present invention provides an incomplete multimodal brain tumor segmentation method based on semantic prior-guided mask mutual learning. The purpose of the present invention is to provide prior knowledge for the model through a semantic prior arbitrary image segmentation model (Segment Anything Model, SAM) to enhance the extraction of discriminative features; at the same time, a direction-guided consistency distillation loss is used to transfer valuable knowledge to obtain more robust knowledge.

[0006] The technical solution adopted by the present invention is as follows, which specifically comprises the following steps:

[0007] 1) Brain tumor MRI images include four modalities (Flair, T1ce, T1, and T2). The images of these four modalities are input into the modality-specific encoder to generate an intermediate feature map. Here i represents the dual-branch network 1 and 2, i∈{1,2}, K represents the number of modes, k∈{1,2,3,4};

[0008] 2) The intermediate feature map obtained in step 1) Simulating missing modality input by randomly masking one or more modality features;

[0009] 3) Masked Multimodal Feature Set Generate fusion feature F through an attention-based multimodal feature fusion module i ;

[0010] 4) Fusion feature F i Generate the initial prediction results of each branch through a fusion decoder

[0011] 5) Input the multimodal brain tumor MRI image into the MedSAM-3D model to obtain the segmentation atlas of each branch

[0012] 6) The segmentation map obtained in step 5) Simulating missing modality input by randomly masking one or more modality features;

[0013] 7) Concatenate the initial prediction results from step 4) and the segmentation map of step 6) Obtaining accurate segmentation predictions through a refinement network

[0014] 8) Based on step 7) get accurate prediction results and the initial segmentation prediction Calculate consistency loss;

[0015] 9) Based on the dual-branch initial prediction segmentation map obtained in step 4) With the true label Calculate cross entropy and Dice loss;

[0016] 10) Based on the accurate segmentation prediction results obtained in step 7) With the true label Calculate cross entropy and Dice loss;

[0017] 11) Based on the initial predicted segmentation map of the two branches obtained in step 4) Calculate distillation losses;

[0018] 12) Calculate the overall loss of the model by combining the consistency loss of step 8), the cross entropy and Dice loss of step 9, the cross entropy and Dice loss of step 10, and the distillation loss of step 11;

[0019] 13) According to the comprehensive loss training model in step 12), a set of parameters with the smallest loss is selected as the optimal parameters of the model.

[0020] The step 2) is specifically as follows: using a random modal mask of each branch 1 means the mode exists, 0 means the mode does not exist, and the corresponding mode intermediate feature Multiply to get the feature input of any missing mode

[0021]

[0022] The specific step 6) is as follows: Similarly, the missing modality is replaced by 0. For the segmentation atlas obtained by MedSAM-3D Simulate any modality missing scenario:

[0023]

[0024] The step 7) is specifically as follows: splicing the initial prediction results Segmentation atlas with MedSAM-3D Get accurate segmentation results through a refined network

[0025]

[0026] in represents the refined network of branch i, which is a simple UNet network, and Cat[,] refers to the concatenation along the channel size;

[0027] The step 8) is specifically as follows: calculating the initial prediction result by KL divergence With accurate segmentation results Consistency loss:

[0028]

[0029] The step 9) is specifically as follows: respectively calculating the task target losses of branches 1 and 2:

[0030]

[0031] Where CE(.) and Dice(.) represent cross entropy and Dice loss;

[0032] The step 10) specifically includes: for the accurate segmentation prediction result obtained in step 7) By comparing with the true label Calculate the cross entropy and Dice loss to get the task loss corresponding to each branch:

[0033]

[0034] The step 11) is specifically as follows: based on the initial prediction results of the two branches The direction of knowledge transfer is determined by cross entropy loss, and the distillation loss is calculated. The specific operations are as follows:

[0035] Based on the initial prediction results obtained in step 4) And the one-hot true label A oh , where H, W, Z represent the feature space dimensions and C represents the number of categories. The cross entropy loss of branches 1 and 2 is calculated according to the following formula:

[0036]

[0037] Where σ(.) represents the softmax function, Q1, Q2∈R H×W×Z represents the pixel-by-pixel cross entropy loss, and the smaller the loss, the more reliable the prediction value. Therefore, the bidirectional knowledge transfer mask is obtained according to the following equation:

[0038]

[0039] in Represents an indicator function, which has a value of 1 when the internal condition is True, and 0 otherwise. M (h,w,z) represents the knowledge transfer direction of pixel point (h,w,z), T M (h, w, z) = 1 indicates that reliable knowledge is transferred from branch 2 to 1 (indicating that the knowledge of the pixels in branch 2 is not transferred).

[0040] According to T M, calculate the pixel-level consistency loss of network branches 1 and 2 respectively:

[0041]

[0042] in N 12 =H*W*ZN 21 , τ represents the temperature factor. N 21 Indicates the number of pixels where knowledge is transferred from branch 2 to branch 1, N 12 The number of pixels that represent the knowledge transfer from branch 1 to branch 2.

[0043] The step 12) is specifically as follows: adding up the losses calculated above to obtain the losses corresponding to the final branches 1 and 2:

[0044]

[0045] Beneficial effects of the present invention:

[0046] The Semantic-guided Masked Mutual Learning (SMML) network model constructed by the present invention is mainly based on the mutual learning distillation framework, and the MedSAM-3D model is introduced in each branch to provide a priori semantic guidance. In addition, the directional pixel-level distillation algorithm designed by the present invention can enable the two-branch network to exchange valuable knowledge learned from each other and further optimize the network parameters. The combination of the two enables the model to achieve SOTA effects on the brain tumor dataset (BraTs2018), surpassing other existing models, achieving the best segmentation effect, and achieving the current SOTA. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] like Figure 1 As shown, assuming that patients receiving different missing pattern imaging protocols should produce consistent brain lesion results, the present invention provides a method for brain tumor segmentation based on semantic prior-guided mask mutual learning, which specifically includes the following steps:

[0050] Step 1): Brain tumor MRI images include four modalities (Flair, T1ce, T1, and T2). The public 3D multimodal brain tumor segmentation dataset images are input into the modality-specific encoder to generate an intermediate feature map. Here i represents the dual-branch network 1 and 2, i∈{1,2}, K represents the number of modes, k∈{1,2,3,4};

[0051] Step 2): The intermediate feature map obtained in step 1) Simulating missing modality input by randomly masking one or more modality features;

[0052] Use a random modality mask for each branch 1 means the mode exists, 0 means the mode does not exist, and the corresponding mode intermediate feature Multiply to get the feature input of any missing mode

[0053]

[0054] Step 3): Masked multimodal feature set Generate fusion feature F through an attention-based multimodal feature fusion module i ;

[0055] Step 4): Fusion feature F i Generate the initial prediction results of each branch through a fusion decoder

[0056] Step 5): Input the multimodal brain tumor MRI image into the MedSAM-3D model to obtain the segmentation map of each branch

[0057] Step 6) The segmentation map obtained in step 5) Simulating missing modality input by randomly masking one or more modality features;

[0058] Similar to step 2), the missing modalities are replaced by 0. For the segmentation maps obtained by MedSAM-3D Simulate any modality missing scenario:

[0059]

[0060] Step 7): Concatenate the initial prediction results from step 4) and the segmentation map of step 6) Obtaining accurate segmentation predictions through a refinement network

[0061] Concatenate initial prediction results Segmentation atlas with MedSAM-3D Get accurate segmentation results through a refined network

[0062]

[0063] in represents the refined network of branch i, which is a simple UNet network, and Cat[,] refers to the concatenation along the channel size;

[0064] Step 8): Get accurate prediction results based on step 7) and the initial segmentation prediction Calculate consistency loss;

[0065] Calculate the initial prediction result by KL divergence With accurate segmentation results Consistency loss:

[0066]

[0067] Step 9): Based on the dual-branch initial prediction segmentation map obtained in step 4 With the true label Calculate cross entropy and Dice loss;

[0068] Calculate the task target loss of branches 1 and 2 respectively:

[0069]

[0070] Where CE(.) and Dice(.) represent cross entropy and Dice loss;

[0071] Step 10): Based on the accurate segmentation prediction results obtained in step 7 With the true label Calculate cross entropy and Dice loss;

[0072] For the accurate segmentation prediction results obtained in step 7) By comparing with the true label Calculate the cross entropy and Dice loss to get the task loss corresponding to each branch:

[0073]

[0074] Step 11): Based on the initial predicted segmentation map of the two branches obtained in step 4) Calculate distillation losses;

[0075] Initial prediction results based on dual branches The direction of knowledge transfer is determined by cross entropy loss, and the distillation loss is calculated. The specific operations are as follows:

[0076] Based on the initial prediction results obtained in step 4) And the one-hot true label Aoh , where H, W, Z represent the feature space dimensions and C represents the number of categories. The cross entropy loss of branches 1 and 2 is calculated according to the following company:

[0077]

[0078] Where σ(.) represents the softmax function, Q1, Q2∈R H×W×Z represents the pixel-by-pixel cross entropy loss, and the smaller the loss, the more reliable the prediction value. Therefore, the bidirectional knowledge transfer mask is obtained according to the following equation:

[0079]

[0080] in Represents an indicator function, which has a value of 1 when the internal condition is True, and 0 otherwise. M (h,w,z) represents the knowledge transfer direction of pixel point (g,w,z), T M (h,w,z)=1 indicates that reliable knowledge is transferred from branch 2 to 1.

[0081] According to T M , calculate the pixel-level consistency loss of network branches 1 and 2 respectively:

[0082]

[0083] in N 12 =H*W*ZN 21 , τ represents the temperature factor. N 21 Indicates the number of pixels where knowledge is transferred from branch 2 to branch 1, N 12 The number of pixels that represent the knowledge transfer from branch 1 to branch 2.

[0084] Step 12): Combine the consistency loss of step 8), the cross entropy and Dice loss of step 9, the cross entropy and Dice loss of step 10, and the distillation loss of step 11 to calculate the overall loss of the model;

[0085] Add up the losses calculated above to get the final losses corresponding to branches 1 and 2:

[0086]

[0087] Step 13): Train the Semantic-guided Masked Mutual Learning (SMML) network model. According to the final loss function of step 12), use error back propagation to adjust the weights and biases of the network model. Select parameters according to the size of the Dice Similarity Coefficient (DSC), and finally determine a set of parameters that maximize the image segmentation accuracy (DSC score).

[0088] The brain tumor segmentation model of the present invention can determine whether each pixel of a three-dimensional MRI image belongs to its corresponding tumor category in any modality missing scenario, and has a significant effect on multi-category image segmentation.

[0089] The effects of the SMML model and some SOTA models on the current BraTs2018 brain tumor segmentation dataset are as follows:

[0090]

[0091] Note:

[0092] (1) MCTSeg, M3AE, U-Net-MFI, RFNet, mmFormer and SMU-Net are the models that previously achieved SOTA on the BraTs2018 dataset, and SMML is the model of the present invention;

[0093] (2) WT, TC, and ET represent Whole Tumor, Tumor Core, and Enhancing Tumor, respectively;

[0094] (3) The value in the table is the Dice Similarity Coefficient (DSC). The larger the value, the better the brain tumor segmentation effect;

[0095] (4) Avg: represents the average DSC score of the three tumor categories. The larger the value, the better the segmentation effect;

[0096] (5) The bold black values ​​in the table indicate the best segmentation performance;

[0097] (6) Compared with the SOTA models MCTSeg, M3AE, U-Net-MFI, RFNet, mmFormer and SMU-Net, the segmentation index of the present invention on the same dataset is improved by 1.1%, 0.6%, and 2.4% for each category, respectively, and the average DSC is improved by 1.5%.

Claims

1. An incomplete multimodal brain tumor segmentation method based on semantic prior guidance, comprising the following steps: 1) Input the four modalities of brain tumor MRI images into the dual-branch network 1 and 2, and generate four intermediate feature maps through the modality-specific encoder 2) The intermediate feature map obtained in step 1) Randomly mask one or more modal features to simulate missing modal input and obtain a masked multimodal feature set Masked Multimodal Feature Sets Generate fusion feature F through an attention-based multimodal feature fusion module i ; Fusion feature F i Generate the initial predicted segmentation map for each branch through a fusion decoder 3) Input the multimodal brain tumor MRI image into the MedSAM-3D model to obtain the segmentation atlas of each branch 4) Segment the atlas Simulating missing modality input by randomly masking one or more modality features; 5) Splice the initial predicted segmentation map from step 2) and the segmentation map of step 3) Then a refinement network is used to obtain accurate segmentation prediction results 6) Calculate the overall loss of the model and train the model based on the overall loss function of step 6); 7) Use the trained model to obtain brain tumor segmentation results in incomplete multimodal scenarios.

2. The incomplete multimodal brain tumor segmentation method based on semantic prior guidance according to claim 1, characterized in that: In the step 1): Brain tumor MRI images include four modalities: Flair, T1ce, T1, and T2; For the intermediate feature map Where i represents the two-branch networks 1 and 2, i∈{1, 2}, and K represents the number of modes, k∈{1, 2, 3, 4}.

3. The incomplete multimodal brain tumor segmentation method based on semantic prior guidance according to claim 1, characterized in that: In step 2), use the random modal mask M of each branch i and the corresponding modal intermediate features Multiply to get the feature input of any missing mode in, 1 means the mode exists, 0 means the mode does not exist.

4. The incomplete multimodal brain tumor segmentation method based on semantic prior guidance according to claim 1, characterized in that: The step 4) is specifically as follows: using the random modal mask M of each branch i Segmentation atlas with corresponding modality Multiply them together to get the segmentation map of any missing mode in, 1 means the mode exists, 0 means the mode does not exist.

5. The incomplete multimodal brain tumor segmentation method based on semantic prior guidance according to claim 1, characterized in that: The step 5) is specifically as follows: splicing the initial predicted segmentation map Segmentation atlas with MedSAM-3D Get accurate segmentation results through a refined network in, Represents the refined network of branch i, the refined network represents a simple UNet network, and Cat[,] represents the splicing along the channel size.

6. The incomplete multimodal brain tumor segmentation method based on semantic prior guidance according to claim 1, characterized in that: The step 6) is specifically as follows: Based on the accurate prediction results obtained in step 5) and the initial predicted segmentation map obtained in step 2) Calculate the consistency loss; based on the dual-branch initial prediction segmentation map obtained in step 2) With the true label Calculate the first cross entropy and Dice loss; based on the accurate segmentation prediction results obtained in step 5) With the true label Calculate the second cross entropy and Dice loss; Based on the initial predicted segmentation map of the two branches obtained in step 2) Calculate distillation losses; The overall loss of the model is calculated by combining consistency loss, first cross entropy and Dice loss, second cross entropy and Dice loss, and distillation loss.

7. The incomplete multimodal brain tumor segmentation method based on semantic prior guidance according to claim 6, characterized in that: The method for calculating the consistency loss is specifically as follows: Calculate the initial prediction result by KL divergence With accurate segmentation results Consistency loss: The method for calculating the first cross entropy and Dice loss is specifically as follows: For the dual-branch initial prediction segmentation map obtained in step 2) By comparing with the true label Calculate the first cross entropy and Dice loss to get the task target loss of branches 1 and 2: Where CE(.) and Dice(.) represent cross entropy and Dice loss; The method for calculating the second cross entropy and Dice loss is specifically as follows: For the accurate segmentation prediction results obtained in step 5) By comparing with the true label Calculate the second cross entropy and Dice loss to get the task loss corresponding to each branch: Where CE(.) and Dice(.) represent cross entropy and Dice loss.

8. The incomplete multimodal brain tumor segmentation method based on semantic prior guidance according to claim 6, characterized in that: The method for calculating the distillation loss is specifically as follows: Initial prediction results based on dual branches The direction of knowledge transfer is determined by cross entropy loss, and the distillation loss is calculated. The specific operations are as follows: Based on the initial prediction results obtained in step 2) And the one-hot true label A oh , where H, W, Z represent the feature space dimensions, and C represents the number of categories; the cross entropy loss of branches 1 and 2 is calculated according to the following formula: Where σ(.) represents the softmax function, Q1, Q2∈R H×W×Z represents the pixel-by-pixel cross entropy loss. The smaller the loss, the more reliable the prediction value. The bidirectional knowledge transfer mask is obtained according to the following formula: in Represents an indicator function, which has a value of 1 when the internal condition is True, and 0 otherwise. M (h,w,z) represents the knowledge transfer direction of pixel point (h,w,z), T M (h,w,z)=1 means reliable knowledge is transferred from branch 2 to 1, T M (h,w,z)=0 means reliable knowledge is transferred from branch 1 to 2; According to T M , calculate the pixel-level consistency loss of network branches 1 and 2 respectively: in, N 12 =H*W*ZN 21 , τ represents the temperature factor; N 21 Indicates the number of pixels where knowledge is transferred from branch 2 to branch 1, N 12 The number of pixels that represent the knowledge transfer from branch 1 to branch 2.

9. The incomplete multimodal brain tumor segmentation method based on semantic prior guidance according to claim 6, characterized in that: Combining the consistency loss, the first cross entropy and Dice loss, the second cross entropy and Dice loss, and the distillation loss, we get the final losses corresponding to branches 1 and 2:

Images