A semantically prior-guided incomplete multimodal brain tumor segmentation method

Abstract

This invention discloses an incomplete multimodal brain tumor segmentation method based on semantic prior guidance. The invention constructs an SMML network model based on a mutual learning distillation framework, introducing a MedSAM-3D model in each branch to provide prior semantic guidance. The directional pixel-level distillation algorithm designed in this invention allows the two-branch networks to exchange valuable learned knowledge, further optimizing the network parameters. The combination of these two approaches enables the model to achieve state-of-the-art (SOTA) performance on brain tumor datasets, surpassing other existing models and achieving the best segmentation results to date.

Description

Technical Field

[0001] This invention belongs to the field of brain tumor segmentation technology in incomplete multimodal scenarios. Specifically, it constructs a method for brain tumor segmentation in incomplete multimodal scenarios based on semantic guidance. Background Technology

[0002] Accurate brain tumor segmentation is crucial for clinical assessment and surgical planning. In clinical practice, accurate brain tumor segmentation is typically achieved by combining MRI scans using multiple imaging modalities, including T1-weighted (T1), contrast-enhanced T1-weighted (T1ce), T2-weighted (T2), and attenuated fluid transformation recovery (FLAIR) sequences. CRNP introduces a multimodal learning model with uncertainty awareness through cross-modal random network prediction. UNETR++ introduces an innovative efficient paired attention (EPA) block that cleverly captures rich, interdependent spatial and channel features by simultaneously using spatial and channel attention in two different branches.

[0003] However, in real-world medical scenarios, the absence of one or more modalities is common due to image corruption, artifacts, acquisition protocols, contrast agent allergies, or cost constraints. Current research often employs incomplete multimodal brain tumor segmentation to address treatment scenarios with various missing modalities. Some methods address various missing modal settings by training models specific to a particular modality, but these methods ignore valuable supplementary information present across modalities. Other methods attempt to effectively learn representative multimodal features that can flexibly adapt to different missing modality scenarios. RFNet introduces a region-aware fusion module to enhance feature representation by considering the varying sensitivities of different modalities to tumor regions. mmFormer proposes a transformer-based approach that combines transformers and CNNs to learn modality-invariant representations. Despite significant progress, extracting robust and discriminative multimodal features remains challenging when dealing with arbitrary missing modalities, severely impacting segmentation performance.

[0004] Given the practical value of incomplete multimodal brain tumor segmentation research and the limitations mentioned above, we find it difficult to study how to extract robust and discriminative multimodal features in scenarios with arbitrary missing modalities. To achieve this, we must solve two problems: first, how to extract robust multimodal features; and second, how to obtain discriminative multimodal features. Summary of the Invention

[0005] To address the problems in the background art, this invention provides an incomplete multimodal brain tumor segmentation method based on semantic prior-guided mask mutual learning. The purpose of this invention is to provide prior knowledge to the model through a semantically prior Segment Anything Model (SAM), enhancing the extraction of discriminative features; simultaneously, it employs a direction-guided consistency distillation loss to transfer valuable knowledge, obtaining more robust knowledge.

[0006] The technical solution adopted in this invention is as follows, specifically including the following steps:

[0007] 1) Brain tumor MRI images include four modalities (Flair, T1ce, T1, and T2). These four modalities are input into a modality-specific encoder to generate intermediate feature maps. Here, i represents the two-branch network 1 and 2, i∈{1,2}, and K represents the number of modes, k∈{1,2,3,4};

[0008] 2) The intermediate feature map obtained in step 1) Missing modal inputs are simulated by randomly masking one or more modal features;

[0009] 3) Multimodal feature set of mask The fused feature F is generated through an attention-based multimodal feature fusion module. i ;

[0010] 4) Fusion feature F i The initial prediction results for each branch are generated through a fusion decoder.

[0011] 5) Input the multimodal brain tumor MRI images into the MedSAM-3D model to obtain the segmentation atlas of each branch.

[0012] 6) The segmentation map obtained in step 5) Missing modal inputs are simulated by randomly masking one or more modal features;

[0013] 7) Initial prediction results from stitching step 4) The segmentation map of step 6) Accurate segmentation prediction results are obtained through a refined network.

[0014] 8) Based on step 7), obtain accurate prediction results. and initial segmentation prediction Calculate the consistency loss;

[0015] 9) Based on the bi-branch initial predicted segmentation map obtained in step 4). With real labels Calculate cross-entropy and Dice loss;

[0016] 10) Based on the accurate segmentation prediction results obtained in step 7). With real labels Calculate cross-entropy and Dice loss;

[0017] 11) Initial predicted segmentation map of the two branches obtained in step 4) Calculate distillation losses;

[0018] 12) Combine the consistency loss from step 8), the cross-entropy and Dice loss from step 9), the cross-entropy and Dice loss from step 10), and the distillation loss from step 11) to calculate the overall loss of the model;

[0019] 13) Based on the comprehensive loss training model in step 12), select the set of parameters with the minimum loss as the optimal parameters of the model.

[0020] Step 2) specifically involves: using the random mode mask for each branch. 1 indicates the presence of a mode, and 0 indicates the absence of a mode. This is determined by comparing the intermediate features of the corresponding modes. Multiply to obtain the feature input of any missing modality.

[0021]

[0022] Step 6) specifically involves: similarly, replacing missing modes with 0 for the segmentation map obtained through MedSAM-3D. Simulate scenarios where any modality is missing:

[0023]

[0024] Step 7) specifically involves: splicing the initial prediction results. With MedSAM-3D segmentation map A precise segmentation result is obtained through a sophisticated network.

[0025]

[0026] in This represents the refined network of branch i, which is a simple UNet network. Cat[,] refers to the stitching along the channel size.

[0027] Step 8) specifically involves: calculating the initial prediction result using KL divergence. With accurate segmentation results Consistency loss:

[0028]

[0029] Step 9) specifically involves calculating the task objective losses for branches 1 and 2 respectively.

[0030]

[0031] Where CE(.) and Dice(.) represent cross-entropy and Dice loss, respectively;

[0032] Step 10) specifically involves: for the accurate segmentation prediction results obtained in step 7), By comparing with real labels Calculate the cross-entropy and Dice loss to obtain the task loss for each branch:

[0033]

[0034] Step 11) specifically involves: based on the initial prediction results of the two branches. The direction of knowledge transfer is determined by cross-entropy loss, and distillation loss is calculated. The specific steps are as follows:

[0035] Based on the initial prediction results obtained in step 4) And the true label of one-hot A oh Where H, W, and Z represent the feature space dimensions, and C represents the number of classes. The cross-entropy loss for branches 1 and 2 is calculated using the following formulas:

[0036]

[0037] Where σ(.) denotes the softmax function, and Q1, Q2 ∈ R H×W×Z This represents the pixel-by-pixel cross-entropy loss; the smaller the loss, the more reliable the prediction. Therefore, the bidirectional knowledge transfer mask is obtained according to the following equation:

[0038]

[0039] in This represents an indicator function whose value is 1 when the internal condition is True, and 0 otherwise. M (h,w,z) represents the knowledge transfer direction of pixel (h,w,z), T M (h,w,z) = 1 indicates that reliable knowledge is transferred from branch 2 to 1 (indicating that knowledge of the pixel in branch 2 is not transferred).

[0040] According to T MCalculate the pixel-level consistency loss for network branches 1 and 2 respectively:

[0041]

[0042] in N 12 =H*W*ZN 21 τ represents the temperature factor. 21 N represents the number of pixels from branch 2 to branch 1 from which knowledge is transferred. 12 This represents the number of pixels from which knowledge migrates from branch 1 to branch 2.

[0043] Step 12) specifically involves summing up the losses calculated above to obtain the losses corresponding to the final branches 1 and 2:

[0044]

[0045] The beneficial effects of this invention are:

[0046] The Semantic-guided Masked Mutual Learning (SMML) network model constructed in this invention is mainly based on a mutual learning distillation framework, introducing a MedSAM-3D model into each branch to provide prior semantic guidance. Furthermore, the directional pixel-level distillation algorithm designed in this invention allows the two-branch networks to exchange valuable learned knowledge, further optimizing the network parameters. This combination enables the model to achieve state-of-the-art (SOTA) performance on the brain tumor dataset (BraTs2018), surpassing other existing models and achieving the best segmentation results to date. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] like Figure 1 As shown, assuming that patients receiving imaging protocols with different missing patterns should produce consistent brain lesion results, the present invention provides a brain tumor segmentation method based on semantic prior guidance and mask mutual learning, which specifically includes the following steps:

[0050] Step 1): Brain tumor MRI images include four modalities (Flair, T1ce, T1, and T2). The publicly available 3D multimodal brain tumor segmentation dataset images were input into a modality-specific encoder to generate intermediate feature maps. Here, i represents the two-branch network 1 and 2, i∈{1,2}, and K represents the number of modes, k∈{1,2,3,4};

[0051] Step 2): The intermediate feature map obtained in Step 1) Missing modal inputs are simulated by randomly masking one or more modal features;

[0052] Use the random modal mask for each branch 1 indicates the presence of a mode, and 0 indicates the absence of a mode. This is determined by comparing the intermediate features of the corresponding modes. Multiply to obtain the feature input of any missing modality.

[0053]

[0054] Step 3): Multimodal feature set of the mask The fused feature F is generated through an attention-based multimodal feature fusion module. i ;

[0055] Step 4): Fuse features F i The initial prediction results for each branch are generated through a fusion decoder.

[0056] Step 5): Input the multimodal brain tumor MRI images into the MedSAM-3D model to obtain the segmentation atlas of each branch.

[0057] Step 6) Use the segmentation map obtained in Step 5) Missing modal inputs are simulated by randomly masking one or more modal features;

[0058] Similar to step 2), missing modes are replaced with 0 for the segmentation map obtained by MedSAM-3D. Simulate scenarios where any modality is missing:

[0059]

[0060] Step 7): Assemble the initial prediction results from Step 4). The segmentation map of step 6) Accurate segmentation prediction results are obtained through a refined network.

[0061] splicing initial prediction results With MedSAM-3D segmentation map A precise segmentation result is obtained through a sophisticated network.

[0062]

[0063] in This represents the refined network of branch i, which is a simple UNet network. Cat[,] refers to the stitching along the channel size.

[0064] Step 8): Obtain accurate prediction results based on Step 7). and initial segmentation prediction Calculate the consistency loss;

[0065] Initial prediction results were calculated using KL divergence. With accurate segmentation results Consistency loss:

[0066]

[0067] Step 9): Based on the bi-branch initial predicted segmentation map obtained in Step 4). With real labels Calculate cross-entropy and Dice loss;

[0068] Calculate the task objective loss for branches 1 and 2 respectively:

[0069]

[0070] Where CE(.) and Dice(.) represent cross-entropy and Dice loss, respectively;

[0071] Step 10): Based on the accurate segmentation prediction results obtained in Step 7). With real labels Calculate cross-entropy and Dice loss;

[0072] For the accurate segmentation prediction results obtained in step 7) By comparing with real labels Calculate the cross-entropy and Dice loss to obtain the task loss for each branch:

[0073]

[0074] Step 11): Initial predicted segmentation map based on the bi-branch obtained in Step 4). Calculate distillation losses;

[0075] Initial prediction results based on two branches The direction of knowledge transfer is determined by cross-entropy loss, and distillation loss is calculated. The specific steps are as follows:

[0076] Based on the initial prediction results obtained in step 4) And the true label of one-hot Aoh Where H, W, and Z represent the feature space dimensions, and C represents the number of categories. Calculate the cross-entropy loss for branches 1 and 2 based on the following companies:

[0077]

[0078] Where σ(.) denotes the softmax function, and Q1, Q2 ∈ R H×W×Z This represents the pixel-by-pixel cross-entropy loss; the smaller the loss, the more reliable the prediction. Therefore, the bidirectional knowledge transfer mask is obtained according to the following equation:

[0079]

[0080] in This represents an indicator function whose value is 1 when the internal condition is True, and 0 otherwise. M (h,w,z) represents the knowledge transfer direction of pixel (g,w,z), T M (h,w,z)=1 indicates that reliable knowledge has been transferred from branch 2 to 1.

[0081] According to T M Calculate the pixel-level consistency loss for network branches 1 and 2 respectively:

[0082]

[0083] in N 12 =H*W*ZN 21 τ represents the temperature factor. 21 N represents the number of pixels from branch 2 to branch 1 from which knowledge is transferred. 12 This represents the number of pixels from which knowledge migrates from branch 1 to branch 2.

[0084] Step 12): Combine the consistency loss from step 8), the cross-entropy and Dice loss from step 9), the cross-entropy and Dice loss from step 10), and the distillation loss from step 11) to calculate the overall loss of the model;

[0085] Adding up the losses calculated above, we get the losses corresponding to the final branches 1 and 2:

[0086]

[0087] Step 13): Train the Semantic-guided Masked Mutual Learning (SMML) network model. Based on the final loss function from Step 12), adjust the weights and biases of the network model using backpropagation. Select parameters based on the Dice Similarity Coefficient (DSC), and finally determine a set of parameters that maximizes the image segmentation accuracy (DSC score).

[0088] The brain tumor segmentation model of this invention can determine whether each pixel of a three-dimensional MRI image belongs to its corresponding tumor category in scenarios where any modality is missing, and it has a significant effect on multi-category image segmentation.

[0089] The performance of the SMML model and some state-of-the-art (SOTA) models on the current BraTs2018 brain tumor segmentation dataset is as follows:

[0090]

[0091] Note:

[0092] (1)MCTSeg, M3AE, U-Net-MFI, RFNet, mmFormer and SMU-Net are models that previously achieved state-of-the-art performance on the BraTs2018 dataset, and SMML is the model of this invention.

[0093] (2) WT, TC, and ET represent Whole Tumor, Tumor Core, and Enhancing Tumor, respectively.

[0094] (3) The values ​​in the table are Dice similarity coefficients (DSC). The higher the value, the better the brain tumor segmentation effect.

[0095] (4) Avg: represents the average DSC score of the three tumor categories. The higher the value, the better the segmentation effect.

[0096] (5) The values ​​in bold black in the table indicate the best segmentation performance;

[0097] (6) Compared with the state-of-the-art models MCTSeg, M3AE, U-Net-MFI, RFNet, mmFormer and SMU-Net, the segmentation metrics of this invention on the same dataset improve the DSC of each class by 1.1%, 0.6% and 2.4% respectively, and the average DSC is improved by 1.5%.

Claims

1. A method for segmenting incomplete multimodal brain tumors based on semantic prior guidance, comprising the following steps: 1) Brain tumor MRI images of four modalities are input into dual-branch networks 1 and 2, and four intermediate feature maps are generated by modality-specific encoders. 2) The intermediate feature map obtained in step 1) Randomly mask one or more modal features to simulate the missing modal input, thus obtaining the masked multimodal feature set. Multimodal feature set of mask The fused feature F is generated through an attention-based multimodal feature fusion module. i ; Fusion feature F i An initial predicted segmentation map for each branch is generated using a fusion decoder. 3) Input the multimodal brain tumor MRI images into the MedSAM-3D model to obtain the segmentation atlas of each branch. 4) Segment the map By randomly masking one or more modal features to simulate the missing modal input, a segmentation map of any missing modality is obtained. 5) Assemble the initial predicted segmentation map from step 2). The segmentation map of step 4). Then, a refined network is used to obtain accurate segmentation prediction results. 6) Calculate the overall loss of the model and train the model based on the overall loss function in step 6); 7) Use the trained model to obtain brain tumor segmentation results in incomplete multimodal scenarios; In step 1): Brain tumor MRI images include four modalities: Flair, T1ce, T1, and T2; For intermediate feature maps Where i represents the two-branch network 1 and 2, i∈{1,2}, and K represents the number of modes, k∈{1,2,3,4}.

2. The method for segmenting incomplete multimodal brain tumors based on semantic prior guidance according to claim 1, characterized in that, In step 2): use the random mode mask M for each branch. i Intermediate features of the corresponding modality Multiply to obtain the feature input of any missing modality. in, 1 indicates that the mode exists, and 0 indicates that the mode does not exist.

3. The method for segmenting incomplete multimodal brain tumors based on semantic prior guidance according to claim 1, characterized in that, Step 4) specifically involves: using the random mode mask M for each branch. i Segmentation map of corresponding modes Multiplying them together yields a segmentation map of any missing mode. in, 1 indicates that the mode exists, and 0 indicates that the mode does not exist.

4. The method for segmenting incomplete multimodal brain tumors based on semantic prior guidance according to claim 1, characterized in that, Step 5) specifically involves: splicing the initial predicted segmentation map. With MedSAM-3D segmentation map A precise segmentation result is obtained through a sophisticated network. Among them, f i refine Let i represent the refined network of branch i. The refined network represents a simple UNet network. Cat[,] represents the splicing along the channel size.

5. The method for segmenting incomplete multimodal brain tumors based on semantic prior guidance according to claim 1, characterized in that, Step 6) specifically involves: Based on the accurate prediction result Y obtained in step 5), i refine and the initial predicted segmentation map obtained in step 2). Calculate the consistency loss; based on the bi-branch initial predicted segmentation map obtained in step 2). With real labels Calculate the first cross-entropy and Dice loss; based on the accurate segmentation prediction results obtained in step 5). With real labels Calculate the second cross-entropy and Dice loss; based on the initial predicted segmentation map of the two branches obtained in step 2). Calculate distillation losses; The overall loss of the model is calculated by combining the consistency loss, the first cross-entropy and Dice loss, the second cross-entropy and Dice loss, and the distillation loss.

6. The method for segmenting incomplete multimodal brain tumors based on semantic prior guidance according to claim 5, characterized in that, The method for calculating the consistency loss is as follows: Initial prediction results were calculated using KL divergence. With precise segmentation results Consistency loss: The method for calculating the first cross-entropy and Dice loss is as follows: For the bi-branch initial predicted segmentation map obtained in step 2) By comparing with real labels Calculate the first cross-entropy and Dice loss to obtain the task objective losses for branches 1 and 2: Where CE(.) and Dice(.) represent cross-entropy and Dice loss, respectively; The method for calculating the second cross-entropy and Dice loss is as follows: For the accurate segmentation prediction results obtained in step 5) By comparing with real labels Calculate the second cross-entropy and Dice loss to obtain the task loss for each branch: Where CE(.) and Dice(.) represent cross-entropy and Dice loss, respectively.

7. The method for segmenting incomplete multimodal brain tumors based on semantic prior guidance according to claim 6, characterized in that, The method for calculating distillation loss is as follows: Initial prediction results based on two branches The direction of knowledge transfer is determined by cross-entropy loss, and distillation loss is calculated; the specific operation is as follows: Based on the initial prediction results obtained in step 2) And the true label of one-hot A oh Where H, W, and Z represent the feature space dimensions, and C represents the number of categories; calculate the cross-entropy loss for branches 1 and 2 using the following formulas respectively: Where σ(.) denotes the softmax function, and Q1, Q2 ∈ R H×W×Z The cross-entropy loss represents the value at each pixel; the smaller the loss, the more reliable the prediction. The bidirectional knowledge transfer mask is then obtained using the following formula: in This represents an indicator function whose value is 1 when the internal condition is True, and 0 otherwise. M (H,w,z) represents the knowledge transfer direction of pixel (h,w,z), T M (h,w,z) = 1 indicates that reliable knowledge has been transferred from branch 2 to branch 1, T M (h,w,z) = 0 indicates that reliable knowledge was transferred from branch 1 to branch 2; According to T M Calculate the pixel-level consistency loss for network branches 1 and 2 respectively: in, N 12 =H*W*ZN 21 τ represents the temperature factor; N 21 N represents the number of pixels from branch 2 to branch 1, where N is the number of pixels from which knowledge is transferred. 12 This represents the number of pixels from which knowledge migrates from branch 1 to branch 2.

8. The method for segmenting incomplete multimodal brain tumors based on semantic prior guidance according to claim 7, characterized in that, Combining the consistency loss, the first cross-entropy and Dice loss, the second cross-entropy and Dice loss, and the distillation loss, we obtain the losses corresponding to the final branches 1 and 2:

Images