A brain tumor magnetic resonance image segmentation method based on dynamic multimodal interactive networks

Abstract

This invention belongs to the field of multimodal magnetic resonance imaging (MRI) image segmentation technology, specifically relating to a brain tumor MRI image segmentation method based on a dynamic multimodal interactive network. The method includes the following steps: Step 1: In the dynamic multimodal interactive network, a two-stage segmentation architecture is designed to complete modality feature extraction and dynamic fusion; Step 2: A dynamic masking weighting module (MASK-AOE) is added to the skip connection part of the encoder and decoder, including a dynamic masking mechanism and a lightweight proxy network; masking and dynamic fusion of input modality features enhance the model's adaptability to modality loss; a dual-path fusion interaction module is added to the connection part of the encoder and decoder; the dual-path fusion interaction module includes modality grouping fusion, a bidirectional interaction mechanism, and a visual Mamba layer; the dual-path fusion interaction module realizes the modeling of collaborative relationships between modalities and feature optimization. This invention can still generate high-quality segmentation results even under modality loss conditions.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal magnetic resonance image segmentation, and particularly relates to a brain tumor magnetic resonance image segmentation method based on a dynamic multi-modal interactive network. BACKGROUND

[0002] Multi-modal magnetic resonance imaging (MRI) has become an important technical means in brain tumor segmentation tasks. Common MRI modalities include T1, T1 contrast-enhanced (T1ce), T2, and FLAIR. In brain tumor imaging, T1 clearly shows the anatomical structure of brain tissue, and tumors usually appear as low signals. T1ce highlights the blood-brain barrier disruption area by injecting contrast agents, and the tumor and its boundary are enhanced and clearly visualized. T2 is highly sensitive to high water content lesions (such as edema or necrosis around the tumor), and the lesions are usually high signals. FLAIR imaging enhances lesion contrast while suppressing cerebrospinal fluid signals, and is particularly suitable for displaying peritumoral edema or lesions near the ventricles. Combining these modalities can comprehensively evaluate the scope, nature of the tumor, and its impact on the surrounding brain tissue. They provide complementary information on the anatomical structure of the tumor, edema area, and active lesions through different imaging mechanisms, providing key evidence for accurate segmentation. In the brain tumor magnetic resonance image segmentation task, the tumor needs to be divided into the following three regions: the whole tumor (WT, Whole Tumor), the tumor core (TC, Tumor Core), and the enhancing tumor (ET, Enhancing Tumor); these segmentation results are of great significance for clinical diagnosis, prognosis evaluation, and preoperative planning. However, in actual clinical applications, due to patient conditions, equipment limitations, or differences in scanning protocols, MRI modalities are often missing, which poses a great challenge to brain tumor magnetic resonance image segmentation tasks.

[0003] To address the above problems, some existing technologies have attempted to solve the influence of modality missing on segmentation performance. For example, some methods in the existing technology calculate the mean and variance of available modality features to simply handle the modality missing problem. However, this method fails to fully capture the complementary information between modalities, limiting further improvement of segmentation performance. Some methods in the existing technology use a generative adversarial network (GAN) to generate missing modalities, thereby alleviating the influence of modality missing. However, when modalities are severely missing, the quality of the generated results is poor. In addition, methods based on Transformers achieve global modeling through self-attention mechanisms and exhibit excellent performance in multi-modal brain tumor segmentation, but their robustness to modality missing still has room for improvement. The current research methods still have the following key problems:

[0004] 1. Modality sensitivity difference problem: For example, the FLAIR modality is more sensitive to the edema region (ED), and the T1ce modality is more sensitive to the enhanced tumor region (ET). The existing method fails to explicitly model the correspondence between the modality and the tumor region characteristics, limiting the further improvement of the segmentation accuracy.

[0005] 2. Inter-modality feature fusion problem: Most traditional methods use simple feature splicing or addition operations, which are difficult to capture complex inter-modality interaction information, resulting in the failure to fully exploit the modality complementarity.

[0006] 3. Stability problem under modality loss: The network is prone to bias towards certain modality-specific features, which not only affects the stability of feature fusion, but also leads to a decline in segmentation performance. SUMMARY

[0007] The purpose of the present application is to provide a brain tumor magnetic resonance image segmentation method based on a dynamic multi-modality interaction network, based on a dynamic multi-modality interaction segmentation network (DMI-Net) for incomplete multi-modality brain tumor segmentation task. DMI-Net can significantly improve the brain tumor segmentation performance under modality loss by introducing a two-stage network architecture, a dynamic mask weighting module MASK-AOE and a double-path fusion interaction module BILGII, solving the problems of insufficient cognitive modality sensitivity difference, insufficient inter-modality feature fusion and training instability, and exhibiting excellent robustness and generalization ability of the model.

[0008] The technical solutions adopted by the present application are as follows:

[0009] The brain tumor magnetic resonance image segmentation method based on the dynamic multi-modality interaction network comprises the following steps:

[0010] Step 1: In the dynamic multi-modality interaction network, a two-stage segmentation architecture is designed to complete modality feature extraction and dynamic fusion, and the functions and specific methods of the two stages are as follows:

[0011] Step 101: Single modality segmentation stage;

[0012] Segment the modality separately, and constrain the inter-modality shared information learning through consistency loss; use the FLAIR and T1ce modality encoder and fusion decoder D fuse to independently generate segmentation result predictions one and two, and constrain the similarity of the segmentation results of different modalities in the class probability distribution by calculating the consistency loss (Consistency Loss) of predictions one and two, thereby learning the shared information between modalities and improving the adaptability of the model to the modality loss condition;

[0013] Step 102: Multi-modality fusion segmentation stage;

[0014] In the multi-modal fusion segmentation stage: two decoders are included: a fusion decoder D fuse and a single-modal decoder D sep , the fusion decoder D fuse is used for shared decoding of multi-modal features to improve multi-modal joint segmentation performance; the fusion decoder D fuse is the same as the one-stage fusion decoder D fuse , through the shared decoder, the model can realize consistent feature decoding logic in single-modal and multi-modal cases, improve the segmentation performance, and enhance the adaptability to the missing modal case; the single-modal decoder D sep decodes the features of each modality into segmentation prediction results independently, in this way, the model can independently evaluate the contribution of each modality to the segmentation task, and at the same time, the independence of such multi-modal segmentation results provides a stronger basis for the fusion strategy of the model; by jointly optimizing the output results of the fusion decoder and the single-modal decoder, the multi-modal fusion segmentation stage further improves the segmentation performance and enhances the robustness of the network to the missing modal case;

[0015] Step 2: A dynamic mask weighting module MASK-AOE is added to the skip connection part of the encoder and decoder, the dynamic mask weighting module MASK-AOE includes a dynamic mask mechanism and a lightweight proxy network; the mask of the input modal feature is dynamically fused to enhance the adaptability of the model to the missing modal case;

[0016] Step 3: A double-path fusion interaction module BILGII is added to the connection part of the encoder and decoder; the double-path fusion interaction module BILGII includes modal grouping fusion, bidirectional interaction mechanism and Vision-Mamba layer (Vision-Mamba); the double-path fusion interaction module BILGII realizes the modeling of the cooperative relationship between modalities and the optimization of features.

[0017] Preferably, the step 2 includes the following sub-steps:

[0018] Step 201: The invalid modal features are shielded through the dynamic mask mechanism, only the valid modal features are retained, so as to force the network to complete the segmentation task with incomplete modalities and improve the robustness of the model;

[0019] Step 202: The masked result obtained in step 201 enters the proxy network, learns multi-view feature representation through feature compression, enhancement and expansion, and realizes weighted fusion of modal features through a dynamic weight mechanism, thereby enhancing the complementarity between modalities.

[0020] Preferably, the step 3 includes the following steps:

[0021] Step 301: Group two sets of modalities flair, t2 and t1ce, t1, respectively, and perform convolution operation after splicing to obtain modality pairs flair_t2 and t1ce_t1; processing modality pairs can make the model explicitly learn the synergy between modalities, rather than simply relying on subsequent global feature stacking;

[0022] Step 302: After global pooling of the modality pairs obtained in step 301, the global semantic information is extracted through a fully connected layer, and scaling factor 1 and scaling factor 2 are generated. The scaling factor is multiplied by the modality pair to obtain the scaling feature, and then the t1ce_t1 scaling feature is added to the flair_t2, and the flair_t2 scaling feature is added to the t1ce_t1, to realize the information interaction between the two modality pairs. This interaction between modality pairs can realize information sharing, so that the modalities can complement the missing information between the modalities

[0023] Preferably, in step 1, the consistency loss is calculated in the first stage to constrain the segmentation result generated by the single modality encoder, so that the model can learn the shared information between modalities. The consistency loss L con The calculation formula is as follows:

[0024]

[0025] Where P1(c|x i ) is the class probability of prediction one, P2(c|x i ) is the class probability of prediction two, N is the sample number, C is the class number, ò=1e-6, to prevent numerical instability.

[0026] Preferably, in step 1, the synergy optimization of the fusion decoder D fuse and the single modality decoder D sep in the second stage improves the comprehensive segmentation performance of multi-modal features; the calculation formula of the fusion loss and the single modality loss includes cross-entropy loss and DICE loss, wherein the cross-entropy loss L CE The calculation formula is as follows:

[0027]

[0028] N is the total number of samples, C is the total number of classes, wherein y i,c is the true label of sample i belonging to class c, is the predicted value output by the model. The summations in the numerator and denominator are both the total sum of pixel points;

[0029] The DICE loss calculation L DICE The formula is as follows:

[0030]

[0031] The parameters are defined in the same way as those in the cross-entropy loss function formula.

[0032] Preferably, the single-mode decoder D sep The single-mode segmentation loss is used to independently optimize the output of the single-mode decoder, and its total loss function L sep The calculation formula is as follows:

[0033]

[0034] in, This indicates that the single-mode decoder D sep The cross-entropy loss is calculated from the output results. This indicates that the single-mode decoder D sep The output shows the calculated DICE loss.

[0035] Preferably, the fusion decoder D fuse The fusion segmentation loss is used for supervised optimization of the output of the multimodal decoder, and its total loss function L fuse The calculation formula is as follows:

[0036]

[0037] in, Indicated by the fusion decoder D fuse The cross-entropy loss is calculated from the output results. Indicated by the fusion decoder D fuse The output shows the calculated DICE loss.

[0038] Preferably, the total loss function of the dynamic multimodal interactive segmentation network proposed in this invention combines consistency loss, fusion loss, and single-modal branch loss, and is optimized in stages. The formula for calculating the total loss function in the first stage is as follows:

[0039] L I total =L con

[0040] The second stage mainly optimizes the fusion loss and single-modal branch loss, while weakening the consistency loss, as shown in the following formula:

[0041] L II total =L fuse +L sep .

[0042] The technical effects achieved by this invention are as follows:

[0043] The application discloses a dynamic multi-modal interaction network (DMI-Net), which aims at the problem of segmentation performance decline caused by missing modalities in incomplete multi-modal brain tumor segmentation tasks, proposes a two-stage dynamic segmentation architecture, and designs a dynamic mask weighting module MASK-AOE and a double-path interactive fusion module BILGII. Through innovative network design, the application can dynamically adjust the feature contribution of the remaining modalities in the case of missing modalities, significantly improving the segmentation accuracy and the adaptability to the missing modalities. Compared with the prior art, the application has superior segmentation performance, and can still generate high-quality segmentation results even under the condition of missing modalities, providing a new solution for brain tumor segmentation of incomplete multi-modal data. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the overall flowchart of the application based on the dynamic multi-modal interaction network DMI-Net for missing modality brain tumor magnetic resonance image segmentation;

[0045] Figure 2 is a schematic diagram of the dynamic mask weighting module MASK-AOE in the application;

[0046] Figure 3 is a schematic diagram of the double-path fusion interaction module BILGII in the application. DETAILED DESCRIPTION

[0047] In order to make the purpose and advantages of the application clearer and more apparent, the application will be specifically described below in combination with embodiments. It should be understood that the following text is only used to describe one or several specific embodiments of the application, and does not strictly limit the specific protection scope requested by the application.

[0048] As shown in Figures 1-3 , the brain tumor magnetic resonance image segmentation method based on the dynamic multi-modal interaction network comprises the following steps:

[0049] Step 1: In the dynamic multi-modal interaction network, a two-stage segmentation architecture is designed to complete modality feature extraction and dynamic fusion, and the functions and specific methods of the two stages are as follows:

[0050] Step 101: single modality segmentation stage;

[0051] The modalities are segmented separately, and the shared information learning between modalities is constrained by consistency loss; the FLAIR and T1ce modality encoders and fusion decoder D fuseThe segmentation result prediction one and the prediction two are independently generated, the consistency loss is calculated by the prediction one and the prediction two to constrain the similarity of segmentation results of different modalities on the class probability distribution, so that the shared information between modalities is learned, and the adaptability of the model to the missing modalities is improved.

[0052] Step 102: multi-modal fusion segmentation stage;

[0053] In the multi-modal fusion segmentation stage, two decoders are included: a fusion decoder D fuse and a single-modal decoder D sep The fusion decoder D fuse is used for shared decoding of multi-modal features to improve multi-modal joint segmentation performance. The fusion decoder D fuse is the same as the fusion decoder D fuse of the first stage. Through the shared decoder, the model can realize consistent feature decoding logic in single-modal and multi-modal cases, improve segmentation performance, and enhance adaptability to missing modalities. The single-modal decoder D sep decodes the features of each modality into segmentation prediction results independently. In this way, the model can independently evaluate the contribution of each modality to the segmentation task, and the independence of multi-modal segmentation results provides a stronger foundation for the fusion strategy of the model. By jointly optimizing the output results of the fusion decoder and the single-modal decoder, the multi-modal fusion segmentation stage further improves the segmentation performance and enhances the robustness of the network to missing modalities.

[0054] Step 2: A dynamic mask weighting module MASK-AOE is added to the skip connection part of the encoder and decoder. The dynamic mask weighting module MASK-AOE includes a dynamic mask mechanism and a lightweight proxy network. The mask of the input modal feature is dynamically fused to enhance the adaptability of the model to the missing modalities.

[0055] Step 3: A dual-path fusion interaction module BILGII is added to the connection part of the encoder and decoder. The dual-path fusion interaction module BILGII includes modal grouping fusion, bidirectional interaction mechanism, and Vision-Mamba layer. The dual-path fusion interaction module BILGII realizes the modeling of the cooperative relationship between modalities and the optimization of features.

[0056] Preferably, step 2 includes the following sub-steps:

[0057] Step 201: The invalid modal features are shielded by the dynamic mask mechanism, and only the valid modal features are retained, so as to force the network to complete the segmentation task using incomplete modalities and improve the robustness of the model.

[0058] Step 202: The result of the mask obtained in step 201 enters the agent network, and through feature compression, enhancement and expansion, learns multi-view feature representation, and realizes weighted fusion of modal features by combining dynamic weight mechanism, thereby enhancing the complementarity between modalities.

[0059] Preferably, step 3 comprises the following steps:

[0060] Step 301: Group the two groups of modalities flair, t2 and t1ce, t1, respectively, and then use convolution operation to obtain the modal pairs flair_t2 and t1ce_t1 after splicing; processing modal pairs can make the model learn the synergy between modalities explicitly, rather than simply relying on subsequent global feature stacking;

[0061] Step 302: After global pooling of the modal pairs obtained in step 301, the global semantic information is extracted through a fully connected layer, and scaling factor 1 and scaling factor 2 are generated. The scaling factor is multiplied by the modal pair to obtain the scaling feature, and then the t1ce_t1 scaling feature is added to the flair_t2, and the flair_t2 scaling feature is added to the t1ce_t1, to realize the information interaction between the two modal pairs. This interaction between modal pairs can realize information sharing, so that the modal pairs can complement the missing information

[0062] Preferably, in step 1, the consistency loss is calculated in the first stage to constrain the segmentation result generated by the single-modal encoder, so that the model can learn the shared information between modalities. The consistency loss L con The calculation formula is as follows:

[0063]

[0064] Where P1(c|x i ) is the class probability of prediction one, P2(c|x i ) is the class probability of prediction two, N is the sample number, C is the class number, ò=1e-6, to prevent numerical instability.

[0065] Preferably, in step 1, the synergy optimization of the fusion decoder D fuse and the single-modal decoder D sep in the second stage improves the comprehensive segmentation performance of multi-modal features; the calculation formula of the fusion loss and the single-modal loss includes cross-entropy loss and DICE loss, wherein the cross-entropy loss L CE The calculation formula is as follows:

[0066]

[0067] N is the total number of samples, C is the total number of classes, wherein y i,c is the true label of sample i belonging to class c, is the prediction value output by the model. The summations in the numerator and denominator are both summations over the pixel points;

[0068] DICE loss calculation L DICE The formula is:

[0069]

[0070] The definitions of the parameters are the same as those in the formula of the cross-entropy loss function.

[0071] Preferably, the single-modal decoder D sep The single-modal segmentation loss of the single-modal decoder is used to independently optimize the output of the single-modal decoder, and the total loss function L sep The calculation formula is as follows:

[0072]

[0073] wherein, represents the cross-entropy loss calculated by the output result of the single-modal decoder D sep represents the DICE loss calculated by the output result of the single-modal decoder D sep

[0074] Preferably, the fusion segmentation loss of the fusion decoder D fuse The fusion segmentation loss of the fusion decoder is used to supervise and optimize the output of the multi-modal decoder, and the total loss function L fuse The calculation formula is as follows:

[0075]

[0076] wherein, represents the cross-entropy loss calculated by the output result of the fusion decoder D fuse represents the DICE loss calculated by the output result of the fusion decoder D fuse

[0077] Preferably, the total loss function of the dynamic multi-modal interactive segmentation network proposed by the present application combines the consistency loss, the fusion loss and the single-modal branch loss, and is optimized in stages. The calculation formula of the total loss function in the first stage is as follows:

[0078] L I total = L con

[0079] In the second stage, the fusion loss and the single-modal branch loss are mainly optimized, and the consistency loss is weakened. The formula is as follows:

[0080] L II total = L​​​​fuse +L sep .

[0081] The application is in specific practical use:

[0082] In the single-modal segmentation stage (Stage I):

[0083] This stage mainly aims at independent segmentation of modalities, and the shared information learning between modalities is constrained by consistency loss.

[0084] First, an independent encoder is designed for each MRI modality to extract high-quality features specific to the modality. The encoder is composed of multiple convolutional layers, pooling layers and skip connections, which can capture local and global information of the modality features.

[0085] Then, the decoder is optimized by consistency constraint, so that the segmentation results of each modality remain consistent in the class probability distribution. Specifically, for the segmentation results generated by the modalities, a consistency loss L con :

[0086]

[0087] Where P1(c|x i ) is the class probability of prediction 1, P2(c|x i ) is the class probability of prediction 2, N is the number of samples, C is the number of classes, and ò=1e-6 is used to prevent numerical instability.

[0088] Finally, through the consistency loss, the network can learn the shared characteristics between modalities, enhance the adaptability to the missing modality condition, and at the same time guarantee the independent segmentation ability of each modality feature.

[0089] In the multi-modal fusion segmentation stage (Stage II):

[0090] This stage mainly realizes dynamic aggregation and shared decoding of multi-modal features through collaborative optimization of fusion decoder and single-modal decoder, improves the segmentation performance, and enhances the adaptability of the network to the missing modality condition.

[0091] First, a dynamic mask mechanism is designed for the dynamic mask weighting module MASK-AOE to apply a dynamic mask operation to the input modality features. For invalid modality features, set the mask to zero, and only keep the valid modality features. The dynamic mask mechanism can mask the missing modality features in the training and inference stages, and force the network to use incomplete modalities to complete the segmentation task. Then a proxy network is designed to compress, enhance and expand the masked modality features to generate multi-view enhanced features. The proxy network learns the complementary information between modalities through convolution operations, thereby improving the feature representation ability. Finally, dynamic weight calculation is performed to calculate the weight of each modality feature, and the formula is as follows:

[0092]

[0093] where w i represents the weight of the i-th branch, represents the weight score of each branch calculated by a normalization (softmax) function after a series of feature extraction. Then the obtained weights are used to weight each branch:

[0094] exp out = w1·aoe1+ w2·aoe2+ w3·aoe3

[0095] aoe1, aoe2, aoe3 are enhanced features, exp out is the output of branch weighting. Through the combination of mask and dynamic weight mechanism, the dynamic mask weighting module MASK-AOE can realize robust and efficient multi-modal feature processing in multi-modal medical image segmentation.

[0096] The decoder of this stage includes a fusion decoder D fuse and a single-modal decoder D sep The fusion decoder D fuse is used for shared decoding of multi-modal features, and generates segmentation results through multi-layer convolution operations. The single-modal decoder D sep provides an independent decoding path for each modality, ensuring that the features of each modality can contribute individually to the segmentation task.

[0097] In the dual-path fusion interaction module BILGII

[0098] This module models the cooperative relationship between modalities through modality grouping fusion, bidirectional interaction mechanism and global feature modeling, and captures long-range dependencies in modality pair features to improve the global consistency and segmentation performance of fused features, thereby enhancing the robustness of the network in the case of missing modalities.

[0099] First, modality grouping fusion is performed, and modalities with similar characteristics are grouped, such as FLAIR-T2 and T1ce-T1. Modality pair features are generated through convolution operations, such as FLAIR-T2 modality pair features.

[0100] Then, bidirectional interaction is performed, and the features of one modality pair are added to another modality pair to realize information sharing between modality pairs. By supplementing missing information, the consistency and segmentation performance of global features are enhanced;

[0101] Finally, the long-range dependencies in the modal pair features are captured by the Vision-Mamba layer, which optimizes the ability of collaborative information fusion between modalities. The Vision-Mamba layer models the global relationship of the modal pair features through the multi-head attention mechanism, further improving the feature fusion accuracy.

[0102] In the experimental process of the present application:

[0103] The experimental platform of the present application is shown in Table 1. The experiment is performed on the BraTS (Brain Tumor Segmentation) dataset and the results are evaluated, including BraTS2018 and BraTS2020 two versions. Each case in the dataset includes four modalities FLAIR, T1ce, T1 and T2.

[0104] BraTS2018 contains 285 cases, and the data is divided into a training set of 199 cases, a validation set of 29 cases and a test set of 57 cases.

[0105] Evaluation index: the present application uses the Dice coefficient as the main evaluation index, which is used to measure the similarity between the segmentation result and the true label, and is defined as:

[0106]

[0107] wherein, is the number of pixels of the predicted segmentation result, |y| is the number of pixels of the true label, is the number of pixels of the intersection region of the predicted segmentation result and the true label. Its value range is 0 to 1, and the closer the value is to 1, the higher the similarity between the segmentation result and the true label, that is, the higher the segmentation accuracy of the model.

[0108] Table 1 Experimental platform

[0109]

[0110] Comparison with other segmentation methods

[0111] In order to compare the advancement of the method of the present application, the present application compares the method DMI-Net with the most advanced method on the BraTS2020 dataset

[0112]

[0113] The method DMI-Net of the present application is compared with the most advanced method, and the comparison results are shown in Table 2:

[0114] As shown in the table, our method is better than the SOTA method in multiple modalities, and compared with the second best method M3AE, we improve the average Dice coefficient by 2.20%, 1.83%, 3.76% respectively on two datasets, which shows the superiority of the method.

[0115] In order to demonstrate the effectiveness of the modules proposed in the present application, the present application uses the Brats2020 dataset to perform an ablation experiment, and the experimental results are as follows:

[0116] Table 3 Ablation experiment results

[0117]

[0118]

[0119] In Table 3, the present application performs an ablation study on DMI-Net, and sequentially adds the modules and improvements designed by us on the baseline. It can be seen that after adding MASK-AOE, the average Dice coefficient of the three tumor regions is improved by 2.39%, 3.68%, and 3.23%, respectively. This is because MASK-AOE effectively solves the problem of missing modalities through the modal mask and dynamic weight fusion mechanism, and at the same time enhances the complementarity and robustness of the modal features, thereby significantly improving the segmentation performance. After adding BILGII, the average Dice coefficient of the three tumor regions is improved by 1.84%, 3.18%, and 4.58%, respectively. It can be seen that the BILGII module fully excavates the complementary information between modalities through the bidirectional modal interaction and context perception enhancement mechanism, and at the same time optimizes the synergistic relationship between modalities, so that the model performs more robustly in dealing with missing modalities and complex scenes.

[0120] Figure 1 The overall flowchart of the network DMI-Net based on dynamic multi-modal interaction for missing modality brain tumor segmentation; the figure shows the overall architecture of DMI-Net, including a two-stage training process. The first stage enhances the learning ability of modal sharing through consistency constraints, and the second stage improves the adaptability and segmentation performance of the model in the missing modality scene through the fusion of the decoder and the single modality decoder.

[0121] Figure 2 Schematic diagram of dynamic mask weighting module MASK-AOE; the figure describes the structure of the MASK-AOE module, which masks the invalid modalities through a dynamic mask mechanism and retains the effective features. The lightweight agent network (AOE) dynamically fuses multi-level features through dynamic weight, effectively adapting to the missing modality scene.

[0122] Figure 3The schematic diagram of the dual-path interactive fusion module BILGII; the figure shows the design of the BILGII module, which realizes the complementation and optimization of modal features through modal grouping and bidirectional interaction mechanism. The visual manba layer further enhances the global consistency and improves the modal fusion effect.

[0123] The application discloses a dynamic multi-modal interactive network (DMI-Net), which is aimed at the problem of performance decline caused by modal loss in the incomplete multi-modal brain tumor segmentation task. A two-stage dynamic segmentation architecture is proposed, and a dynamic mask weighting module MASK-AOE and a dual-path interactive fusion module BILGII are designed. Through the innovative network design, the application can dynamically adjust the feature contribution of the remaining modal under the condition of modal loss, significantly improving the segmentation accuracy and the adaptability to the modal loss condition. Compared with the prior art, the application has superior segmentation performance, and even under the condition of modal loss, it can still generate high-quality segmentation results, providing a new solution for brain tumor segmentation of incomplete multi-modal data.

[0124] The above is only the preferred embodiment of the application, it should be pointed out that, for those skilled in the art, without departing from the principles of the application, can make a number of improvements and refinements, these improvements and refinements should be considered as the protection scope of the application. The structures, devices and operation methods not specifically described and explained in the application, such as no special description and limitation, are implemented according to the conventional means in the art.

Claims

1. A brain tumor magnetic resonance image segmentation method based on a dynamic multi-modal interaction network, characterized in that: Comprising the following steps: Step 1: In the dynamic multi-modal interaction network, a two-stage segmentation architecture is designed to complete the modal feature extraction and dynamic fusion, and the specific methods of the two stages are as follows: Step 101: Single-modal segmentation stage; The modalities are segmented separately, and the shared information learning between the modalities is constrained by a consistency loss; a FLAIR and T1ce modality encoder and fusion decoder D fuse to independently generate segmentation result predictions one and two, and the similarity of the segmentation results of different modalities in the class probability distribution is constrained by calculating a consistency loss on the predictions one and two. Step 102: Multi-modal fusion segmentation stage; In the multi-modal fusion segmentation stage: containing two decoders: fusion decoder D fuse and single-modal decoder D sep , the fusion decoder D fuse is used for shared decoding of multi-modal features; the fusion decoder D fuse is the same as the one-stage fusion decoder D fuse ; the single-modal decoder D sep decodes the features of each modality into a segmentation prediction result separately; the output results of the fusion decoder and the single-modal decoder are optimized jointly; Step 2: A dynamic mask weighting module is added to the skip connection part of the encoder and decoder, which includes a dynamic mask mechanism and a lightweight proxy network; the mask of the input modal feature and the dynamic fusion; Step 3: A double-path fusion interaction module BILGII is added to the connection part of the encoder and decoder; the double-path fusion interaction module BILGII includes modal grouping fusion, bidirectional interaction mechanism and Vision-Mamba layer (Vision-Mamba); the double-path fusion interaction module BILGII realizes the modeling of the cooperative relationship between modalities and the optimization of features.

2. The method of claim 1, wherein the method is based on a dynamic multimodal interaction network. The step 2 comprises the following sub-steps: Step 201: The invalid modal features are shielded by the dynamic mask mechanism, and only the valid modal features are retained; Step 202: The masked result obtained in step 201 enters the proxy network, and through feature compression, enhancement and expansion, multi-view feature representation is learned, and dynamic weight mechanism is combined to realize weighted fusion of modal features.

3. The method of claim 2, wherein the method further comprises: determining a plurality of brain tumor segmentation masks based on the plurality of brain tumor segmentation masks and the plurality of brain tumor segmentation masks. The step 3 comprises the following steps: Step 301: Group the two groups of modalities flair, t2 and t1ce, t1, respectively, and after splicing, use convolution operation to obtain the modal pairs flair_t2 and t1ce_t1; Step 302: After global pooling of the modal pairs obtained in step 301, the global semantic information is extracted through the fully connected layer, and the scaling factor 1 and the scaling factor 2 are generated, and then the scaling characteristics are obtained by multiplying the modal pairs and the scaling factors, and then the t1ce_t1 scaling characteristics are added to the flair_t2, and the flair_t2 scaling characteristics are added to the t1ce_t1, to realize the information interaction between the two modal pairs.

4. The method of claim 3, wherein the method further comprises: In step 1, a consistency loss is calculated in a first stage to constrain the segmentation result generated by the single-modal encoder, so that the model can learn the shared information between modalities; the consistency loss L con The calculation formula is as follows: Wherein, P1(c|x i P2(c|x) represents the class probability of predicting 1. i ) is the predicted class probability of 2, N is the number of samples, C is the number of classes, and ò = 1e-6.

5. The method of claim 4, wherein: The two-stage in step 1 is decoded by fusion decoder D fuse and the collaborative optimization of the single-modal decoder D sep ; the calculation formula of the fusion loss and the single-modal loss includes cross-entropy loss and DICE loss, wherein the calculation formula of the cross-entropy loss L CE is: N is the total number of samples, C is the total number of classes, wherein, y i,c is the true label of sample i belonging to class c, is the predicted value output by the model; the summations in the numerator and denominator are both summations over all pixel points; DICE loss computes L DICE The formula is: The definitions of the parameters are the same as those in the formula of the cross-entropy loss function.

6. The method of claim 5, wherein the method further comprises: Single modality decoder D sep The single modality segmentation loss Lsmis used to optimize the output of the single modality decoder independently, and the total loss function L of the single modality decoder is as follows: sep The calculation formula is as follows: wherein, represents the output result computed by the single-modal decoder D sep cross-entropy loss of the output result computation, represents the output result computed by the single-modal decoder D sep DICE loss of the output result computation.

7. The method of claim 6, wherein the method further comprises: Fusion decoder D fuse The fusion segmentation loss is used to supervise the optimization of the output of the multi-modal decoder, and the total loss function L fuse The calculation formula is as follows: wherein, represents the output result computed by the fusion decoder D fuse cross-entropy loss computed for the output result, represents the output result computed by the fusion decoder D fuse DICE loss computed for the output result.

8. The method of claim 7, wherein the method further comprises: determining a plurality of brain tumor segmentation masks based on the plurality of brain tumor segmentation masks and the plurality of brain tumor segmentation masks. The formula of the one-stage total loss function is as follows: L I total = L con In the two-stage, the fusion loss and the single-modal branch loss are mainly optimized, and the consistency loss is weakened, and the formula is as follows: L II total = L fuse + L sep .

Images