Brain tumor magnetic resonance image segmentation method based on dynamic multi-modal interaction network

Abstract

The invention belongs to the technical field of multi-modal magnetic resonance image segmentation processing, and particularly relates to a brain tumor magnetic resonance image segmentation method based on a dynamic multi-modal interaction network, which comprises the following steps: step 1, in the dynamic multi-modal interaction network, designing a two-stage segmentation architecture to complete modal feature extraction and dynamic fusion; step 2, adding a dynamic mask weighting module MASK-AOE to a jump connection part of the encoder and the decoder, wherein the dynamic mask weighting module MASK-AOE comprises a dynamic mask mechanism and a lightweight proxy network; mask and dynamic fusion is carried out on input modal features, and the adaptive capacity of the model to the modal missing condition is enhanced; a double-path fusion interaction module is added to the connection part of the encoder and the decoder; the dual-path fusion interaction module comprises modal grouping fusion, a bidirectional interaction mechanism and a visual Mangbar layer; and the dual-path fusion interaction module is used for realizing collaborative relationship modeling and feature optimization between modes. According to the method, a high-quality segmentation result can still be generated even under the condition of mode missing.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal magnetic resonance image segmentation processing, and in particular relates to a brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interactive network. Background Art

[0002] Multimodal magnetic resonance imaging (MRI) has become an important technical means in the task of brain tumor segmentation. Common MRI modalities include T1, T1 enhancement (T1ce), T2, and FLAIR. In brain tumor imaging, T1 clearly shows the anatomical structure of brain tissue, and tumors usually appear as low signals; T1ce highlights the area of ​​blood-brain barrier damage through the injection of contrast agents, and the tumor and its boundaries are clearly enhanced; T2 is highly sensitive to lesions with high water content (such as edema or necrosis around the tumor), and the lesions usually appear as high signals; FLAIR imaging enhances the contrast of lesions while suppressing cerebrospinal fluid signals, and is particularly suitable for showing edema around the tumor or lesions close to the ventricles. Combining these modalities can comprehensively assess the extent, nature, and impact of the tumor on surrounding brain tissue. They provide complementary information such as the anatomical structure of the tumor, edema areas, and active lesions through different imaging mechanisms, providing a key basis for accurate segmentation. In the brain tumor MRI segmentation task, the tumor needs to be divided into three regions: the whole tumor (WT), the tumor core (TC), and the enhancing tumor (ET). These segmentation results are of great significance for clinical diagnosis, prognosis assessment, and preoperative planning. However, in actual clinical applications, MRI modality loss is often encountered due to differences in patient conditions, equipment limitations, or scanning protocols, posing a significant challenge to brain tumor MRI segmentation.

[0003] In response to the above problems, some studies in the prior art have attempted to address the impact of modality loss on segmentation performance. For example, some methods in the prior art simply deal with the problem of modality loss by calculating the mean and variance of the available modality features. However, this method fails to fully capture the complementary information between modalities, limiting the further improvement of segmentation performance. There are also methods in the prior art that use generative adversarial networks (GANs) to generate missing modalities, thereby alleviating the impact of modality loss, but when the modality is severely missing, the quality of the generated results is poor. In addition, the Transformer-based method achieves global modeling through a self-attention mechanism, and exhibits excellent performance in multimodal brain tumor segmentation, but its robustness to modality loss still has room for improvement. The current research methods still have the following key issues:

[0004] 1. Modality sensitivity differences: For example, the FLAIR modality is more sensitive to edema (ED), while the T1ce modality is more sensitive to enhanced tumor (ET). Existing methods fail to explicitly model the correspondence between modality and tumor region characteristics, limiting further improvements in segmentation accuracy.

[0005] 2. Inter-modal feature fusion problem: Most traditional methods use simple feature splicing or addition operations, which makes it difficult to capture complex inter-modal interaction information, resulting in the failure to fully utilize modal complementarity.

[0006] 3. Stability issues under modality loss: The network is prone to bias in certain modality-specific features, which not only affects the stability of feature fusion but also leads to a decline in segmentation performance. Summary of the Invention

[0007] The present invention aims to provide a brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interaction network (DMI-Net), specifically designed for incomplete multimodal brain tumor segmentation tasks. By introducing a two-stage network architecture, a dynamic mask weighting module (MASK-AOE), and a dual-path fusion interaction module (BILGII), DMI-Net significantly improves brain tumor segmentation performance in the absence of modalities. This method addresses issues such as insufficient recognition of modality sensitivity differences, insufficient intermodal feature fusion, and training instability, demonstrating excellent robustness and generalization capabilities.

[0008] The technical solutions adopted by the present invention are as follows:

[0009] A brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interaction network includes the following steps:

[0010] Step 1: In a dynamic multimodal interaction network, a two-stage segmentation architecture is designed to complete modal feature extraction and dynamic fusion. The functions and specific methods of the two stages are as follows:

[0011] Step 101: Single modality segmentation stage;

[0012] The modalities are segmented separately and the shared information learning between the modalities is constrained by consistency loss; FLAIR and T1ce modality encoders and fusion decoders D are used. fuse To independently generate segmentation result predictions 1 and 2, and calculate the consistency loss for predictions 1 and 2 to constrain the similarity of the category probability distribution of segmentation results of different modalities, thereby learning the shared information between modalities and improving the model's adaptability to modality loss.

[0013] Step 102: multimodal fusion segmentation stage;

[0014] In the multimodal fusion segmentation stage: it contains two decoders: fusion decoder D fuse and the unimodal decoder D sep , the fusion decoder D fuse It is used to perform shared decoding on multimodal features to improve the multimodal joint segmentation performance; the fusion decoder D fuse and a one-stage fusion decoder D fuse Similarly, by sharing the decoder, the model can achieve consistent feature decoding logic in single-modal and multi-modal situations, improve segmentation performance, and enhance the adaptability to modality loss; single-modal decoder D sep The features of each modality are decoded separately into segmentation prediction results. This allows the model to independently evaluate the contribution of each modality to the segmentation task. At the same time, the independence of the multimodal segmentation results provides a stronger foundation for the model's fusion strategy. By jointly optimizing the output results of the fusion decoder and the single-modality decoder, the multimodal fusion segmentation stage further improves segmentation performance and enhances the network's robustness to modality loss.

[0015] Step 2: A dynamic mask-weighted module, MASK-AOE, is added to the jump connection between the encoder and decoder. The dynamic mask-weighted module MASK-AOE includes a dynamic masking mechanism and a lightweight proxy network. It masks and dynamically fuses the input modality features to enhance the model's adaptability to modality loss.

[0016] Step 3: A dual-path fusion interaction module (BILGII) is added to the connection between the encoder and decoder. The dual-path fusion interaction module (BILGII) includes modal group fusion, a bidirectional interaction mechanism, and a vision-mamba layer. The dual-path fusion interaction module (BILGII) implements collaborative relationship modeling and feature optimization between modalities.

[0017] Preferably, step 2 includes the following sub-steps:

[0018] Step 201: Invalid modal features are shielded through a dynamic masking mechanism, retaining only valid modal features, thereby forcing the network to complete the segmentation task using incomplete modalities and improving the robustness of the model;

[0019] Step 202: The masked result obtained in step 201 enters the proxy network, which learns multi-view feature representation through feature compression, enhancement and expansion, and combines the dynamic weight mechanism to achieve weighted fusion of modal features, thereby enhancing the complementarity between modalities.

[0020] Preferably, the step 3 comprises the following steps:

[0021] Step 301: Group the two modal pairs flair, t2 and t1ce, t1, concatenate them separately, and then use convolution operations to obtain the modal pairs flair_t2 and t1ce_t1. Processing modal pairs allows the model to explicitly learn the synergy between modalities, rather than simply relying on subsequent global feature stacking.

[0022] Step 302: After global pooling of the modality pair obtained in step 301, the global semantic information is extracted through the fully connected layer, and scaling factors 1 and 2 are generated. The scaling features are obtained by multiplying the modality pair and the scaling factors. Then, the scaling features are added to add the t1ce_t1 scaling features to flair_t2, and the flair_t2 scaling features to t1ce_t1 to realize information interaction between the two modality pairs. This interaction operation between the modality pairs can realize information sharing, so that the modality pairs can complement each other in missing information.

[0023] Preferably, in step 1, the segmentation results generated by the unimodal encoder are constrained by calculating the consistency loss in one stage, so that the model can learn the shared information between the modalities. The consistency loss L con The calculation formula is as follows:

[0024]

[0025] Among them, P1(c|x i ) is the probability of predicting the class one, P2(c|x i ) is the probability of predicting the second category, N is the number of samples, C is the number of categories, and ò = 1e-6 is used to prevent numerical instability.

[0026] Preferably, in the second stage of step 1, the fusion decoder D fuse and the unimodal decoder D sep The collaborative optimization of multimodal features improves the comprehensive segmentation performance of multimodal features; the calculation formula of the fusion loss and the single modal loss includes the cross entropy loss and the DICE loss, where the cross entropy loss L CE The calculation formula is:

[0027]

[0028] N is the total number of samples, C is the total number of categories, where y i,c is the true label of sample i belonging to category c, is the predicted value output by the model. The sums in the numerator and denominator are the sums of the pixels;

[0029] DICE loss calculation L DICE The formula is:

[0030]

[0031] The definition of each parameter is the same as that in the cross entropy loss function formula.

[0032] Preferably, the single-mode decoder D sep The unimodal segmentation loss is used to independently optimize the output of the unimodal decoder, and its total loss function L sep The calculation formula is as follows:

[0033]

[0034] in, Represented by the unimodal decoder D sep Output the calculated cross entropy loss, Represented by the unimodal decoder D sep Output the calculated DICE loss.

[0035] Preferably, the fusion decoder D fuse The fusion segmentation loss is used to supervise the output of the multimodal decoder and its total loss function L fuse The calculation formula is as follows:

[0036]

[0037] in, Represented by the fusion decoder D fuse Output the calculated cross entropy loss, Represented by the fusion decoder D fuse Output the calculated DICE loss.

[0038] Preferably, the total loss function of the dynamic multimodal interactive segmentation network proposed in the present invention combines consistency loss, fusion loss and single-modal branch loss, and is optimized in stages. The total loss function calculation formula for one stage is as follows:

[0039] L I total =L con

[0040] In the second stage, the fusion loss and single-modal branch loss are mainly optimized, and the consistency loss is weakened. The formula is as follows:

[0041] L II total =L fuse +L sep .

[0042] The technical effects achieved by the present invention are:

[0043] The present invention discloses a dynamic multimodal interaction network (DMI-Net). This method addresses the problem of decreased segmentation performance due to modality loss in incomplete multimodal brain tumor segmentation tasks, proposes a two-stage dynamic segmentation architecture, and designs a dynamic mask weighting module MASK-AOE and a dual-path interaction fusion module BILGII. Through innovative network design, the present invention can dynamically adjust the feature contribution of the remaining modalities in the case of modality loss, significantly improving the segmentation accuracy and adaptability to modality loss. Compared with the existing technology, the present invention has superior segmentation performance and can generate high-quality segmentation results even under modality loss conditions, providing a new solution for brain tumor segmentation with incomplete multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the overall flow chart of the present invention's missing modality brain tumor magnetic resonance image segmentation based on the dynamic multimodal interaction network DMI-Net;

[0045] Figure 2 Schematic diagram of the dynamic mask weighting module MASK-AOE in the present invention;

[0046] Figure 3 It is a schematic diagram of the dual-path fusion interaction module BILGII in the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is described in detail below with reference to the following examples. It should be understood that the following text is only used to describe one or more specific embodiments of the present invention and does not strictly limit the scope of protection of the present invention.

[0048] like Figure 1-Figure 3 As shown, the brain tumor magnetic resonance image segmentation method based on dynamic multimodal interaction network includes the following steps:

[0049] Step 1: In a dynamic multimodal interaction network, a two-stage segmentation architecture is designed to complete modal feature extraction and dynamic fusion. The functions and specific methods of the two stages are as follows:

[0050] Step 101: Single modality segmentation stage;

[0051] The modalities are segmented separately and the shared information learning between the modalities is constrained by consistency loss; FLAIR and T1ce modality encoders and fusion decoders D are used. fuseTo independently generate segmentation result predictions 1 and 2, and calculate the consistency loss for predictions 1 and 2 to constrain the similarity of the category probability distribution of segmentation results of different modalities, thereby learning the shared information between modalities and improving the model's adaptability to modality loss.

[0052] Step 102: multimodal fusion segmentation stage;

[0053] In the multimodal fusion segmentation stage: it contains two decoders: fusion decoder D fuse and the unimodal decoder D sep , fusion decoder D fuse Used to share and decode multimodal features to improve multimodal joint segmentation performance; fusion decoder D fuse and a one-stage fusion decoder D fuse Similarly, by sharing the decoder, the model can achieve consistent feature decoding logic in single-modal and multi-modal situations, improve segmentation performance, and enhance the adaptability to modality loss; single-modal decoder D sep The features of each modality are decoded separately into segmentation prediction results. This allows the model to independently evaluate the contribution of each modality to the segmentation task. At the same time, the independence of the multimodal segmentation results provides a stronger foundation for the model's fusion strategy. By jointly optimizing the output results of the fusion decoder and the single-modality decoder, the multimodal fusion segmentation stage further improves segmentation performance and enhances the network's robustness to modality loss.

[0054] Step 2: A dynamic mask-weighted AOE module is added to the jump connection between the encoder and decoder. The dynamic mask-weighted AOE module includes a dynamic masking mechanism and a lightweight proxy network. It combines the masking and dynamic fusion of input modal features to enhance the model's adaptability to modality loss.

[0055] Step 3: A dual-path fusion interaction module (BILGII) is added to the connection between the encoder and decoder. The dual-path fusion interaction module (BILGII) includes modal group fusion, a bidirectional interaction mechanism, and a vision-mamba layer. The dual-path fusion interaction module (BILGII) implements collaborative relationship modeling and feature optimization between modalities.

[0056] Preferably, step 2 includes the following sub-steps:

[0057] Step 201: Invalid modal features are shielded through a dynamic masking mechanism, retaining only valid modal features, thereby forcing the network to complete the segmentation task using incomplete modalities and improving the robustness of the model;

[0058] Step 202: The masked result obtained in step 201 enters the proxy network, which learns multi-view feature representation through feature compression, enhancement and expansion, and combines the dynamic weight mechanism to achieve weighted fusion of modal features, thereby enhancing the complementarity between modalities.

[0059] Preferably, step 3 comprises the following steps:

[0060] Step 301: Group the two modal pairs flair, t2 and t1ce, t1, concatenate them separately, and then use convolution operations to obtain the modal pairs flair_t2 and t1ce_t1. Processing modal pairs allows the model to explicitly learn the synergy between modalities, rather than simply relying on subsequent global feature stacking.

[0061] Step 302: After global pooling of the modality pair obtained in step 301, the fully connected layer is used to extract global semantic information and generate scaling factors 1 and 2. The scaling features are obtained by multiplying the modality pair and the scaling factors. Then, the scaling features are added to add the t1ce_t1 scaling features to flair_t2 and the flair_t2 scaling features to t1ce_t1 to realize information interaction between the two modality pairs. This interaction between the modality pairs can realize information sharing, so that the modality pairs can complement each other in missing information.

[0062] Preferably, in step 1, the segmentation results generated by the unimodal encoder are constrained by calculating the consistency loss in one stage, so that the model can learn the shared information between the modalities. con The calculation formula is as follows:

[0063]

[0064] Among them, P1(c|x i ) is the probability of predicting the class one, P2(c|x i ) is the probability of predicting the second category, N is the number of samples, C is the number of categories, and ò = 1e-6 is used to prevent numerical instability.

[0065] Preferably, in step 1, the second stage is performed by fusion decoder D fuse and the unimodal decoder D sep The collaborative optimization of multimodal features improves the comprehensive segmentation performance of multimodal features; the calculation formula of fusion loss and single modal loss includes cross entropy loss and DICE loss, where cross entropy loss L CE The calculation formula is:

[0066]

[0067] N is the total number of samples, C is the total number of categories, where y i,c is the true label of sample i belonging to category c, is the predicted value output by the model. The sums in the numerator and denominator are the sums of the pixels;

[0068] DICE loss calculation L DICE The formula is:

[0069]

[0070] The definition of each parameter is the same as that in the cross entropy loss function formula.

[0071] Preferably, the single-mode decoder D sep The unimodal segmentation loss is used to independently optimize the output of the unimodal decoder, and its total loss function L sep The calculation formula is as follows:

[0072]

[0073] in, Represented by the unimodal decoder D sep Output the calculated cross entropy loss, Represented by the unimodal decoder D sep Output the calculated DICE loss.

[0074] Preferably, the fusion decoder D fuse The fusion segmentation loss is used to supervise the output of the multimodal decoder and its total loss function L fuse The calculation formula is as follows:

[0075]

[0076] in, Represented by the fusion decoder D fuse Output the calculated cross entropy loss, Represented by the fusion decoder D fuse Output the calculated DICE loss.

[0077] Preferably, the total loss function of the dynamic multimodal interactive segmentation network proposed in the present invention combines consistency loss, fusion loss and single-modal branch loss, and is optimized in stages. The total loss function calculation formula for one stage is as follows:

[0078] L I total =L con

[0079] In the second stage, the fusion loss and single-modal branch loss are mainly optimized, and the consistency loss is weakened. The formula is as follows:

[0080] L II total =Lfuse +L sep .

[0081] In practical use, the present invention:

[0082] In the unimodal segmentation stage (Stage I):

[0083] This stage mainly performs segmentation on the modalities independently and constrains the shared information learning between the modalities through consistency loss.

[0084] First, a separate encoder is designed for each MRI modality to extract high-quality modality-specific features. The encoder consists of multiple layers of convolutional layers, pooling layers, and skip connections, capturing both local and global information about the modality.

[0085] Then, the decoder is optimized by consistency constraints to ensure that the segmentation results of each modality remain consistent in the category probability distribution. Specifically, for the segmentation results generated by each modality, a consistency loss L is introduced. con :

[0086]

[0087] Where P1(c|x i ) is the class probability of prediction 1, P2(c|x i ) is the class probability of prediction 2, N is the number of samples, C is the number of categories, and ò = 1e-6 is used to prevent numerical instability.

[0088] Finally, through consistency loss, the network can learn the shared characteristics between modalities, enhance its adaptability to modality loss, and at the same time ensure the independent segmentation capability of each modal feature.

[0089] In the multimodal fusion segmentation stage (Stage II):

[0090] This stage mainly realizes the dynamic aggregation and shared decoding of multimodal features through the collaborative optimization of the fusion decoder and the single-modal decoder, thereby improving the segmentation performance and enhancing the network's adaptability to modality loss.

[0091] First, a dynamic masking mechanism is designed for the dynamic mask weighting module MASK-AOE, which applies a dynamic masking operation to the features of the input modality. A mask is set for invalid modal features, setting them to zero and retaining only valid modal features. The dynamic masking mechanism can shield missing modal features during training and inference, forcing the network to use incomplete modalities to complete the segmentation task. Then, a proxy network is designed to compress, enhance, and expand the masked modal features to generate multi-view enhanced features. The proxy network learns the complementary information between modalities through convolution operations, thereby improving the representation ability of features. Finally, dynamic weight calculation is performed to calculate the weight of each modal feature. The formula is as follows:

[0092]

[0093] where w i represents the weight of the i-th branch, It represents the weight score of each branch after a series of feature extraction and calculation by normalization (softmax) function. Then use the obtained weight to weight each branch:

[0094] exp out =w1·aoe1+w2·aoe2+w3·aoe3

[0095] aoe1, aoe2, aoe3 are enhanced features, exp out is the weighted output of the branch. By combining the mask with the dynamic weight mechanism, the dynamic mask weighting module MASK-AOE can achieve robust and efficient multimodal feature processing in multimodal medical image segmentation.

[0096] The decoder at this stage includes the fusion decoder D fuse and the unimodal decoder D sep , fusion decoder D fuse Used for shared decoding of multimodal features, generating segmentation results through multi-layer convolution operations. Single-modal decoder D sep Providing an independent decoding path for each modality ensures that the features of each modality can contribute independently to the segmentation task.

[0097] In the dual-path fusion interaction module BILGII

[0098] This module explicitly models the collaborative relationship between modalities through modal grouping fusion, bidirectional interaction mechanism and global feature modeling, and improves the global consistency and segmentation performance of fused features by capturing the long-range dependency of modalities on features, thereby enhancing the robustness of the network in the absence of modalities.

[0099] The first is modality grouping fusion, which groups modalities with similar characteristics, such as FLAIR-T2 and T1ce-T1, and generates modality pair features through convolution operations; such as FLAIR-T2 modality pair features.

[0100] Then, a two-way interactive operation is performed to add the features of one modality pair to the other modality pair, achieving information sharing between the modality pairs. By supplementing the missing information, the consistency of global features and segmentation performance are enhanced.

[0101] Finally, the Vision-Mamba layer captures long-range dependencies between modal pairs of features and optimizes the collaborative information fusion capabilities between modalities. The Vision-Mamba layer uses a multi-head attention mechanism to model the global relationship between modal pairs of features, further improving feature fusion accuracy.

[0102] The present invention is in the experimental process:

[0103] The experimental platform of the present invention is shown in Table 1. The experiments were conducted and the results were evaluated on the BraTS (Brain Tumor Segmentation) dataset, including two versions: BraTS2018 and BraTS2020. Each case in the dataset includes four modalities: FLAIR, T1ce, T1, and T2.

[0104] BraTS2018 contains 285 cases, and the data is divided into a training set of 199 cases, a validation set of 29 cases, and a test set of 57 cases.

[0105] Evaluation indicators: This paper uses the Dice coefficient as the main evaluation indicator to measure the similarity between the segmentation result and the true label, which is defined as:

[0106]

[0107] in, is the number of pixels of the predicted segmentation result, |y| is the number of pixels of the true label, The number of pixels in the intersection area between the predicted segmentation result and the true label. Its value range is 0 to 1. The closer the value is to 1, the higher the similarity between the segmentation result and the true label, that is, the higher the segmentation accuracy of the model.

[0108] Table 1 Experimental platform

[0109]

[0110] Comparison with other segmentation methods

[0111] In order to compare the advancedness of the method of the present invention, the present invention is based on the BraTS2020 dataset.

[0112]

[0113] The proposed method DMI-Net is compared with the most advanced methods. The comparison results are shown in Table 2:

[0114] As shown in the table, our method outperforms the SOTA method in multiple modalities. Compared with the second-best method M3AE, our average Dice coefficient on the two datasets is improved by 2.20%, 1.83%, and 3.76%, respectively, which demonstrates the superiority of our method.

[0115] In order to demonstrate the effectiveness of the module proposed in this paper, we conducted an ablation experiment using the Brats2020 dataset. The experimental results are as follows:

[0116] Table 3 Ablation experiment results

[0117]

[0118]

[0119] In Table 3, we performed an ablation study on DMI-Net, sequentially adding our designed modules and improvements to the baseline. We show that after adding MASK-AOE, the average Dice coefficients of the three tumor regions increased by 2.39%, 3.68%, and 3.23%. This is because MASK-AOE effectively addresses the problem of modality-missing scenarios through modality masking and dynamic weight fusion, while also enhancing the complementarity and robustness of modal features, significantly improving segmentation performance. After adding BILGII, the average Dice coefficients of the three tumor regions increased by 1.84%, 3.18%, and 4.58%. This shows that the BILGII module, through bidirectional modality pair interaction and context-aware enhancement, fully exploits the complementary information between modalities and optimizes the synergistic relationship between modalities, making the model more robust in handling modality-missing and complex scenarios.

[0120] Figure 1 A flowchart of the dynamic multimodal interaction network DMI-Net for modality-missing brain tumor segmentation. This diagram illustrates the overall architecture of DMI-Net, which includes a two-stage training process. The first stage enhances modality sharing learning capabilities through consistency constraints, while the second stage improves the model's adaptability and segmentation performance in modality-missing scenarios by integrating a decoder and a single-modality decoder.

[0121] Figure 2 Schematic diagram of the dynamic mask-weighted AOE module; this diagram illustrates the structure of the MASK-AOE module, which uses a dynamic masking mechanism to block invalid modalities while retaining valid features. The lightweight agent network (AOE) fuses multi-level features using dynamic weighting, effectively adapting to scenarios where modalities are missing.

[0122] Figure 3Schematic diagram of the dual-path fusion interaction module BILGII. This diagram illustrates the design of the BILGII module, which complements and optimizes modal features through modal grouping and a bidirectional interaction mechanism. The Visual Mamba layer further enhances global consistency and improves modal fusion.

[0123] The present invention discloses a dynamic multimodal interaction network (DMI-Net). This method addresses the problem of decreased segmentation performance due to modality loss in incomplete multimodal brain tumor segmentation tasks, proposes a two-stage dynamic segmentation architecture, and designs a dynamic mask weighting module MASK-AOE and a dual-path interaction fusion module BILGII. Through innovative network design, the present invention can dynamically adjust the feature contribution of the remaining modalities in the case of modality loss, significantly improving the segmentation accuracy and adaptability to modality loss. Compared with the existing technology, the present invention has superior segmentation performance and can generate high-quality segmentation results even under modality loss conditions, providing a new solution for brain tumor segmentation with incomplete multimodal data.

[0124] The foregoing is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained herein shall, unless otherwise specified or limited, be implemented in accordance with conventional means in the art.

Claims

1. A brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interactive network, characterized by: The following steps are involved: Step 1: In a dynamic multimodal interaction network, a two-stage segmentation architecture is designed to complete modal feature extraction and dynamic fusion. The specific methods of the two stages are as follows: Step 101: Single modality segmentation stage; The modalities are segmented separately and the shared information learning between the modalities is constrained by consistency loss; FLAIR and T1ce modality encoders and fusion decoders D are used. fuse To independently generate segmentation result prediction 1 and prediction 2, and calculate the consistency loss for prediction 1 and prediction 2 to constrain the similarity of the category probability distribution of different modal segmentation results; Step 102: multimodal fusion segmentation stage; In the multimodal fusion segmentation stage: it contains two decoders: fusion decoder D fuse and the unimodal decoder D sep , the fusion decoder D fuse Used to perform shared decoding on multimodal features; the fusion decoder D fuse and a one-stage fusion decoder D fuse Same; Single-mode decoder D sep Decode the features of each modality separately into segmentation prediction results; and jointly optimize the output of the fusion decoder and the single-modality decoder; Step 2: A dynamic mask weighting module is added to the jump connection part of the encoder and decoder. The dynamic mask weighting module includes a dynamic mask mechanism and a lightweight proxy network; the mask and dynamic fusion of the input modality features; Step 3: A dual-path fusion interaction module (BILGII) is added to the connection between the encoder and decoder. The dual-path fusion interaction module (BILGII) includes modal group fusion, a bidirectional interaction mechanism, and a vision-mamba layer. The dual-path fusion interaction module (BILGII) implements collaborative relationship modeling and feature optimization between modalities.

2. The brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interactive network according to claim 1, characterized in that: The step 2 includes the following sub-steps: Step 201: shield invalid modal features through a dynamic masking mechanism, and only retain valid modal features; Step 202: The masked result obtained in step 201 enters the proxy network, which learns multi-view feature representation through feature compression, enhancement and expansion, and realizes weighted fusion of modal features in combination with a dynamic weight mechanism.

3. The brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interactive network according to claim 2, characterized in that: The step 3 comprises the following steps: Step 301: Group the two modal pairs flair, t2 and t1ce, t1, concatenate them respectively, and then use convolution operation to obtain the modal pairs flair_t2 and t1ce_t1; Step 302: After global pooling of the modal pair obtained in step 301, the global semantic information is extracted through a fully connected layer, and scaling factors 1 and 2 are generated. The modal pair and the scaling factors are multiplied to obtain scaling features. Then, the scaling features are added to add the t1ce_t1 scaling features to flair_t2, and the flair_t2 scaling features to t1ce_t1, so as to realize information interaction between the two modal pairs.

4. The brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interactive network according to claim 3, characterized in that: In step 1, the segmentation results generated by the single-modal encoder are constrained by calculating the consistency loss in one stage, so that the model can learn the shared information between modalities; the consistency loss L con The calculation formula is as follows: Among them, P1(c|x i ) is the probability of predicting the class one, P2(c|x i ) is the probability of predicting class 2, N is the number of samples, C is the number of classes, and ò = 1e-6.

5. The brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interactive network according to claim 4, characterized in that: In the second stage of step 1, the fusion decoder D fuse and the unimodal decoder D sep The calculation formula of the fusion loss and the single modal loss includes the cross entropy loss and the DICE loss, where the cross entropy loss L CE The calculation formula is: N is the total number of samples, C is the total number of categories, where y i,c is the true label of sample i belonging to category c, is the predicted value output by the model; the sums in the numerator and denominator are the sums of the pixels; DICE loss calculation L DICE The formula is: The definition of each parameter is the same as that in the cross entropy loss function formula.

6. The brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interactive network according to claim 5, characterized in that: Single-mode decoder D sep The unimodal segmentation loss is used to independently optimize the output of the unimodal decoder, and its total loss function L sep The calculation formula is as follows: in, Represented by the unimodal decoder D sep Output the calculated cross entropy loss, Represented by the unimodal decoder D sep Output the calculated DICE loss.

7. The brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interactive network according to claim 6, characterized in that: Fusion decoder D fuse The fusion segmentation loss is used to supervise the output of the multimodal decoder and its total loss function L fuse The calculation formula is as follows: in, Represented by the fusion decoder D fuse Output the calculated cross entropy loss, Represented by the fusion decoder D fuse Output the calculated DICE loss.

8. The brain tumor magnetic resonance image segmentation method based on a dynamic multimodal interactive network according to claim 7, characterized in that: The calculation formula of the total loss function in one stage is as follows: L I total =L con In the second stage, the fusion loss and single-modal branch loss are mainly optimized, and the consistency loss is weakened. The formula is as follows: L II total =L fuse +L sep 。

Images