Segmentation method for incomplete multi-modal medical image

Abstract

The invention relates to a segmentation method for an incomplete multi-modal medical image. The segmentation method comprises the following steps: selecting a public incomplete multi-modal MRI image data set; a segmentation model UDMAF is constructed based on the U-NET; selecting an MRI sample from the multi-modal MRI image data set, and inputting the MRI sample into an encoder-decoder group in the UDMAF to obtain a single-modal feature; obtaining single-mode specific representation of the single-mode features through a decoder; inputting all the single-mode features into DMAF to obtain multi-mode fusion features, and decoding the multi-mode fusion features layer by layer through a fusion decoder to obtain multi-mode specific representation; the multi-modality specific representation is a segmentation result of the medical image of the incomplete modality. By using the method, the segmentation work of the incomplete multi-modal medical image can be completed under the condition that the modal missing rate is unbalanced.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation, and in particular to a segmentation method for incomplete multimodal medical images. Background Technology

[0002] In clinical practice, multimodal imaging techniques such as magnetic resonance imaging (MRI) are being increasingly adopted because the complementary information across modalities can significantly improve segmentation performance. However, obtaining complete MRI datasets in real-world scenarios remains challenging due to factors such as data corruption, image degradation, and motion-related artifacts. Therefore, despite significant progress in multimodal medical segmentation, its practical clinical deployment continues to face obstacles in the absence of modalities.

[0003] Current research on incomplete multimodal segmentation mainly focuses on three methodological directions. The most direct approach utilizes generative adversarial networks or diffusion-based image inpainting techniques to synthesize missing modalities using complete cross-modal data, achieving full-modal segmentation by generating synthetic images. Another common strategy employs knowledge distillation, transferring diagnostic patterns from a fully trained teacher model (using complete modalities) to a specialized student model, enabling it to adapt to scenarios with missing modalities. The current state-of-the-art paradigm uses shared representation learning, training a unified segmentation model through cross-modal collaborative space mapping. This architecture combines a modality-specific encoder with a shared decoder, achieving superior operational flexibility while maintaining computational efficiency compared to the aforementioned two research directions. However, existing research often prioritizes optimizing segmentation performance, oversimplifying training conditions and neglecting the inherent modality imbalance problem in multimodal tasks. In clinical practice, this imbalance manifests as two interrelated challenges: modality missing rate imbalance and modality contribution heterogeneity, complicating the development of robust and generalizable models.

[0004] Imbalanced modality missing rates refer to the difference in missing rates among different modalities in real-world scenarios. For example, in MRI modalities, T1-weighted sequences have lower missing rates due to their robustness to motion artifacts, while T2-weighted sequences have higher missing rates due to their susceptibility to patient movement and higher acquisition costs. This difference causes the model to prioritize learning features of high-frequency modalities (i.e., common modalities), resulting in insufficient representation of low-frequency modalities (i.e., rare modalities). The dominant modality not only dominates gradient updates but also inhibits the learning process of other modalities, leading to different convergence speeds among modalities. Ultimately, this weakens the potential advantages of multimodal learning, causing it to perform even worse than single-modal methods in certain diagnostic scenarios—a phenomenon known as "modality competition." It is worth noting that most existing modality rebalancing techniques assume "Perfect Data Training" (PDT) conditions, meaning all modalities have the same missing rate. Typical PDT implementations either randomly mask modalities at each training epoch (ensuring the periodicity of modalities is visible) or pre-mask modalities before training. However, this idealized assumption conflicts with the "Imperfect Data Training" (IDT) scenario—which is closer to clinical reality, where modalities have heterogeneous missing rates and remain invisible once masked.

[0005] Modal contribution heterogeneity stems from the unequal importance of different modalities in diagnosis. Although existing methods implicitly assume that all modalities are equally important, their actual contributions to the segmentation task differ significantly. This oversight leads to underdevelopment or distortion of key pathological features in the embedding space, thereby impairing fusion performance. For example, some modalities may contain richer discriminative information about specific structures (such as the tumor core), but this potential cannot be utilized due to the inflexible weighting mechanisms of existing frameworks.

[0006] Recent work has made significant progress in modality rebalancing; however, it still has some limitations. First, static fusion mechanisms (such as feature concatenation) cannot dynamically suppress noise from missing modalities, nor can they adaptively adjust the contribution weights of each modality. Second, traditional knowledge distillation methods only partially alleviate the imbalance problem by mimicking the output probability distribution of the teacher model (i.e., soft labels), but ignore the semantic structural relationships across modalities. Finally, some methods use loss reweighting or gradient scaling techniques to balance the convergence speed of different modalities, but these methods are mainly aimed at classification tasks rather than segmentation tasks. Summary of the Invention

[0007] In view of the above-mentioned problems in the existing technology, the technical problem to be solved by the present invention is: how to improve the segmentation accuracy of incomplete modal medical images.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A segmentation method for incomplete multimodal medical images includes the following steps:

[0010] S100: Select a publicly available, incomplete multimodal MRI image dataset. The MRI image dataset contains N MRI samples, each MRI sample consists of M modalities, and each modality contains P pixels. The nth MRI sample is denoted as... m represents the presence of m modalities, and p represents the presence of p pixels; where n∈{1,2,…,N}, m∈{1,2,…,M}, and p∈{1,2,…,P}; all modalities contained in the same sample have the same true segmentation label;

[0011] S200: Building a segmentation model based on U-NET DMAF U DMAF It includes a feature extraction module, a feature fusion module, and a segmentation result output module;

[0012] The feature extraction module includes several sets of encoder-decoder pairs. The feature extraction module is used to extract single-modal features of each modality of the MRI image and obtain single-modal specific characterization.

[0013] The feature fusion module includes a Dynamic Modality Aware Fusion Module (DMAF) and a fusion decoder. The feature fusion module is used to fuse and decode all single-modality feature representations of MRI images.

[0014] The segmentation result output module is used to output the segmentation results of the MRI image;

[0015] S300: Select the i-th MRI sample from the multimodal MRI image dataset. One modality corresponds to one encoder-decoder set. m i Each mode is accessed via m i Each encoder generates a single-modal feature corresponding to each modality. l represents the l-th convolutional layer in the encoder, and s represents It contains the s-th mode and s∈{1,2,…,m} i};

[0016] S400: Each m is obtained through their respective decoders. i Each modality corresponds to a specific single-modal representation. Where “1” represents the first convolutional layer in the decoder;

[0017] All of In the input feature fusion module, multimodal fusion features f are obtained through DMAF. i,L Where L represents the number of convolutional layers in DMAF, and f i,L The multimodal specific representation z is obtained by layer-by-layer decoding using a fusion decoder. i,1 , where “1” represents the first convolutional layer in the fusion decoder;

[0018] z i,1 That is, medical images with incomplete modalities. The segmentation results;

[0019] S500: Constructing the segmentation model U DMAF loss function Using all samples in the multimodal MRI image dataset as U DMAF The input is processed using the AdamW optimizer tool and the dynamic training monitoring strategy DTM to optimize U. DMAF Training is performed, and U is updated in reverse using gradient descent. DMAF The parameters are set, and training stops when the maximum number of iterations is reached, resulting in a trained segmentation model U′. DMAF ;

[0020] S600: Input the MRI image X to be predicted into U′ DMAF The output is the medical image prediction and segmentation result of X.

[0021] Preferably, in S200, the Dynamic Modality Awareness Fusion (DMAF) module is a combination of the Transformer attention mechanism and a dynamic masking strategy.

[0022] Preferably, the segmentation model U is constructed in S500. DMAF loss function The steps are as follows:

[0023] S510: Combine Dice loss with weighted cross-entropy loss to form a combined loss. and Jointly construct a single-mode loss function The calculation expression is as follows:

[0024]

[0025] Among them, I n,m This indicates whether mode m exists in sample n. y represents the single-modal representation of a sample n containing m modalities in the final output layer. n Indicates a separator label;

[0026] S520: will With z n,lJointly construct a multimodal loss function The calculation expression is as follows:

[0027]

[0028] Where l represents the l-th convolutional layer, l=1 represents the final output layer; L represents the total number of convolutional layers; λ l Indicates the layer loss weights of layer l; Indicates pressing 2 l Upsampling is performed by a factor of 1; z n,l This represents the multimodal representation of the nth sample in layer l;

[0029] S530: Constructing the relational distillation loss function The calculation expression is as follows:

[0030]

[0031] Where α1 represents the learnable equilibrium hyperparameter, and |M| represents the number of modes. This represents the covariance loss function in relational distillation. Represents the attention loss function in relational distillation;

[0032] S540: Constructing a prototype distillation loss function using single-modal prototype pixels and fused-modal prototype pixels. The calculation expression is as follows:

[0033]

[0034] in, Indicates a single-modal prototype pixel. f represents the prototype pixel of the fused modality. n,L (·) indicates a multimodal fusion feature. Ω represents a single-modal characteristic. c Represents the set of pixels of category c, ∈ is used to prevent division by zero error, τ m This indicates temperature hyperparameters;

[0035] S550: Constructing the segmentation model U DMAF loss function The calculation expression is as follows:

[0036]

[0037] Among them, λ1, λ2, λ3, and λ4 are all model equilibrium hyperparameters.

[0038] Preferably, the relational distillation loss function is constructed in S530. The steps are as follows:

[0039] S531: Relational distillation is constructed using covariance consistency alignment (CCA) and masked attention alignment (MAA), and f is calculated using CCA. n,l and The spatial covariance matrix Cov(·):

[0040]

[0041] in, This represents the fused multimodal or single-modal features, where μ represents the feature average.

[0042] Calculate the covariance loss function The calculation expression is as follows:

[0043]

[0044] in, Represents a linear mapping;

[0045] S532: MAA adopts single-modal features As the query matrix Q, the fused multimodal features f are used. n,L As key-value pairs K and V, attention weight matrices are generated through the multi-head attention mechanism MHA(·). The calculation expression is as follows:

[0046]

[0047] Among them, M mask ∈{-∞,0} represents an attention mask generated based on the modality existence matrix C;

[0048] Calculate the attention loss function The expression is as follows:

[0049]

[0050] S533: Utilization and Calculated

[0051]

[0052] Preferably, the content of the Dynamic Training Monitoring Strategy (DTM) in S500 is as follows:

[0053] First, calculate the distillation difference g. r (·) and the difference between the prototype distillation and g p (·), the calculation formula is as follows:

[0054]

[0055] Where α1 represents the learnable equilibrium hyperparameter;

[0056] Then calculate the relationship between m and distillation difference at time step t. Difference from prototype distillation The calculation formula is as follows:

[0057]

[0058] Where, α decay The adaptive attenuation coefficient is calculated using the following formula:

[0059]

[0060] α decay =0.9·(1-S(ratio) m ))

[0061] Where S(·) represents the sigmoid function, and ∈ is used to prevent errors caused by division by zero;

[0062] The total difference is then calculated using a weighted summation method.

[0063]

[0064] Where α2 represents the learnable equilibrium hyperparameter, Historical exponential moving average representing the relationship distillation gap. Historical exponential moving average representing the prototype distillation gap. This represents the general notation, specifically the EMA for all types of distillation gaps. EMA stands for Exponential Moving Average.

[0065] Then, to Inverse scaling yields w (t) (m), the calculation expression is as follows:

[0066]

[0067] Among them, w (t) (m) represents the weight value in the adversarial weighting mechanism, and m′ represents the traversal symbol;

[0068] Application w (t) (m) Calculate the single-mode segmentation loss The weighted formula is defined as follows:

[0069]

[0070] For the encoder Encoder(m) corresponding to the m-th mode, its gradient descent update mechanism is as follows:

[0071]

[0072] in, γ represents the encoder gradient of mode m at time step t. (t) (m) represents the gradient scaling factor that is inversely proportional to the distillation difference of mode m, sim (t) (m) represents the cosine similarity between continuous gradient directions;

[0073] When sim (t) When (m) < -0.5, it indicates that there is a conflict in the gradient direction during the iteration process. At this time, the update magnitude will be adaptively reduced by a factor of 0.7.

[0074] Compared with the prior art, the present invention has at least the following advantages:

[0075] This invention proposes a Dynamic Modality-Aware Fusion Network (DMAF-Net) for incomplete multimodal segmentation under conditions of imbalanced modality missing rates. First, it introduces a Dynamic Modality-Aware Fusion (DMAF) module, which dynamically assigns weights to the specific contributions of each modality while masking missing modalities, thereby improving the quality of subsequent multimodal fusion features. Then, based on this, a collaborative framework is developed, integrating relational distillation and prototype distillation for multimodal segmentation. Finally, it proposes a Dynamic Training Monitoring (DTM) strategy, which dynamically tracks modality-specific distillation gaps and adaptively adjusts loss weights and gradient scaling factors to balance the convergence speed of each modality during training.

[0076] 1. This invention proposes a Dynamic Modality-Aware Fusion (DMAF) module, which integrates a Transformer-based attention mechanism with a dynamic masking strategy. This design can adaptively weight modal contributions while suppressing interference from missing modalities, overcoming the limitations of static fusion approaches in existing methods.

[0077] 2. This invention introduces a two-level distillation framework that synergistically combines relational distillation and prototype distillation to achieve complementary alignment of global statistical information and local details, and enforces cross-modal semantic consistency through class-specific prototype alignment. Specifically, the relational distillation module combines covariance consistency alignment—maintaining higher-order statistical relationships through covariance matrix similarity constraints—and masked attention alignment. The latter enables local cross-modal feature interaction, thereby achieving robust global-local knowledge transfer in modality-missing scenarios. The prototype distillation component introduces class-specific semantic prototypes to align single-modal and fused features in a shared embedding space, preventing model bias towards the dominant modality through prototype anchoring regularization while maintaining semantic consistency between missing modalities.

[0078] 3. This invention designs a Dynamic Training Monitoring (DTM) strategy that can track the distillation gap specific to real-time modalities. Through adaptive loss reweighting and gradient direction constraints, the system can dynamically balance the optimization pace among modalities with heterogeneous missing rates, solving a key problem of training stability in imbalanced scenarios. Attached Figure Description

[0079] Figure 1 This is a schematic diagram of the model of the present invention.

[0080] Figure 2 This is a schematic diagram of the Dynamic Modality Awareness Fusion (DMAF) module proposed in this invention.

[0081] Figure 3 Visualization of segmentation on the BraTS2020 dataset using MR = 0.2 / 0.4 / 0.6 / 0.8 (T1 / T1c / FLAIR / T2); Figure (a) shows the visualization of segmentation using the T2 modality; Figure (b) shows the visualization of segmentation combining the T1c and T2 modalities; Figure (c) shows the visualization of segmentation combining the T1c, FLAIR, and T2 modalities; Figure (d) shows the visualization of segmentation combining all four modalities: T1, T1c, FLAIR, and T2.

[0082] Figure 4The following are visualizations of segmentation results on the MyoPS2020 dataset using MR = 0.3 / 0.5 / 0.7 (bSSFP / LGE / T2). Figure (a) shows the visualization results of segmentation using the T2 modality; Figure (b) shows the visualization results of segmentation using the LGE modality; Figure (c) shows the visualization results of segmentation combining the bSSFP and T2 modalities; Figure (d) shows the visualization results of segmentation combining the bSSFP, LGE, and T2 modalities. Detailed Implementation

[0083] The present invention will now be described in further detail.

[0084] See Figures 1-4 A segmentation method for incomplete multimodal medical images includes the following steps:

[0085] S100: Select a publicly available, incomplete multimodal MRI image dataset. The MRI image dataset contains N MRI samples, each MRI sample consists of M modalities, and each modality contains P pixels. The nth MRI sample is denoted as... m represents the presence of m modalities, and p represents the presence of p pixels; where n∈{1,2,…,N}, m∈{1,2,…,M}, and p∈{1,2,…,P}; all modalities contained in the same sample have the same true segmentation label;

[0086] S200: Building a segmentation model based on U-NET DMAF U DMAF It includes a feature extraction module, a feature fusion module, and a segmentation result output module; U-NET is an existing technology;

[0087] The feature extraction module includes several sets of encoder-decoders. The feature extraction module is used to extract single-modal features of each modality of the MRI image and obtain single-modal specific characterization. The encoder-decoder uses the encoder-decoder structure that comes with U-NET, which is an existing technology.

[0088] The feature fusion module includes a Dynamic Modality Aware Fusion (DMAF) module and a fusion decoder. The feature fusion module is used to fuse and decode all single-modality feature representations of the MRI image. The fusion decoder uses a Transformer-based decoder structure, which is a prior art technology.

[0089] The segmentation result output module is used to output the segmentation results of the MRI image;

[0090] In S200, the Dynamic Modality Aware Fusion (DMAF) module is a combination of the Transformer attention mechanism and a dynamic masking strategy. By integrating the Transformer attention mechanism and the dynamic masking strategy, this module can achieve adaptive fusion of cross-modal features while effectively suppressing the interference of missing modalities on the model.

[0091] DMAF includes the following key steps:

[0092] Feature processing: For the multimodal features output from the l-th layer of the encoder This invention first applies a dynamic mask C n,m Features of missing modalities are set to zero, retaining only features of valid modalities to ensure that invalid modalities do not interfere with subsequent calculations. Then, adaptive convolutional downsampling is used to unify the resolution of each modality's features to a fixed size S. d =H d ×W d ×D D This significantly reduces the computational complexity of subsequent attention mechanisms.

[0093] Cross-modal attention learning: Downsampled features are flattened into a labeled sequence These sequences are then concatenated and input into the K-layer MMHA module. The computation process of MMHA at the k-th layer is as follows: first, the labels from different modalities are concatenated into a tensor suitable for the Transform input format. Then, the underlying multimodal associations are learned through the MMHA module; specifically, the MMHA calculation for the Kth layer is as follows:

[0094] T′ k =MMHA(LN(T) K-1 ),M mask )+T k-1 T k =FFN(LN(T′) k ))+T′ k

[0095] in, It is a dynamically generated attention mask based on C, used to force the model to ignore labels from missing modalities, thereby preventing the introduction of noise interference.

[0096] Feature reweighted fusion: DMSA labels the output of each modality. After deserialization and upsampling, a softmax operation is then performed to generate a spatial weight map. The final formula for calculating the fused features is as follows:

[0097]

[0098] Here, ⊙ represents element-wise multiplication; this mechanism enables the model to dynamically focus on the discriminative region while suppressing noise interference.

[0099] S300: Select the i-th MRI sample from the multimodal MRI image dataset. One modality corresponds to one encoder-decoder set. m i Each mode is accessed via m i Each encoder generates a single-modal feature corresponding to each modality. l represents the l-th convolutional layer in the encoder, and s represents It contains the s-th mode and s∈{1,2,…,m} i};

[0100] S400: Each m is obtained through their respective decoders. i Each modality corresponds to a specific single-modal representation. Where “1” represents the first convolutional layer in the decoder;

[0101] All of In the input feature fusion module, multimodal fusion features f are obtained through DMAF. i,L Where L represents the number of convolutional layers in DMAF, and f i,L The multimodal specific representation z is obtained by layer-by-layer decoding using a fusion decoder. i,1 , where “1” represents the first convolutional layer in the fusion decoder;

[0102] z i,1 That is, medical images with incomplete modalities. The segmentation results;

[0103] S500: Constructing the segmentation model U DMAF loss function Using all samples in the multimodal MRI image dataset as U DMAF The input is processed using the AdamW optimizer tool and the dynamic training monitoring strategy DTM to optimize U. DMAF Training is performed, and U is updated in reverse using gradient descent. DMAF The parameters are set, and training stops when the maximum number of iterations is reached, resulting in a trained segmentation model U′. DMAF The AdamW optimizer tool and the Dynamic Training Monitoring Strategies (DTM) are both existing technologies.

[0104] The segmentation model U is constructed in S500. DMAF loss function The steps are as follows:

[0105] S510: Combine Dice loss with weighted cross-entropy loss to form a combined loss. and Jointly construct a single-mode loss function The calculation expression is as follows:

[0106]

[0107] Among them, I n,m Indicates whether mode m exists in sample n, I n,m =1 indicates that mode m exists in sample n; otherwise, it is 0. y represents the single-modal representation of a sample n containing m modalities in the final output layer. n The segmentation labels are represented; both Dice loss and weighted cross-entropy loss are existing technologies.

[0108] S520: will With z n,l Jointly construct a multimodal loss function The calculation expression is as follows:

[0109]

[0110] Where l represents the l-th convolutional layer, l=1 represents the final output layer; L represents the total number of convolutional layers; λ l Indicates the layer loss weights of layer l; Indicates pressing 2 l Upsampling is performed by a factor of 1; z n,l This represents the multimodal representation of the nth sample in layer l;

[0111] By applying deep supervision to the fusion encoder, the loss function This enhanced the model's feature representation capabilities and training efficiency; simultaneously, By independently supervising the segmentation results of each modality, the model is forced to retain the unique information of each modality, thereby enhancing the robustness of single-modality fusion and the accuracy of multi-modal fusion.

[0112] S530: Constructing the relational distillation loss function The calculation expression is as follows:

[0113]

[0114] Where α1 represents the learnable equilibrium hyperparameter, and |M| represents the number of modes. This represents the covariance loss function in relational distillation. The attention loss function in relation distillation is represented by the covariance mechanism, which ensures the consistency of global statistical properties between unimodal features and multimodal fusion features, enhancing the model's robustness to distribution drift. The attention mechanism captures the attention dependencies of the multimodal model in key regions, strengthening the alignment of local features. Through complementary constraints at both global and local levels, comprehensive knowledge transfer from the multimodal model to the unimodal model is achieved, enabling the unimodal model to inherit the global structure and local details of the multimodal model even when other modalities are missing.

[0115] The relational distillation loss function is constructed in S530. The steps are as follows:

[0116] S531: Relation distillation is constructed using covariance consistency alignment (CCA) and masked attention alignment (MAA). Both CCA and MAA are existing technologies. f is calculated using CCA. n,l and The spatial covariance matrix Cov(·):

[0117]

[0118] in, This represents the fused multimodal or single-modal features, where μ represents the feature average.

[0119] Calculate the covariance loss function The calculation expression is as follows:

[0120]

[0121] in, The covariance matrix Cov(·) represents a linear mapping; it captures the statistical correlation between features, while the single-modal covariance is represented by a linear mapping. The model is projected onto a fused feature space for alignment. This process forces the unimodal model to learn the distribution patterns of multimodal features and constrains them using mean squared error.

[0122] This covariance alignment mechanism effectively guides the unimodal branch to learn the global statistical properties of multimodal features. Through explicit constraints, it reduces the unimodal model's dependence on other modalities (e.g., it can retain the statistical properties of multimodal features even when some modalities are missing), alleviates the feature distribution shift caused by missing modalities, and enhances the model's generalization ability under incomplete modal conditions.

[0123] S532: MAA adopts single-modal features As the query matrix Q, the fused multimodal features f are used. n,LAs key-value pairs K and V, attention weight matrices are generated through the multi-head attention mechanism MHA(·). The calculation expression is as follows:

[0124]

[0125] Among them, M mask ∈{-∞,0} represents an attention mask generated based on the modality existence matrix C;

[0126] Calculate the attention loss function The expression is as follows:

[0127]

[0128] These matrices can dynamically capture cross-modal interactions. They force the model to ignore key-value pairs corresponding to missing modalities. By minimizing the mean squared error (MSE) loss between attention output and fused features, the model can capture the attention patterns of multimodal models in key regions (e.g., lesion boundaries), ensuring that unimodal models can focus on equally important regions as multimodal models.

[0129] S533: Utilization and Calculated

[0130]

[0131] S540: Constructing a prototype distillation loss function using single-modal prototype pixels and fused-modal prototype pixels. The calculation expression is as follows:

[0132]

[0133] in, Indicates a single-modal prototype pixel. f represents the prototype pixel of the fused modality. n,L (·) indicates a multimodal fusion feature. Ω represents a single-modal characteristic. c Represents the set of pixels of category c, ∈ is used to prevent division by zero error, τ m The temperature hyperparameter is represented here. A prototype alignment loss is employed to enforce prototype similarity between the fused representation and the unimodal representation. By explicitly aligning cross-modal category prototypes in real time, this method forces the unimodal encoder to maintain semantic consistency with the fused representation even in the case of missing modalities. Furthermore, it simultaneously mitigates the distribution shift caused by missing data and enhances generalization ability in incomplete multimodal scenarios.

[0134] S550: Constructing the segmentation model U DMAF loss function The calculation expression is as follows:

[0135]

[0136] Among them, λ1, λ2, λ3, and λ4 are all model equilibrium hyperparameters.

[0137] The content of the Dynamic Training Monitoring Strategy (DTM) in S500 is as follows:

[0138] First, calculate the distillation difference g. r (·) and the difference between the prototype distillation and g p (·), the calculation formula is as follows:

[0139]

[0140] Where α1 represents the learnable equilibrium hyperparameter;

[0141] Then calculate the relationship between m and distillation difference at time step t. Difference from prototype distillation The calculation formula is as follows:

[0142]

[0143] Where, α decay The adaptive attenuation coefficient is calculated using the following formula:

[0144]

[0145] α decay =0.9·(1-S(ratio) m ))

[0146] Where S(·) represents the sigmoid function, ∈ is used to prevent the error of division by zero; the decay rate is adjusted according to the historical gap ratio between relation loss and prototype loss to ensure faster tracking when the gap diverges significantly.

[0147] The total difference is then calculated using a weighted summation method.

[0148]

[0149] Where α2 represents the learnable equilibrium hyperparameter, Historical exponential moving average representing the relationship distillation gap. Historical exponential moving average representing the prototype distillation gap. This represents the general notation, specifically the EMA for all types of distillation gaps. EMA stands for Exponential Moving Average.

[0150] Then, to Inverse scaling yields w (t) (m), the calculation expression is as follows:

[0151]

[0152] Among them, w (t) (m) represents the weight value in the adversarial weighting mechanism, and m′ represents the traversal symbol;

[0153] The adversarial weighting mechanism introduced here is designed to enhance robustness to changes in modal features. This mechanism performs an inverse scaling operation on the contribution of the modality based on its temporal stability pattern.

[0154] Application w (t) (m) Calculate the single-mode segmentation loss The weighted formula is defined as follows:

[0155]

[0156] For the encoder Encoder(m) corresponding to the m-th mode, its gradient descent update mechanism is as follows:

[0157]

[0158] in, γ represents the encoder gradient of mode m at time step t. (t) (m) represents the gradient scaling factor that is inversely proportional to the distillation difference of mode m, sim (t) (m) represents the cosine similarity between continuous gradient directions;

[0159] When sim (t) When (m) < -0.5, it indicates a conflict in gradient directions during the iteration process. In this case, the update magnitude will be adaptively reduced by a factor of 0.7. The purpose of this operation is to suppress unstable optimization behavior. This specific coefficient was determined through a comprehensive grid search experiment on candidate values ​​{0.5, 0.6, 0.7, 0.8}, where a value of 0.7 can achieve optimal stability while preserving the crucial gradient information to the maximum extent.

[0160] In handling incomplete multimodal medical image segmentation tasks with imbalanced missing rates, the significant differences in the proportion of missing modalities between different imaging types can severely disrupt training stability and impair model performance. To alleviate this problem, this invention proposes a Dynamic Training Monitoring (DTM) strategy. This strategy achieves modal rebalancing by monitoring two key indicators in real time: 1) relation distillation gap, used to quantify feature interaction differences, and 2) prototype distillation gap, used to measure semantic alignment deviation. By dynamically adjusting the loss weights and gradient scaling factors across modalities, DTM effectively slows down the convergence speed of the dominant modality while accelerating the convergence of the undertrained modality, thereby establishing a balanced dynamic process for parameter optimization.

[0161] S600: Input the MRI image X to be predicted into U′ DMAF The output is the medical image prediction and segmentation result of X.

[0162] Experimental Design and Content

[0163] 1. Dataset and Evaluation Metrics

[0164] This experiment performed segmentation experiments on two publicly available MRI datasets, including:

[0165] 1) BraTS2020: The BraTS2020 dataset contains 369 multiparametric MRI scans, including T1-weighted, enhanced T1 (T1ce), T2-weighted, and fluid-attenuated inversion recovery (FLAIR) sequences. Each case is meticulously annotated to distinguish healthy brain tissue from three tumor subregions: necrotic core and non-enhancing tumor (NCR / NET), peritumoral edema (ED), and enhancing tumor (ET). Following the challenge protocol, these subregions are integrated into three clinically relevant targets: whole tumor (WT: NCR+NET+ED+ET), tumor core (TC: NCR+NET+ET), and active tumor (ET). The dataset is divided into 219 training cases, 50 validation cases, and 100 test cases.

[0166] 2) MyoPS2020: The MyoPS2020 dataset contains 45 multi-sequence cardiac MRI scans, including balanced steady-state free precession (bSSFP), gadolinium delayed enhancement (LGE), and T2-weighted sequences. Annotation focuses on left ventricular myocardial pathology, initially classifying structures into six categories: normal myocardium, left / right ventricular blood pool (LVB / RVB), myocardial edema, scar tissue, and background. To simplify clinical analysis, pathological regions (edema and scar) are merged with normal myocardium into a single composite label, namely left ventricular myocardium (MYO). This dataset consists of 25 training cases with multi-slice annotations and 20 independent test cases for testing.

[0167] Following the approach in [4,10], this experiment cropped the black background region outside the brain, performed center cropping on the heart region, and further normalized the intensity of each volume to zero mean and unit variance. To demonstrate the flexibility of DMAF-Net, this experiment performed 3D segmentation on the BraTS2020 dataset and 2D segmentation on the MyoPS2020 dataset.

[0168] The evaluation framework employs two main performance metrics: the Dice similarity coefficient (DSC) for volumetric segmentation overlap analysis and the Hausdorff distance (HD) for boundary alignment assessment. DSC quantifies pixel-level segmentation accuracy through set intersection measurements, while HD characterizes the maximum surface deviation between the segmented contour and the true contour. Optimal segmentation quality is characterized by a maximum DSC value (ideal score of 1) and a minimum HD value (ideal value of 0). All quantitative comparisons are performed at the full volumetric analysis level, consistent with the methodology of clinical evaluation procedures, in accordance with existing biomedical segmentation benchmarks such as BraTS and MyoPS.

[0169] 2. Implementation details

[0170] To comprehensively compare with state-of-the-art (SOTA) multimodal learning methods for handling imbalanced data (such as ModDrop, PMR, and PASSION), this experiment uses mmFormer as the baseline architecture to evaluate the performance of each method under different modality missing rates. Furthermore, this experiment also incorporates three other state-of-the-art multimodal segmentation frameworks—RFNet, M2FTrans, and PASSION—and uses the same missing rate protocol to systematically verify the impact of different modality combinations on segmentation accuracy. All experiments were conducted on a single NVIDIA GeForce RTX 3090 GPU using the AdamW optimizer with the following hyperparameters: 300 training epochs, initial learning rate of 2e-4, weight decay of 1e-4, batch size of 1, and hyperparameters α1, λ1, λ2, λ3, and λ4 initialized to 0.6, 2.0, 1.0, 0.5, and 0.5, respectively. These values ​​were determined through comprehensive grid search optimization.

[0171] 3. BraTS2020 Assessment

[0172] Quantitative Evaluation. Using mmFormer as the baseline model on the BraTS2020 dataset, different modality rebalancing methods were systematically compared (see Table 1). The baseline model showed significantly better performance under perfect training data (PDT) conditions than under imbalanced training data (IDT) conditions, highlighting the greater challenge of segmentation under imbalanced modality missing rates. The ModDrop method even showed a slight performance decline relative to the baseline model because its effectiveness highly depends on precise control of the dropout probability, which cannot adapt to heterogeneous missing rates. Although PMR achieved a slight performance improvement relative to the baseline, its prototype-based approach has limitations: the prototype requires sufficiently balanced and rich samples to avoid estimation bias in medical imaging scenarios with inherent class imbalance (e.g., sparse lesion regions), leading to poor lesion segmentation performance. In contrast, PASSION achieves significant performance improvements through self-distillation and preference-aware regularization for adaptive modality learning alignment. However, this still focuses on mimicking output probabilities while neglecting crucial structural relationships necessary for cross-modal knowledge transfer, such as spatial dependencies and pathological topological patterns. Crucially, existing modality rebalancing methods fail to address the issue of heterogeneous modality contributions. For example, FLAIR and T2 modalities contain significantly more diagnostic information than T1 / T1c. Without a dynamic weighting mechanism for heterogeneous modality contributions, suboptimal multimodal fusion features are generated, leading to performance degradation. The proposed DMAF-Net comprehensively addresses the issues of modality missing rate imbalance and modality contribution heterogeneity through dynamic modality fusion, relation and prototype distillation, and dynamic training monitoring strategies. It achieves continuous performance improvements across all missing rate scenarios, demonstrating superior modality balancing capabilities under IDT conditions.

[0173] Table 1. Quantitative comparisons under different settings on the BraTS2020 and MyoPS2020 datasets.

[0174]

[0175]

[0176] Regarding Table 1, MR is an abbreviation for Modality Missing Rate. For the BraTS2020 dataset, it refers to the missing rates of the T1 / T1c, FLAIR, and T2 modalities; for the MyoPS2020 dataset, it refers to the missing rates of the bSSFP, LGE, and T2 modalities. s, m, and l correspond to small, medium, and large levels, respectively. In the BraTS2020 dataset, s = 0.2, m = 0.5, and l = 0.8; in the MyoPS2020 dataset, s = 0.3, m = 0.5, and l = 0.7. PDT represents modality balance, i.e., perfect training data with MR = (0,0,0). Here, T1 and T1c have the same missing rate, but are treated as two independent modalities in the experiment.

[0177] Qualitative assessment. Figure 3 This paper presents a qualitative comparison of different modality rebalancing methods on the BraTS2020 dataset under various modality combinations. When only the T2 modality is provided, the segmentation results show significant limitations; however, with the addition of other modalities, performance is significantly improved, demonstrating the inherent advantages of multimodal learning compared to single-modal methods. Notably, ModDrop and PMR generate a large number of false positives due to their failure to address the imbalance in modality missing rates; in contrast, the DMAF-Net of this invention effectively suppresses false positives and generates the most accurate segmentation results.

[0178] 4. MyoPS 2020 Evaluation

[0179] Quantitative Evaluation. The quantitative comparison results of MyoPS2020 are summarized in Table 1. Compared with BraTS2020, MyoPS2020 exhibits smaller differences in intermodal information and closer semantic contributions, thus eliminating the dominant modality effect; therefore, the impact of modality missing rate imbalance on segmentation performance is relatively weakened; consequently, all modality rebalancing methods show significant performance improvements over the baseline model, among which the proposed DMAF-Net achieves the best performance, validating its superior feature extraction and fusion capabilities in medical segmentation tasks.

[0180] Qualitative assessment. Figure 4 This paper presents a qualitative comparison of different methods under various modality combinations on the MyoPS2020 dataset. Due to the balanced semantic contributions and minimal information differences among modalities, the performance fluctuations of the segmentation results are small under different modality configurations. Compared with baseline methods, all modality rebalancing methods produce more accurate segmentation results, with DMAF-Net generating the fewest false positives and achieving the best results. This further demonstrates the effectiveness of the proposed dynamic modality fusion strategy in handling balanced but challenging medical imaging scenarios.

[0181] 5. Evaluation results of backbone network under different combinations of deletion patterns

[0182] To comprehensively analyze the impact of different modality missing data combinations on model performance, experiments were conducted on the BraTS2020 dataset. Under the same modality missing data rates (0.2, 0.4, 0.6, 0.8), DMAF-Net was compared with state-of-the-art (SOTA) incomplete multimodal medical image segmentation methods. Detailed results are shown in Table 2. The experiments reveal that methods such as RFNet, mmFormer, and M2Ftrans, which do not consider modality imbalance and missing data heterogeneity, perform well in PDT (perfect data training) scenarios, but their performance significantly declines under IDT (imperfect data training) conditions. This indicates their inadequacy in handling the modality missing data imbalance problem prevalent in clinical practice. In comparison, PASSION, which explicitly considers modality imbalance, and the DMAF-Net proposed in this invention exhibit significantly smaller performance degradation under IDT conditions compared to the aforementioned methods. Notably, in WT (whole tumor) segmentation, the performance metrics of DMAF-Net are almost comparable to those under PDT settings, highlighting the robustness of the proposed framework in handling heterogeneity in missing data rates. Furthermore, according to quantitative metrics, DMAF-Net outperforms PASSION by 1-2% in IDT scenarios. This performance improvement underscores the superiority of DMAF-Net's dynamic fusion and training monitoring mechanisms in mitigating the challenges posed by modality imbalance, especially under clinically relevant imperfect data conditions.

[0183] Table 2. Comparison of quantitative evaluation results of different backbone networks with different modality missing combinations on the BraTS2020 dataset.

[0184]

[0185]

[0186] Table 2 shows that for the IDT (i.e., training on imperfect data) scenario, the modality missing rate (MR) was set to (0.2, 0.4, 0.6, 0.8); while for the PDT (i.e., training on perfect data) scenario, MR was set to (0, 0, 0, 0). In the results, the values ​​labeled for RFNet, mmFormer, and M2FTrans were generated by methods that considered the IDT scenario, while the values ​​labeled for PASSION and DMAF-Net were generated by methods that did not consider the IDT scenario.

[0187] Table 3. Component-by-component ablation data on the BraTS2020 and MyoPS2020 datasets

[0188]

[0189] 6. Component-by-component ablation study

[0190] This experiment conducted a comprehensive component-by-component ablation study on the BraTS2020 and MyoPS2020 datasets, systematically evaluating different combinations of DMAF, Distination (including relational distillation and prototype distillation), and DTM components. The results are summarized in Table 3. Overall, introducing each component individually can improve performance. However, using only the DMAF module results in relatively limited performance improvement because it primarily addresses the heterogeneity of modal contributions, but is less effective for the problem of imbalanced modal missing rates. When all components are jointly integrated, the model can simultaneously alleviate the imbalance of modal missing rates and modal contributions, achieving optimal segmentation performance and significantly outperforming the baseline model.

[0191] 7. Conclusion

[0192] This invention proposes a Dynamic Modality-Aware Fusion (DMAF-Net) framework for incomplete multimodal medical image segmentation, aiming to address the challenges of multimodal feature fusion and parameter optimization under modality missing rate imbalance (IDT) conditions. To overcome the limitations of existing methods in handling modality missing rate heterogeneity and modality contribution heterogeneity, this invention innovatively introduces three core components: Dynamic Modality-Aware Fusion (DMAF), relation and prototype distillation, and a Dynamic Training Monitoring (DTM) strategy. These modules work synergistically to dynamically adjust fusion weights, enforce cross-modal semantic alignment, and adaptively stabilize training dynamics, thereby collaboratively solving the modality imbalance problem. Comprehensive evaluations on the BraTS2020 and MyoPS2020 datasets demonstrate that the proposed DMAF-Net outperforms state-of-the-art modality rebalancing methods, confirming its superiority.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A segmentation method for incomplete multimodal medical images, characterized in that: Includes the following steps: S100: Select a publicly available, incomplete multimodal MRI image dataset. The MRI image dataset contains N MRI samples, each MRI sample consists of M modalities, and each modality contains P pixels. The nth MRI sample is denoted as... m represents the presence of m modalities, and p represents the presence of p pixels; where n∈{1,2,…,N}, m∈{1,2,…,M}, and p∈{1,2,…,P}; all modalities contained in the same sample have the same true segmentation label; S200: Building a segmentation model based on U-NET DMAF U DMAF It includes a feature extraction module, a feature fusion module, and a segmentation result output module; The feature extraction module includes several sets of encoder-decoder pairs. The feature extraction module is used to extract single-modal features of each modality of the MRI image and obtain single-modal specific characterization. The feature fusion module includes a dynamic modality-aware fusion module (DMAF) and a fusion decoder. The feature fusion module is used to fuse and decode all single-modal feature representations of MRI images. The segmentation result output module is used to output the segmentation results of the MRI image; S300: Select the i-th MRI sample from the multimodal MRI image dataset. One modality corresponds to one encoder-decoder set. m i Each mode is accessed via m i Each encoder generates a single-modal feature corresponding to each modality. l represents the l-th convolutional layer in the encoder, and s represents It contains the s-th mode and s∈{1,2,…,m} i }; S400: Each m is obtained through their respective decoders. i Each modality corresponds to a specific single-modal representation. Where "1" represents the first convolutional layer in the decoder; All of In the input feature fusion module, multimodal fusion features f are obtained through DMAF. i,L Where L represents the number of convolutional layers in DMAF, and f i,L The multimodal specific representation z is obtained by layer-by-layer decoding using a fusion decoder. i,1 Where "1" represents the first convolutional layer in the fusion decoder; z i,1 That is, medical images with incomplete modalities. The segmentation results; S500: Constructing the segmentation model U DMAF loss function Using all samples in the multimodal MRI image dataset as U DMAF The input is M, which is trained using the AdamW optimizer and the dynamic training and monitoring strategy DTM. U is then updated in reverse using gradient descent. DMAF The parameters are set, and training stops when the maximum number of iterations is reached, resulting in a trained segmentation model U′. DMAF ; S600: Input the MRI image X to be predicted into U′ DMAF The output is the medical image prediction and segmentation result of X.

2. The segmentation method for incomplete multimodal medical images as described in claim 1, characterized in that: In S200, the Dynamic Modality Awareness Fusion (DMAF) module is a combination of the Transformer attention mechanism and the dynamic masking strategy.

3. The segmentation method for incomplete multimodal medical images as described in claim 2, characterized in that: The segmentation model U is constructed in S500. DMAF loss function The steps are as follows: S510: Combine Dice loss with weighted cross-entropy loss to form a combined loss. and Jointly construct a single-mode loss function The calculation expression is as follows: Among them, I n,m This indicates whether mode m exists in sample n. y represents the single-modal representation of a sample n containing m modalities in the final output layer. n Indicates a separator label; S520: will With z n,l Jointly construct a multimodal loss function The calculation expression is as follows: Where l represents the l-th convolutional layer, l=1 represents the final output layer; L represents the total number of convolutional layers; λ l Indicates the layer loss weights of layer l; Indicates pressing 2 l Upsampling is performed by a factor of 1; z n,l This represents the multimodal representation of the nth sample in layer L; S530: Constructing the relational distillation loss function The calculation expression is as follows: Where α1 represents the learnable equilibrium hyperparameter, and |M| represents the number of modes. This represents the covariance loss function in relational distillation. Represents the attention loss function in relational distillation; S540: Constructing a prototype distillation loss function using single-modal prototype pixels and fused-modal prototype pixels. The calculation expression is as follows: in, Indicates a single-modal prototype pixel. f represents the prototype pixel of the fused modality. n,L (·) indicates a multimodal fusion feature. Ω represents a single-modal characteristic. c Represents the set of pixels of category c, ∈ is used to prevent division by zero error, τ m This indicates temperature hyperparameters; S550: Constructing the segmentation model U DMAF loss function The calculation expression is as follows: Among them, λ1, λ2, λ3, and λ4 are all model equilibrium hyperparameters.

4. The segmentation method for incomplete multimodal medical images as described in claim 3, characterized in that: The relational distillation loss function is constructed in S530. The steps are as follows: S531: Relational distillation is constructed using covariance consistency alignment (CCA) and masked attention alignment (MAA), and f is calculated using CCA. n,l and The spatial covariance matrix Cov(·): in, This represents the fused multimodal or single-modal features, where μ represents the feature average. Calculate the covariance loss function The calculation expression is as follows: in, Represents a linear mapping; S532: MAA adopts single-modal features As the query matrix Q, the fused multimodal features f are used. n,L As key-value pairs K and V, attention weight matrices are generated through the multi-head attention mechanism MHA(·). The calculation expression is as follows: Among them, M mask ∈{-∞,0} represents an attention mask generated based on the modality existence matrix C; Calculate the attention loss function The expression is as follows: S533: Utilization and Calculated 5. The segmentation method for incomplete multimodal medical images as described in claim 4, characterized in that: The content of the Dynamic Training Monitoring Strategy (DTM) in S500 is as follows: First, calculate the distillation difference g. r (·) and the difference between the prototype distillation and g p (·), the calculation formula is as follows: Where α1 represents the learnable equilibrium hyperparameter; Then calculate the relationship between m and distillation difference at time step t. Difference from prototype distillation The calculation formula is as follows: Where, α decay The adaptive attenuation coefficient is calculated using the following formula: a decay =0.9·(1-S(ratio m )) Where S(·) represents the sigmoid function, and ∈ is used to prevent errors caused by division by zero; The total difference is then calculated by weighted summation. Where α2 represents the learnable equilibrium hyperparameter, Historical exponential moving average representing the relationship distillation gap. Historical exponential moving average representing the prototype distillation gap. This represents the general notation, specifically the EMA for all types of distillation gaps. EMA stands for Exponential Moving Average. Then, to Inverse scaling yields w (t) (m), the calculation expression is as follows: Among them, w (t) (m) represents the weight value in the adversarial weighting mechanism, and m′ represents the traversal symbol; Application w (t) (m) Calculate the single-mode segmentation loss The weighted formula is defined as: For the encoder Encoder(m) corresponding to the m-th mode, its gradient descent update mechanism is as follows: in, γ represents the encoder gradient of mode m at time step t. (t) (m) represents the gradient scaling factor that is inversely proportional to the distillation difference of mode m, sim (t) (m) represents the cosine similarity between continuous gradient directions; When sim (t) When (m) < -0.5, it indicates that there is a conflict in the gradient direction during the iteration process. At this time, the update magnitude will be adaptively reduced by a factor of 0.7.

Images