A Multimodal Medical Image Segmentation Method and Device with Dynamic Gated Adaptive Feature Fusion

Abstract

This application discloses a multimodal medical image segmentation method and device with dynamic gating adaptive feature fusion, belonging to the field of image segmentation technology. The method includes: constructing a dynamic gating adaptive feature fusion architecture; the dynamic gating adaptive feature fusion architecture includes: a stochastic modality missing simulator and a medical image segmentation network connected sequentially; the medical image segmentation network includes: an independent feature encoding module, a quality-aware scoring sub-network, a dynamic gating controller, and a decoder connected sequentially; using a historical medical image dataset and the model's total loss function, the dynamic gating adaptive feature fusion architecture is trained using a five-fold cross-validation experiment, and the trained medical image segmentation network is determined as the segmentation model for multimodal medical image segmentation of target areas. This application achieves high robustness segmentation performance for modality missing and quality-impaired scenes by constructing a dynamic gating adaptive feature fusion architecture.

Description

Technical Field

[0001] This application relates to the field of image segmentation technology, and in particular to a multimodal medical image segmentation method and device with dynamic gating adaptive feature fusion. Background Technology

[0002] With the rapid development of medical imaging technology, multimodal medical image segmentation methods, by fusing complementary information from different modalities, have been widely applied in clinical diagnosis and treatment processes such as oncology and neurology. Brain tumors, as the most common primary tumors of the central nervous system, are characterized by high invasiveness, high recurrence rates, and poor prognosis. Their preoperative diagnosis, treatment plans, and prognostic assessment heavily rely on accurate segmentation of the tumor region. Magnetic Resonance Imaging (MRI), due to its superior soft tissue contrast and rich tissue detail, plays a crucial role in the diagnosis and assessment of brain tumors. Multimodal MRI (such as T1-weighted imaging-1, T1ce (contrast-enhanced imaging-1), T2-weighted imaging-2, and FLAIR (fluid attenuation inversion recovery imaging)) can provide complementary information from different physiological and pathological perspectives: T1 displays anatomical structures, T1c delineates areas of blood-brain barrier disruption and locates tumor enhancement, T2 clearly shows lesions reflecting the extent of edema, and FLAIR suppresses cerebrospinal fluid signals to more clearly display the extent of tumor infiltration. This allows for precise labeling of lesion regions and significantly improves the ability to identify, segment, and detect lesion boundaries, greatly providing reliable evidence and methods for the accurate diagnosis, treatment planning, and surgical navigation of brain tumors. However, in actual clinical applications, problems such as patient motion artifacts, scan failures, time and equipment limitations, or image noise contamination often lead to the loss or severe degradation of some modalities. These modal loss or degradation issues directly result in insufficient or unreliable multimodal information, severely limiting the performance and clinical applicability of existing multimodal segmentation algorithms and models, thus significantly reducing the accuracy and reliability of medical image segmentation. To address this issue, Dar et al. used an explicit generation method based on Generative Adversarial Networks (GANs) to complete the missing modal images before segmentation. However, this method often comes with huge computational overhead and introduces the risk of imperceptible artifacts, and generation errors can easily affect the segmentation results. Havaei et al. and Zhou et al. avoided explicit image generation and attempted to fuse or compensate at the feature level by calculating the mean square of existing modal features or by using attention mechanisms and uncertainty estimation to weight different modal features. However, the fusion strategies of these methods are mostly static or passive, and cannot be dynamically and adaptively adjusted according to the specific combination and quality of the input modalities.

[0003] Multimodal Medical Image Segmentation: In the field of medical image segmentation, multimodal information fusion has become an important means to improve image segmentation accuracy. Classical U-Net and its numerous variants generally employ simple early feature fusion methods when applied to multimodal data. For example, nnU-Net proposed by Isensee et al. uses early fusion through automated network configuration and training strategies, significantly improving segmentation performance on datasets such as BraTS, and has become one of the most representative baseline frameworks. Myronenko et al. designed a 3D network based on variational autoencoders (VAEs), achieving leading performance on BraTS2018 by learning multimodal representations through shared latent variables. Havaei et al. proposed a dual-path CNN structure, simultaneously extracting local and global features, enabling the model to better capture the contextual dependencies between multimodalities. Although these methods are simple to implement and allow the network to learn the relationships between modalities at a shallow level, they ignore the differences in the contributions of different modalities to the segmentation task, easily leading to information redundancy and even performance degradation. To more flexibly handle complex relationships between modalities, researchers have begun to focus on how to efficiently utilize multimodal features. Kamnitsas et al. employed a static fusion strategy using equal-weighted averaging, directly encoding modal features in parallel under multimodal input and merging them in a fixed manner, assuming all modalities are equally important. However, this method cannot adaptively adjust to the differences in modal quality in specific cases and is prone to introducing invalid features when noise or artifacts are present. Furthermore, aiming to adaptively measure the importance of features from different modalities or spatial locations, Zhang et al. proposed a cross-modal attention fusion network. By introducing a cross-modal attention mechanism, they improved the interaction ability between modalities, achieving a significant improvement in the accuracy of multimodal brain tumor segmentation. Zhou et al. designed a multimodal fusion network, introducing channel attention and spatial attention modules to refine the features before fusion. Although the above methods have significantly promoted the development of multimodal segmentation, most methods assume that all modalities are complete and of good quality, heavily relying on the availability of all modalities. Once a modality is missing, the performance of models based on the "full-modal assumption" will significantly decrease.

[0004] Modality Missing Medical Image Segmentation: Researchers have proposed various methods to address the problem of modality missing data in multimodal learning. For example, Dar et al. proposed synthesizing missing MRI contrast sequences from any available modality subset, and then feeding the synthesized images into a segmentation network for auxiliary segmentation. Huo et al.'s SynSeg-Net utilizes a Cyclic Generative Adversarial Network (CycleGAN) to achieve image transformation between different modalities, completing cross-modal synthesis without requiring real annotations of the target modality. Shen et al. proposed using a Generative Adversarial Network (GAN) with a representation unwrapping scheme to extract shared content encoding and separation pattern encoding across multiple domains for image reconstruction. This modality generation and completion method, which uses complete "full-modality" data for segmentation by completing missing modalities, is intuitive and can seamlessly integrate with traditional methods. However, they have significant drawbacks: image-level generation is usually computationally expensive, and errors and artifacts are inevitably introduced into the synthesized images, which are easily propagated to subsequent segmentation stages. To avoid the difficulties of generating images and the high computational cost, researchers have proposed operating directly at the feature level. By learning a modality-invariant shared feature space, models can still obtain robust feature representations even when some modalities are missing. For example, HeMIS proposed by Havaei et al. addresses modality missingness by encoding each modality separately and calculating the mean and variance of existing modal features at the feature layer to achieve modality-independent feature aggregation. However, HeMIS relies on simple statistical aggregation and cannot fully capture complex long-range contextual dependencies. Dorent et al. proposed a method at the feature level using variational risk upper bounds and domain adaptation to achieve robust segmentation of heterogeneous modalities and data with missing modalities. While these methods have made progress in feature compensation and avoided explicit artifact generation, their latent space may be too compact, leading to the loss of some modality-specific information, easily overlooking quality differences between different modalities, and having limited dynamic adaptability to complex missing patterns. They lack responsiveness to changes in modality "real-time quality," do not adequately model inter-modal dependencies, and remain susceptible to interference from noisy modalities.

[0005] In summary, existing methods for handling modal missingness have alleviated the limitations of "full modality dependency" to some extent, but still have shortcomings: generative methods are computationally intensive and have the risk of artifacts, latent feature learning ignores modal quality differences, and adaptive fusion strategies are difficult to achieve true dynamic decision-making. Summary of the Invention

[0006] The purpose of this application is to provide a multimodal medical image segmentation method and device with dynamic gating and adaptive feature fusion. By using a modality quality-aware scoring subnetwork and a dynamic gating controller, the fusion strategy is adaptively and dynamically adjusted according to the currently available modalities and their quality scores, thereby achieving high robust segmentation performance for modality-deficient and quality-impaired scenarios, while taking into account accuracy and generalization, providing a new approach to the usability of medical image segmentation in actual clinical practice.

[0007] To achieve the above objectives, this application provides the following solution:

[0008] Firstly, this application provides a multimodal medical image segmentation method with dynamic gating and adaptive feature fusion, including:

[0009] Obtain a historical medical image dataset; the historical medical image dataset includes multiple samples and the segmentation and annotation results of each sample; any one of the samples includes multiple single-modal magnetic resonance imaging images of the target site;

[0010] A dynamic gated adaptive feature fusion architecture and a model total loss function are constructed. The dynamic gated adaptive feature fusion architecture includes: a stochastic modality missing simulator and a medical image segmentation network connected in sequence; the medical image segmentation network includes: an independent feature encoding module, a quality-aware scoring sub-network, a dynamic gate controller, and a decoder connected in sequence.

[0011] Using the historical medical image dataset and the model's total loss function, the dynamic gated adaptive feature fusion architecture was trained using a five-fold cross-validation experiment, and the trained medical image segmentation network was determined to be the segmentation model.

[0012] Acquire multimodal magnetic resonance imaging images of the target region to be segmented;

[0013] The multimodal magnetic resonance imaging image to be segmented is input into the segmentation model to obtain the segmented image of the target region.

[0014] Optionally, after obtaining the historical medical image dataset, the following may also be included:

[0015] The acquired historical medical image dataset is preprocessed; the preprocessing includes: spatial registration processing, uniform voxel spacing processing, intensity normalization processing, and uniform size processing.

[0016] The unified voxel spacing processing is achieved through resampling;

[0017] The uniform size processing is achieved through cropping and padding.

[0018] Optionally, for each sample in the historical medical image dataset, the random modality missing simulator is used to randomly generate a binary mask as an availability mask for each modality, and multiply the availability mask with the corresponding single-modality magnetic resonance imaging image to obtain the input sample after mask processing;

[0019] The availability mask is:

[0020] ;

[0021] ;

[0022] in, Represents the availability mask for the m-th mode; when When, it means setting the m-th mode to zero, or adding Gaussian noise to the m-th single-mode magnetic resonance imaging image, or performing a random affine transformation on the m-th single-mode magnetic resonance imaging image; when When, it means that the m-th modality remains available in the current training iteration; Indicates the Bernoulli distribution; Indicates the probability of mode absence;

[0023] The input mode obtained after masking the m-th single-mode magnetic resonance imaging image in the sample is:

[0024] ;

[0025] in, This is the m-th input mode; This is the m-th single-mode magnetic resonance imaging image.

[0026] Optionally, the independent feature encoding module includes multiple feature extraction branches;

[0027] The feature extraction branches are set to correspond one-to-one with the modalities of the samples;

[0028] The feature extraction branch is a 3D convolutional subnet;

[0029] The feature extraction branch is used to obtain multi-scale features of the corresponding modality. ;

[0030] in, ;

[0031] In the formula, This indicates that the feature extraction branch corresponding to the m-th mode is... l The output features of the layer; This indicates the total number of layers in the feature extraction branch; Indicates the feature extraction branch number 1 l Number of channels per layer; Indicates the feature extraction branch number 1 l Floor height; Indicates the feature extraction branch number 1 l Layer width; Indicates the feature extraction branch number 1 l Layer depth.

[0032] Optionally, for the feature extraction branch corresponding to the m-th modality, the... l Layer output features The quality-perceived scoring subnetwork is used for:

[0033] The output features are processed by global adaptive average pooling. Compressed into a channel description vector, the m-th mode is obtained in the first channel. l Layer channel description vector ; ;

[0034] Using a perceptron to describe the channel vector Perform feature filtering and mapping to obtain the m-th mode. l The perceptron output corresponding to the layer ; ;

[0035] Output of the sensor As the m-th mode in the... l Layer quality score ; .

[0036] Optionally, the dynamic gating controller includes: a weighted feature fusion module and multiple modal weight determination branches;

[0037] The outputs of all the modality weight determination branches are connected to the weighted feature fusion module;

[0038] The modal weight determination branch is set to correspond one-to-one with the modality of the sample;

[0039] The modality weight determination branch includes a first three-dimensional convolutional layer, a second three-dimensional convolutional layer, a global average pooling layer, and a two-layer perceptron mechanism connected in sequence.

[0040] The first three-dimensional convolutional layer and the second three-dimensional convolutional layer have the same structure;

[0041] The first three-dimensional convolutional layer includes: a first convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence;

[0042] The working principle of determining the branch for the mode weights corresponding to the m-th mode is as follows:

[0043] ;

[0044] ;

[0045] ;

[0046] In the formula, The intermediate semantic representation of the high-level semantic feature mapping of the m-th modality; and Both represent the ReLU activation function; Indicates batch normalization; Indicates the second convolutional layer; Indicates the first convolutional layer; Represents a high-level semantic feature mapping for m modalities; Indicates intermediate semantic representation A one-dimensional vector obtained by compression through a global average pooling layer; This represents the modal quality score of the m-th mode; For a one-dimensional vector The embedding vector obtained by embedding modal quality scores into the channel description; Let be the modal weights for the m-th mode; This represents a two-layer sensing mechanism; This represents the Softmax activation function; This represents the weight matrix of the second fully connected layer; This represents the weight matrix of the first fully connected layer; This represents the bias vector of the first fully connected layer; This represents the bias vector of the second fully connected layer;

[0047] The working principle of the weighted feature fusion module is as follows:

[0048] ;

[0049] In the formula, This represents the fusion feature; M is the total number of modalities in the sample.

[0050] Optionally, the total loss function of the model is:

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] In the formula, This represents the total loss of the model; Indicates the primary segmentation loss; This represents the weighting coefficient for quality-assisted loss; Indicates quality auxiliary loss; Indicates the weight coefficient of the dynamic gating balance constraint term; This represents the dynamic gating balance constraint term; Indicates Dice loss; Represents cross-entropy loss; Indicates the model's predicted probability; Indicates the true label; It is a smoothing constant; For the number of categories, The total number; This represents the mean value of the encoder output features for the m-th mode. This represents the global mean of all modal features.

[0058] In a second aspect, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multimodal medical image segmentation method with dynamic gating adaptive feature fusion as described above.

[0059] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal medical image segmentation method with dynamic gating adaptive feature fusion as described above.

[0060] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal medical image segmentation method with dynamic gating adaptive feature fusion as described above.

[0061] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0062] This application provides a multimodal medical image segmentation method and device with dynamic gating adaptive feature fusion. First, during the training phase, a random modality loss simulator is introduced to randomly discard or add noise to one or more modalities, enhancing the network's adaptability to various loss scenarios and enabling the network to learn under multiple combinations of lost or damaged modalities. Second, instead of a fixed fusion method, the fusion strategy is intelligently adjusted based on the specific input. This is achieved through a lightweight modality quality-aware scoring system and a dynamic gating controller, automatically and dynamically adjusting and completing feature fusion based on different modality combinations and quality. This more fully utilizes information from high-quality modalities, avoids interference from invalid features, and thus greatly enhances the model's generalization ability and segmentation robustness. In summary, the main contributions of this application can be summarized in the following four points:

[0063] (1) A random modal missing simulator was constructed: modalities were randomly discarded or degraded according to independent probability, thereby improving its robustness in the face of real missing scenarios.

[0064] (2) A lightweight modal quality perception scoring method was designed: it realizes the real-time evaluation of the credibility of each modal feature, autonomously distinguishes effective features from noise interference, and thus provides quality quantification information for the fusion module.

[0065] (3) A dynamic gating controller is proposed: it can dynamically adjust the feature fusion weights according to the combination of input modes and quality.

[0066] (4) Comprehensive experiments on the dataset show that the Dynamic Gated Adaptive Feature Fusion (DGAF) proposed in this application exhibits superior segmentation performance and robustness in various scenarios with missing modalities. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart of a multimodal medical image segmentation method with dynamic gating and adaptive feature fusion according to an embodiment of this application;

[0069] Figure 2 This is a schematic diagram of a dynamic gating adaptive feature fusion architecture in one embodiment of this application;

[0070] Figure 3 This is a schematic diagram of the structure for random simulation of mode loss in one embodiment of this application;

[0071] Figure 4 This is a schematic diagram of a modal independent coding structure in one embodiment of this application;

[0072] Figure 5 This is a first schematic diagram of a modal quality perception scoring structure in one embodiment of this application;

[0073] Figure 6 This is a first schematic diagram of a modal quality perception scoring structure in one embodiment of this application;

[0074] Figure 7 This is a schematic diagram of the structure of a dynamic gate controller in one embodiment of this application;

[0075] Figure 8 This is a visualization comparison of tumor segmentation between DGAF and nnU-Net in a Flair modality-deficient scenario according to one embodiment of this application.

[0076] Figure 9 This is a visualization comparison of tumor segmentation between DGAF and nnU-Net in a scenario where the T1+T1ce modality is missing, as shown in one embodiment of this application. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0078] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] In one exemplary embodiment, such as Figure 1 As shown, a multimodal medical image segmentation method with dynamic gating and adaptive feature fusion is provided, including:

[0080] Step 101: Obtain the historical medical image dataset. The historical medical image dataset includes multiple samples and the segmentation and annotation results for each sample. Each sample includes multiple single-modal magnetic resonance imaging images of the target site.

[0081] Step 102: Construct the dynamically gated adaptive feature fusion architecture and the model's total loss function. The dynamically gated adaptive feature fusion architecture includes: a stochastic modality missing simulator and a medical image segmentation network connected in sequence. The medical image segmentation network includes: independent feature encoding modules connected in sequence (such as...). Figure 4 ), quality perception scoring subnetwork (such as Figure 5 and Figure 6 ), dynamic gating controller (such as Figure 7 ) and decoder.

[0082] For each sample in the historical medical image dataset, the random modality missing simulator is used to randomly generate a binary mask as an availability mask for each modality. The availability mask is multiplied with the corresponding single-modality magnetic resonance imaging image to obtain the input sample after mask processing.

[0083] The availability mask is:

[0084] .

[0085] .

[0086] in, This represents the availability mask for the m-th mode. When When , it indicates setting the m-th mode to zero, adding Gaussian noise to the m-th single-mode magnetic resonance imaging image, or performing a random affine transformation on the m-th single-mode magnetic resonance imaging image. When , it means that the m-th modality remains available in the current training iteration. This represents the Bernoulli distribution. This represents the probability of a mode being missing.

[0087] The input mode obtained after masking the m-th single-mode magnetic resonance imaging image in the sample is:

[0088] .

[0089] in, This represents the m-th input mode. This is the m-th single-mode magnetic resonance imaging image.

[0090] The independent feature encoding module includes multiple feature extraction branches.

[0091] The feature extraction branches are set to correspond one-to-one with the modalities of the samples.

[0092] The feature extraction branch is a 3D convolutional subnet.

[0093] The feature extraction branch is used to obtain multi-scale features of the corresponding modality. .

[0094] in, .

[0095] In the formula, This indicates that the feature extraction branch corresponding to the m-th mode is... l Output characteristics of the layer. This indicates the total number of layers in the feature extraction branch. Indicates the feature extraction branch number 1 l Number of channels per layer. Indicates the feature extraction branch number 1 l Floor height. Indicates the feature extraction branch number 1 l Layer width. Indicates the feature extraction branch number 1 l Layer depth.

[0096] For the feature extraction branch corresponding to the m-th mode, the ... l Layer output features The quality perception scoring subnetwork is used for:

[0097] The output features are processed by global adaptive average pooling. Compressed into a channel description vector, the m-th mode is obtained in the first channel. l Layer channel description vector . .

[0098] Using a perceptron to describe the channel vector Perform feature filtering and mapping to obtain the m-th mode. l The perceptron output corresponding to the layer . .

[0099] Output of the sensor As the m-th mode in the... l Layer quality score . .

[0100] The dynamic gating controller includes a weighted feature fusion module and multiple modal weight determination branches.

[0101] The outputs of the multiple modality weight determination branches are all connected to the weighted feature fusion module.

[0102] Modal weights are set to correspond one-to-one with the modes of the samples.

[0103] The modality weight determination branch consists of a first three-dimensional convolutional layer, a second three-dimensional convolutional layer, a global average pooling layer, and a two-layer perceptron mechanism connected in sequence.

[0104] The first and second 3D convolutional layers have the same structure.

[0105] The first three-dimensional convolutional layer consists of: a first convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence.

[0106] The working principle of determining the branch for the mode weights corresponding to the m-th mode is as follows:

[0107] .

[0108] .

[0109] .

[0110] In the formula, The intermediate semantic representation represents the high-level semantic feature mapping of the m-th modality. and Both represent the ReLU activation function. This indicates batch normalization. This indicates the second convolutional layer. This indicates the first convolutional layer. This represents a high-level semantic feature mapping for m modalities. Indicates intermediate semantic representation A one-dimensional vector obtained by compression through a global average pooling layer. This represents the modal quality score of the m-th mode. For a one-dimensional vector The embedding vector is obtained by embedding the modal quality score into the channel description. Let be the modal weights of the m-th mode. This indicates a two-layer sensing mechanism. This represents the Softmax activation function. This represents the weight matrix of the second fully connected layer; This represents the weight matrix of the first fully connected layer; This represents the bias vector of the first fully connected layer; This represents the bias vector of the second fully connected layer.

[0111] The working principle of the weighted feature fusion module is as follows:

[0112] .

[0113] In the formula, This represents the fusion feature. M is the total number of modalities in the sample.

[0114] The model's total loss function is:

[0115] .

[0116] .

[0117] .

[0118] .

[0119] .

[0120] .

[0121] In the formula, This represents the total loss of the model. This represents the primary segment loss. This represents the quality-assisted loss weighting coefficient. This indicates a quality-related loss. This represents the weight coefficient of the dynamic gating balance constraint term. This represents the dynamic gating balance constraint term. This indicates Dice's loss. This represents the cross-entropy loss. This represents the probability predicted by the model. This indicates the actual label. This is the smoothing constant. For the number of categories, This represents the total number. This represents the mean value of the encoder output features for the m-th mode. This represents the global mean of all modal features.

[0122] Step 103: Using historical medical image datasets and the model's total loss function, a dynamic gating adaptive feature fusion architecture is trained using a five-fold cross-validation experiment to determine the trained medical image segmentation network as the segmentation model.

[0123] Step 104: Obtain the multimodal magnetic resonance imaging image of the target area to be segmented.

[0124] Step 105: Input the multimodal magnetic resonance imaging image to be segmented and its binary availability mask into the segmentation model to obtain the segmented image of the target region.

[0125] Following step 101, the process further includes preprocessing the acquired historical medical image dataset. Preprocessing includes spatial registration, uniform voxel spacing, intensity normalization, and uniform size processing. Uniform voxel spacing is achieved through resampling. Uniform size processing is achieved through cropping and padding.

[0126] The multimodal medical image segmentation method with dynamic gating and adaptive feature fusion provided in this embodiment aims to improve the segmentation performance and robustness of the model under conditions of incomplete or quality-limited modalities in actual clinical practice. The following section uses brain magnetic resonance imaging as an example to specifically illustrate the multimodal medical image segmentation method with dynamic gating and adaptive feature fusion provided in this embodiment.

[0127] Step 1: Acquire multimodal magnetic resonance imaging (MRI) images.

[0128] Specifically, historical medical images were acquired. These images included multimodal brain magnetic resonance imaging (MRI) images from the BraTS2018 and BraTS2021 datasets, along with their corresponding tumor segmentation annotations. The BraTS2018 dataset contained 285 cases, each with four MRI modalities (T1, T1ce, T2, and Flair) and three manually annotated tumor regions: enhancing tumor (ET), tumor core (TC), and whole tumor (WT). The BraTS2021 dataset is one of the largest and highest-quality publicly available brain tumor segmentation benchmarks in recent years, containing 1251 cases with significant differences in acquisition equipment and parameters, better simulating modal heterogeneity in clinical settings. Compared to BraTS2018, BraTS2021 showed improvements in annotation consistency and signal-to-noise ratio, providing a reliable basis for validating the model's robustness. The historical medical images were divided into five non-overlapping subsets using a five-fold cross-validation method. One subset was selected as the test set, and the remaining four subsets were used as the training set. This process was repeated five times until all training and testing were completed. Furthermore, each multimodal MRI image in both the training and test sets underwent preprocessing, including spatial registration, resampling to a uniform voxel spacing, intensity normalization, and cropping / filling to a uniform size, to obtain multimodal MRI data that met the input requirements of the subsequent neural network model.

[0129] Step 2: Establish a medical image segmentation network that remains highly robust even in scenarios with missing or damaged modalities.

[0130] Specifically, Figure 2 The overall DGAF framework of this application is shown. Its network structure includes a random modality missing simulator, a modality independent coding module, a modality quality-aware scoring module, and a dynamic gating controller.

[0131] First, the network input consists of complete MRI data with arbitrary modal combinations. During the training phase, this application processes each modality in the input batch using a random modality missing simulator, randomly zeroing out or adding noise to one or more modalities to simulate the complex and varied data incompleteness in clinical settings. Subsequently, the preprocessed, incomplete multimodal data is fed into independent feature encoding to ensure sufficient extraction of the specific features of each modality and avoid mutual interference from early cascading. At the encoder's end, a quality-aware scoring sub-network first generates a quality score in the [0,1] interval for the feature map of each modality, and fuses it with a binary availability mask (0 for missing, 1 for present). Then, a dynamic gating controller is used to jointly constrain "quality × availability" as a gating prior, combining the features and quality scores of all available modalities to dynamically predict a set of optimal fusion weights. Finally, these weights are weighted to obtain adaptive fusion features. This fusion feature is ultimately fed into the decoder to generate a segmentation map. The entire network is trained end-to-end, and by jointly optimizing the segmentation loss and quality prediction loss, the model learns to make accurate decisions under conditions of missing modes or poor quality.

[0132] More specifically, Figure 2 The Random Modal Missing Simulator in the training phase is used to randomly miss and perturb the multimodal MRI inputs obtained in step 1. Figure 3 As shown, this simulator, in each iteration, probabilistically masks some input modalities to simulate multimodal loss or damage caused by differences in acquisition conditions, equipment, or incomplete data uploads in real clinical settings, thereby improving the model's adaptability to modal loss and damage scenarios.

[0133] For each sample in the training set , usually includes Modality These correspond to Flair, T1, T1ce, and T2, respectively. For each training sample, the system randomly generates a binary mask for each modality. ,in Follows Bernoulli distribution:

[0134]

[0135] in represents the modality missing probability. When... When, it indicates that the modality is completely zeroed and is "missing" in the current training iteration; when This mode remains available. The input mode after masking. for:

[0136]

[0137] Among the settings (For example This ensures that the network sees different combinations of missing modalities in each training iteration, such as missing only Flair, missing T1ce+T2, etc. Furthermore, to further simulate poor-quality modalities (such as those containing noise or motion artifacts), this application replaces the "missing" operation with a "damaged" operation with a low probability. At that time, there is a certain probability that this application will not include Instead of setting it to zero, high-intensity Gaussian noise or random affine transformations are added to simulate quality degradation. This design forces the network not only to learn to handle modal missingness but also to learn to identify and suppress information from low-quality modes.

[0138] This application constructs 14 typical modality missing combinations (including 4 single-modal missing combinations, 6 bimodal missing combinations, and 4 trimodal missing combinations) on the BraTS2018 and BraTS2021 datasets, and independently evaluates the model performance for each combination. Through this mechanism, the model is exposed to arbitrary subset modal input distributions during the training phase, and the subsequent dynamic gating controller automatically adjusts the weight allocation to achieve adaptive response to different modal availability and quality. This not only improves DGAF's ability to identify missing modalities, but also enhances its generalization ability under cross-mechanism and non-ideal input conditions.

[0139] The independent encoder configures an independent feature extraction branch for each modality, and for each modality input... Multi-scale features are extracted using an independent 3D convolutional subnet. ,in , These represent the number of channels, height, width, and depth, respectively, yielding the corresponding modal feature set. For missing or noisy modes (i.e.... Its encoder input is a zero tensor, but it still outputs a set of features (the principle of output features is the same as that of non-missing modalities), ensuring the network architecture's compatibility with any combination of inputs. Furthermore, to further simulate poor-quality modalities (such as those containing noise or motion artifacts), this application replaces the "missing" operation with a "damaged" operation with a low probability. There is a certain probability that it will not be Instead of setting it to zero, high-intensity Gaussian noise or random affine transformations are added to simulate quality degradation. This design forces the network not only to learn to handle modality loss but also to learn to identify and suppress information from low-quality modalities. This split-then-fuse approach avoids distribution bias caused by early cascading, allowing each modality to form a stable and transferable representation within its own channel, ensuring that the specific features of each modality can be fully extracted.

[0140] The quality perception score is the output feature of each layer. First, it is compressed into channel description vectors using global adaptive average pooling (GAP). .

[0141] Then, features are filtered and mapped using a perceptron (MLP):

[0142]

[0143] in , Compression ratio , For the Sigmoid function, the output is... This is the quality score of the mode in that layer. This score reflects the reliability of the modality's features at the current sample and scale.

[0144] In addition, the system also inputs an availability mask. (Missing mode 0, present mode 1), joint weight priors are formed through dot product:

[0145]

[0146] This prior is used in the subsequent dynamic gating fusion controller to suppress missing modal weights and enhance high-quality modal contributions, thereby achieving joint modeling of soft quality and hard availability.

[0147] To effectively address the information imbalance problem caused by modal quality inhomogeneity and missing features in multimodal medical images, the core design of this application, a dynamic door controller, is located between the encoder and decoder. It achieves dynamic weighted fusion of modalities at the feature level, thereby fully utilizing high-quality modal information and suppressing interference from low-quality or missing modal features. This part can adaptively generate fusion weights based on the semantic feature distribution, availability mask, and quality score of each modality, realizing nonlinear selection and dynamic reconstruction of cross-modal features.

[0148] The dynamic gating controller takes high-level semantic feature maps from four modalities as input. and its corresponding perceived quality score. Modal availability mask The joint constraint of "quality × availability" is used as a gating prior, and the output is a unified feature representation after fusion. This is used as the decoder input for subsequent segmentation prediction. To reduce computational complexity and enhance semantic comparability between modalities, two consecutive... The 3D convolutional layer performs compressed mapping on features of each modality:

[0149]

[0150] Among them, Conv1 and Conv These are convolutional layers with 32 kernels each, a stride of 1, and padding of 1; BN represents batch normalization. This represents the ReLU activation function. This structure extracts more discriminative intermediate semantic representations while maintaining spatial resolution. Subsequently, global average pooling was used to extract the 3D features. Compressed into a one-dimensional vector This reflects the global distribution of modes along the channel dimension. Then, the modal quality score is calculated. Embedded in the channel description:

[0151]

[0152] After a two-layer MLP:

[0153]

[0154] in , The activation function is ReLU, and Softmax is used to ensure that the weights of all modes are normalized. If a mode is missing, the corresponding mask is used. At this time, the weight The network automatically blocks this modal channel.

[0155] After obtaining the modal weights Then, weighted fusion is performed on all modal features:

[0156]

[0157] The resulting fused multimodal semantic feature representation is then fed into the decoder to generate a segmentation map. This fusion method not only preserves the salient information of high-quality modalities but also relies on other modalities to complete information when a modality is missing, thereby enhancing segmentation robustness.

[0158] More specifically, the total loss of the model can be expressed as:

[0159]

[0160] in, Main segmentation loss, For quality auxiliary loss, For dynamic gating balance constraints, and These are the weighting coefficients.

[0161] In multimodal medical image segmentation tasks, class imbalance is a common problem, and using cross-entropy (CE) alone can easily lead to the neglect of small lesion regions. Therefore, this application adopts a combined approach of Dice loss and cross-entropy loss, simultaneously considering both region overlap accuracy and pixel classification consistency. Its definition is:

[0162]

[0163]

[0164]

[0165] in Indicates the model's predicted probability. Indicates the true label, This is a smoothing constant (to prevent the denominator from being zero). For the number of categories, The total number is 1. Dice loss is sensitive to region overlap and can effectively improve the accuracy of tumor boundary prediction; cross-entropy loss focuses on pixel-level prediction consistency, and the combination of the two significantly improves the overall segmentation performance of the network.

[0166] Furthermore, since this application trains the network under conditions of missing random modalities, the number of different modal input combinations grows exponentially. Without constraints, the network may experience performance bias under some modal combinations. Therefore, this application introduces a modality regularization term:

[0167]

[0168] in Representing modes The encoder output feature mean, This represents the global mean of all modal features. This constraint ensures that the network maintains a consistent modal distribution at the feature level, thereby improving the fusion stability and generalization ability under different modal combinations.

[0169] In the implementation, this regularization term is computed during the DGAF input stage, coinciding with the modal mask. The combined effect exists only in this mode. When a mode is missing, it participates in gradient propagation to avoid gradient misleading for missing modes, so that the model can still converge stably even when modes are missing.

[0170] To prevent the dynamic gating controller from relying too heavily on a single mode, DGAF also introduces a gating balance constraint:

[0171]

[0172] Maintaining a balanced distribution of modal weights ensures the network maintains a reasonable proportion among different modal features. By appropriately penalizing excessive weight bias, it prevents the network from relying solely on a single high-quality modality and ignoring auxiliary information from other modalities in the later stages of training. This mechanism complements the modal masking mechanism: the masking removes missing modalities, while gating constraints balance the weights among available modalities, enabling DGAF to maintain robust fusion behavior even in modality-deficient scenarios.

[0173] Step 3: Train the robust segmentation model DGAF established in Step 2.

[0174] Specifically, the segmentation network described in step 2 is trained end-to-end, enabling the model to learn to make accurate decisions under conditions of missing or poor-quality modalities. During training, a training set is first constructed based on the multimodal MRI data and their corresponding segmentation annotations obtained in step 1. Uniform spatial alignment, resampling, and intensity normalization are then performed on each modality image to ensure consistent network input size and voxel spacing.

[0175] To ensure the stability and generalization of the training results, this application employs a five-fold cross-validation experiment: all training samples are divided into five non-overlapping subsets. Each time, one subset is selected as the validation set, and the remaining four subsets are used as the training set. This process is repeated five times, and the results from the five iterations are averaged or the optimal model weights are selected for subsequent inference. During each cross-validation training iteration, for each training iteration, a batch of multimodal inputs is fed into a random modality missing data simulator. For each modality, missing data is zeroed out or quality degradation perturbations (such as noise injection, blurring, or intensity perturbation) are independently executed according to a preset probability. This continuously generates diverse missing data combinations and damage patterns during the training phase and simultaneously updates the modality availability mask.

[0176] Subsequently, the simulated multimodal inputs are fed into modality-independent coding branches to extract modality feature representations, and the quality score of each modality is output by the modality quality perception score. The quality score and the usability mask are jointly constrained and used as the gating prior input to the dynamic gating controller. The controller adaptively and dynamically generates the fusion weights of each modality and obtains the fusion features. The fusion features are then decoded to output a segmentation probability map.

[0177] The training objective employs a joint optimization of segmentation loss, quality-assisted loss, and dynamic gating balance constraints. The segmentation loss is preferably a combination of Dice loss and cross-entropy loss to improve the consistency between predicted segments and ground truth annotations. The quality-assisted loss standardizes the matching relationship between quality scores and modality missing or degraded states, causing the weights of missing modalities to approach zero and suppressing the weights of low-quality modalities. The parameters of the modality-independent encoder, modality quality score, dynamic gating controller, and decoder are iteratively updated through backpropagation, and periodically evaluated at the validation fold to preserve the optimal weights. Ultimately, a training model with robust segmentation capabilities is obtained even in scenarios with arbitrary modality missing or quality degradation.

[0178] Step 4: Based on the robust segmentation model trained in Step 3, perform segmentation inference on the multimodal MRI images in Step 1 to obtain the segmentation results.

[0179] Specifically, the multimodal MRI data to be segmented is input into the DGAF model. The network first automatically constructs a modality availability mask based on the input, and skips the random simulation module during the inference phase, directly encoding and calculating the quality score of the real modalities independently. Subsequently, the dynamic gating controller generates adaptive fusion weights based on the features of each modality and the prior quality availability, obtains fusion features, and outputs a segmentation probability map through the decoder. Thresholding or maximum probability selection is performed on the probability map to obtain the final segmentation label. The predicted segmentation result is compared with the corresponding ground truth label, and the Dice similarity coefficient (DSC) and intersection-over-union ratio (IoU) are calculated as evaluation indicators to verify the segmentation performance and robustness of the model under conditions of modality loss or impaired modality quality, thereby obtaining the medical image segmentation result.

[0180] Multimodal magnetic resonance imaging (MRI) is widely used for brain tumor segmentation by providing complementary tissue features, but its performance is severely affected by modality loss and damage in clinical settings. Traditional multimodal medical image segmentation methods often rely on complete modality input or fixed fusion strategies and processing of specific missing modalities, making it difficult to flexibly and effectively address the problem of incomplete features and different combinations of missing modalities, which seriously restricts the performance and robustness of existing segmentation methods. To address this challenge, this application proposes a multimodal brain tumor segmentation method based on Dynamic Gated Adaptive Feature Fusion (DGAF) to improve the segmentation accuracy and robustness in missing scenarios. Specifically, firstly, this application constructs a random modality loss simulator to enhance the model's adaptability to different modality combinations, forcing the network to learn robust representations independent of any specific modality. Secondly, the feature credibility is evaluated in real time using a lightweight modality quality-aware scoring system designed in this application. Subsequently, each modality feature is combined with its quality score and introduced into the proposed dynamic gating controller. Adaptive fusion weights are dynamically predicted based on the currently available modalities and their quality, thereby effectively suppressing interference from missing or low-quality modalities. Extensive experimental results on the BraTS2018 and BraTS2021 brain tumor segmentation datasets show that the DGAF proposed in this application achieves the highest average DSC (Digital Subtraction Capacity) in different modality loss scenarios: WT=89.44%, TC=86.64%, and ET=75.02%, all of which are significantly better than existing methods, demonstrating superior segmentation robustness and clinical application potential.

[0181] To verify the effectiveness and robustness of the proposed multimodal medical image segmentation network based on Dynamically Gated Adaptive Feature Fusion (DGAF) in scenarios with missing and damaged modalities, this application conducted systematic experiments on two publicly available brain tumor segmentation datasets, BraTS2018 and BraTS2021. Five-fold cross-validation was used in both experiments to ensure stable and reliable results. To simulate real-world scenarios of missing modalities in a clinical setting, this application generated 14 different combinations of missing modalities based on complete multimodal samples. Each missing modality scenario was independently used for model inference and evaluation to verify the adaptability of DGAF under arbitrary missing modality conditions.

[0182] To comprehensively evaluate the performance of the DGAF network, this application selected the following representative multimodal segmentation methods for comparison: nnU-Net: an adaptive multimodal segmentation benchmark model; HeMIS: an early robust method for missing modalities, achieving robust inference through modal mean fusion; U-HVED: an implicit modal compensation model based on variational autoencoders; RobustMSeg: achieving robust feature fusion through latent space alignment; RFNet: employing a multi-branch modal reconstruction and fusion strategy; M2FTrans: a modal adaptive fusion network based on Transformer. All comparison models were implemented under the same data preprocessing, five-fold partitioning, and training strategies to ensure fairness. The performance of all methods was systematically evaluated in 14 missing modal scenarios, including single-modal missing (4 scenarios), bimodal missing (6 scenarios), and trimodal missing (4 scenarios). Evaluation metrics included: DSC (Dice Similarity Coefficient) to measure the overlap between the predicted and ground truth segments; and Iou to measure the spatial consistency of the predicted results.

[0183] Table 1 shows the DSC comparison results of the proposed DGAF and several representative methods on the BraTS2018 dataset under different missing modal combinations. In the WT (Whole Tumor) region, DGAF achieves high performance across all modality combinations, especially maintaining over 90% performance even with missing key modalities (such as T1ce or T2), significantly outperforming M. 2 FTrans. DGAF also performed exceptionally well in the TC (Tumor Core) region, with particularly significant improvements under conditions of missing most modalities, achieving an average DSC of 86.64%, significantly outperforming other methods. For the most challenging ET (Enhancing Tumor) region, DGAF still achieved the highest average performance of 66.30%, compared to M. 2 FTrans improved by approximately 2.43%. Overall, DGAF achieved the best average DSC in all three tumor regions (WT, TC, and ET), fully validating the feature contributions of the introduced random modality loss simulation, modality quality-aware scoring, and dynamic gating controller adaptively adjusted under challenges such as modality loss and quality degradation, thus achieving more stable and robust segmentation performance. These results fully validate the effectiveness of DGAF in addressing real-world clinical multimodal incompleteness issues.

[0184] Table 1. Comparison of DSC (%) between BraTS2018 dataset and existing methods in different modality missing scenarios.

[0185]

[0186]

[0187] Tables 2 and 3 show the comparison results of DGAF with baseline nnU-Net and Dropout models on the BraTS2021 dataset under 14 different modality loss combinations in terms of Dice coefficient and Iou index. It can be seen that DGAF exhibits significant advantages in all loss combinations for the WT, TC, and ET tumor regions. In terms of the Dice index (Table 2), DGAF demonstrates stronger robustness to the three tumor regions. The average Dice values ​​for WT, TC, and ET reach 0.934, 0.857, and 0.750, respectively, significantly higher than the nnU-Net and Dropout models. This trend is also consistent with the more stringent Iou index (Table 3): the average Iou values ​​for WT, TC, and ET increase to 0.812, 0.737, and 0.720, respectively. It is noteworthy that the ET region is most sensitive to modality information, but DGAF maintains stable performance under various loss scenarios, indicating that the dynamic gating fusion strategy has stronger robustness in handling missing modalities. Together, they verified that DGAF effectively improves segmentation accuracy and structural consistency even when multimodal information is incomplete, and its advantages are demonstrated in all missing scenarios.

[0188] Table 2. Comparison of Dice metrics between BraTS2021 dataset and other models.

[0189]

[0190] Table 3. Comparison of Iou metrics with other models on the BraTS2021 dataset.

[0191]

[0192] Table 4 shows the average DSC performance of different segmentation methods across all modality loss combinations on the BraTS2021 dataset, i.e., the average segmentation performance of the three tumor regions (WT, TC, and ET) under arbitrary modality loss scenarios (including single-modality loss, bimodality loss, and even multimodality loss). This table provides a more comprehensive reflection of the overall robustness under complex and variable clinical modality loss conditions. The results show that the proposed DGAF method achieves the best or significantly leading average DSC in all three tumor regions. Specifically, in the WT region, DGAF achieves an average DSC of 89.44%, outperforming current mainstream methods, validating that DGAF's dynamic gating mechanism can more effectively integrate key structural information under multimodal incompleteness conditions. In the TC region, DGAF achieves the highest average DSC of 85.73%, significantly surpassing M... 2FTrans (83.20%) and MAML (81.49%) demonstrate their advantage in addressing structural heterogeneity within the core tumor region. In the ET region, DGAF remains the leading choice (75.02%), compared to M... 2 Both FTrans (73.76%) and RFNet (70.13%) showed significant improvements.

[0193] Table 4. Comparison of DSC (%) between BraTS2021 dataset and existing methods in different modality missing scenarios.

[0194]

[0195] Overall, DGAF outperforms existing methods on average in any modality loss scenario, further validating its strong robustness to modality loss, cross-modal quality modeling ability, and the effectiveness of its dynamic adaptive fusion strategy. These results fully demonstrate that DGAF has stronger generalization and reliability under diverse imaging conditions in real-world clinical settings.

[0196] Tumor segmentation visualization:

[0197] To further evaluate the structural consistency and visual robustness of DGAF under different modality loss scenarios, this application further demonstrates typical case images (such as...) on the BraTS2021 dataset. Figure 8 , Figure 9 The results show the segmentation results of DGAF and baseline nnU-Net under different modal combinations (such as Flair deletion and T1+T1ce deletion). When the key modality is missing, the segmentation results of nnU-Net show significant degradation, especially with large areas of missed detection and false positives at the tumor boundary.

[0198] like Figure 9 The presentation showcases the complete modalities corresponding to four original MRI modalities (T1, T1ce, T2, and Flair). After simulating the input with the T1+T1ce modality missing, the segmentation results on nnU-Net are observed: WT (green), TC (red), and ET (blue) regions exhibit blurred boundaries and structural collapse when the T1+T1ce modality is missing, and severe missed detections occur in ET, demonstrating a high dependence on specific modality-specific model bias. In contrast, DGAF maintains more stable tumor morphology and structure and more complete region coverage under the same modality-missing condition. The ET core is intact, clearly identifiable, and has compact boundaries that more closely resemble real-world annotations; the WT and TC structures also remain highly stable. This further illustrates that DGAF's dynamic gating adaptive fusion mechanism can dynamically adjust according to the availability of each modality, thus maintaining reliable feature representation even with modality loss.

[0199] Overall, the visualization results are consistent with the quantitative evaluation indicators, demonstrating the boundary stability, regional consistency, and structural recovery ability of DGAF under modal incompleteness conditions. This further confirms that DGAF maintains excellent segmentation performance in different modal missing scenarios, significantly improving the model's clinical robustness and practical clinical applicability.

[0200] Ablation experiment:

[0201] To comprehensively evaluate the independent contributions and synergistic gains of each component in the DGAF framework, this application designed systematic ablation experiments for three key components—the Random Modal Missing Simulator (RMMS), the Modal Quality Perceived Score (QPS), and the Dynamic Gated Controller (DDC). Table 5 shows the performance variations of the model on the BraTS2018 dataset under different combinations.

[0202] Table 5. Results of ablation experiments for each part

[0203]

[0204] First, when only RMMS is enabled, without considering modality quality differentiation and dynamic fusion strategies, the random modality missing simulator mainly enhances data diversity and improves model robustness, but still struggles to fully utilize incomplete modality information. Introducing QPS further improves the overall model performance compared to the RMMS-only version, with an average DSC improvement of nearly 1.2%, indicating that introducing quality-aware scoring effectively measures the reliability of each modality, making the network more selective in the face of artifacts, noise, and modality differences. Meanwhile, applying only DDC without QPS results in a similar performance improvement to QPS, with the advantage primarily stemming from the dynamic fusion mechanism's ability to adaptively adjust the importance of different modality features, making the network more structured and flexible in its expression. When both QPS and DDC are enabled, the performance increase further expands, improving by approximately 0.57%–1.83% compared to individual modules, demonstrating a clear complementary relationship between quality-awareness and dynamic gating: the former provides a stable modality reliability assessment, while the latter relies on this reliability to achieve targeted dynamic fusion. Experimental results show that the three have a significant synergistic effect: RMMS provides a diversified training space, QPS provides modal quality priors, and DDC provides dynamic fusion capabilities, which together constitute the core mechanism for improving the robustness and generalization ability of the model in scenarios with missing modalities and unstable quality.

[0205] This application addresses the common problems of modality loss and quality degradation in clinical multimodal MRI brain tumor segmentation. It proposes a multimodal medical image segmentation framework based on Dynamic-Gated Adaptive Fusion (DGAF). This framework enhances generalization ability through a random modality loss simulator, avoids mutual interference from early stitching by modality-independent encoding, estimates the credibility of each modality feature using quality-aware scoring, and adaptively predicts fusion weights using a dynamic gating controller. While maintaining a lightweight model structure, it improves robustness to arbitrary combinations of missing modalities. Experimental results show that DGAF achieves significant performance improvements on BraTS2018 and BraTS2021, especially maintaining relatively stable tumor boundary and core region segmentation capabilities even with missing key modalities, outperforming HeMIS, U-HVED, RobustMSeg, and M... 2 AE, M 2 Methods such as FTrans and baseline nnU-Net were used. Further analysis of tumor segmentation visualization demonstrated that DGAF can effectively mitigate interference from artifact modalities, improve the coherence of tumor internal structures, and significantly reduce missed and false detections.

[0206] DGAF offers a solution that adapts to the incompleteness of real-world clinical modalities, improving segmentation accuracy and robustness in scenarios with missing or compromised modalities. It provides a feasible, effective, and clinically promising new approach for robust segmentation of brain tumors in multimodal MRI scenarios with missing modalities. Future work will further combine implicit modality compensation and teacher-student distillation strategies to achieve modality-invariant feature learning, enhancing the network's robustness and cross-domain adaptability under extreme modality-missing conditions. Furthermore, it can be extended to more organs and multi-center datasets to improve inter-domain generalization performance.

[0207] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection.

[0208] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0209] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0210] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0211] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0212] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0213] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0214] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0215] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal medical image segmentation method with dynamic gating and adaptive feature fusion, characterized in that, include: Obtain a historical medical image dataset; the historical medical image dataset includes multiple samples and the segmentation and annotation results of each sample; Each of the samples includes multiple single-modal magnetic resonance imaging images of the target region; A dynamic gated adaptive feature fusion architecture and a model total loss function are constructed. The dynamic gated adaptive feature fusion architecture includes: a stochastic modality missing simulator and a medical image segmentation network connected in sequence; the medical image segmentation network includes: an independent feature encoding module, a quality-aware scoring sub-network, a dynamic gate controller, and a decoder connected in sequence. The dynamic gating controller includes: a weighted feature fusion module and multiple modality weight determination branches; The outputs of all the modality weight determination branches are connected to the weighted feature fusion module; The modal weight determination branch is set to correspond one-to-one with the modality of the sample; The modality weight determination branch includes a first three-dimensional convolutional layer, a second three-dimensional convolutional layer, a global average pooling layer, and a two-layer perceptron mechanism connected in sequence. The first three-dimensional convolutional layer and the second three-dimensional convolutional layer have the same structure; The first three-dimensional convolutional layer includes: a first convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence; The working principle of determining the branch for the mode weights corresponding to the m-th mode is as follows: ； ； ； In the formula, The intermediate semantic representation of the high-level semantic feature mapping of the m-th modality; and Both represent the ReLU activation function; Indicates batch normalization; Indicates the second convolutional layer; Indicates the first convolutional layer; Represents a high-level semantic feature map of m modalities; Indicates intermediate semantic representation A one-dimensional vector obtained by compression through a global average pooling layer; This represents the modal quality score of the m-th mode; For a one-dimensional vector The embedding vector obtained by embedding modal quality scores into the channel description; Let be the modal weights of the m-th mode; This represents a two-layer sensing mechanism; This represents the Softmax activation function; This represents the weight matrix of the second fully connected layer; This represents the weight matrix of the first fully connected layer; This represents the bias vector of the first fully connected layer; This represents the bias vector of the second fully connected layer; The working principle of the weighted feature fusion module is as follows: ； In the formula, This represents the fusion feature; M is the total number of modalities in the sample. Using the historical medical image dataset and the model's total loss function, the dynamic gated adaptive feature fusion architecture was trained using a five-fold cross-validation experiment, and the trained medical image segmentation network was determined to be the segmentation model. Acquire multimodal magnetic resonance imaging images of the target region to be segmented; The multimodal magnetic resonance imaging image to be segmented is input into the segmentation model to obtain the segmented image of the target region.

2. The multimodal medical image segmentation method with dynamic gating adaptive feature fusion according to claim 1, characterized in that, After obtaining the historical medical image dataset, the following is also included: The acquired historical medical image dataset is preprocessed; the preprocessing includes: spatial registration processing, uniform voxel spacing processing, intensity normalization processing, and uniform size processing. The unified voxel spacing processing is achieved through resampling; The uniform size processing is achieved through cropping and padding.

3. The multimodal medical image segmentation method with dynamic gating adaptive feature fusion according to claim 1, characterized in that, For each sample in the historical medical image dataset, the random modality missing simulator is used to randomly generate a binary mask as an availability mask for each modality, and multiply the availability mask with the corresponding single-modality magnetic resonance imaging image to obtain the input sample after mask processing; The availability mask is: ； ； in, Represents the availability mask for the m-th mode; when When, it means setting the m-th mode to zero, or adding Gaussian noise to the m-th single-mode magnetic resonance imaging image, or performing a random affine transformation on the m-th single-mode magnetic resonance imaging image; when When, it means that the m-th modality remains available in the current training iteration; Indicates the Bernoulli distribution; Indicates the probability of mode absence; The input mode obtained after masking the m-th single-mode magnetic resonance imaging image in the sample is: ； in, This is the m-th input mode; This is the m-th single-mode magnetic resonance imaging image.

4. The multimodal medical image segmentation method with dynamic gating adaptive feature fusion according to claim 3, characterized in that, The independent feature encoding module includes multiple feature extraction branches; The feature extraction branches are set to correspond one-to-one with the modalities of the samples; The feature extraction branch is a 3D convolutional subnet; The feature extraction branch is used to obtain multi-scale features of the corresponding modality. ; in, ; In the formula, This indicates that the feature extraction branch corresponding to the m-th mode is... l The output features of the layer; This indicates the total number of layers in the feature extraction branch; Indicates the feature extraction branch number 1 l Number of channels per layer; Indicates the feature extraction branch number 1 l Floor height; Indicates the feature extraction branch number 1 l Layer width; Indicates the feature extraction branch number 1 l Layer depth.

5. The multimodal medical image segmentation method with dynamic gating adaptive feature fusion according to claim 4, characterized in that, For the feature extraction branch corresponding to the m-th mode, the... l Layer output features The quality-perceived scoring subnetwork is used for: The output features are processed by global adaptive average pooling. Compressed into a channel description vector, the m-th mode is obtained in the first channel. l Layer channel description vector ; ; Using a perceptron to describe the channel vector Perform feature filtering and mapping to obtain the m-th mode. l The perceptron output corresponding to the layer ; ; Output of the sensor As the m-th mode in the... l Layer quality score ; .

6. The multimodal medical image segmentation method with dynamic gating adaptive feature fusion according to claim 1, characterized in that, The total loss function of the model is: ； ； ； ； ； ； In the formula, This represents the total loss of the model; Indicates the primary segmentation loss; This represents the weighting coefficient for quality-assisted loss; Indicates quality auxiliary loss; Indicates the weight coefficient of the dynamic gating balance constraint term; This represents the dynamic gating balance constraint term; Indicates Dice loss; Represents cross-entropy loss; Indicates the model's predicted probability; Indicates the true label; It is a smoothing constant; For the number of categories, The total number; This represents the mean value of the encoder output features for the m-th mode. This represents the global mean of all modal features.

7. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the multimodal medical image segmentation method of dynamic gating adaptive feature fusion as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the multimodal medical image segmentation method with dynamic gating adaptive feature fusion as described in any one of claims 1-6.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the multimodal medical image segmentation method with dynamic gating adaptive feature fusion as described in any one of claims 1-6.

Images