An uncertainty-based missing modality adaptive completion method for brain tumor segmentation

Abstract

This invention discloses an uncertainty-guided modality adaptive completion brain tumor segmentation method, belonging to the field of medical image segmentation technology. The invention constructs an uncertainty-guided modality adaptive completion segmentation network, UniMAC, using a single encoder and single decoder as the backbone network. Through the modality adaptive completion MAC mechanism, it dynamically completes the feature information of any missing modality using learnable modules and two masking operations. It introduces modality-aware uncertainty loss, combining Dice Loss, weighted Softmax Loss, and Focal Loss, and dynamically balances multi-dimensional supervision signals through learnable weights. This invention is validated on the BraTS2018, BraTS2020, and BraTS2021 datasets, achieving high-precision brain tumor segmentation in any modality-missing scenario. It effectively reduces the number of model parameters and computational overhead, improves segmentation robustness and boundary accuracy, and solves the technical problems of poor modality mismatch, high computational cost, and fixed loss weights in incomplete multimodal brain tumor segmentation.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation technology, and in particular to a brain tumor segmentation method based on uncertainty-based adaptive completion of missing modalities. Background Technology

[0002] Brain tumor segmentation is a crucial step in clinical diagnosis and treatment. Multimodal MRI imaging, due to its ability to provide complementary tumor information, has become the mainstream data format for brain tumor segmentation. However, modality loss often occurs during clinical acquisition, leading to a significant decline in the performance of existing segmentation models.

[0003] Existing incomplete multimodal brain tumor segmentation methods mainly suffer from the following drawbacks:

[0004] The number of parameters in a multi-encoder single-decoder architecture increases dramatically, and the computational cost grows exponentially, making it difficult to deploy applications.

[0005] Modal missing completion methods often use pre-trained general modules or fixed masks set to zero, which cannot dynamically adapt to the context differences of different missing modal combinations, resulting in insufficient completion accuracy.

[0006] The loss function uses a fixed weight combination, which cannot quantify the uncertainty caused by modality loss and is difficult to balance multi-dimensional supervision signals of region, category and pixel, resulting in poor segmentation effect for small target tumor regions.

[0007] Uncertainty theory has not been systematically applied in the field of incomplete brain tumor segmentation, and the model has a weak ability to suppress noise and fuzzy features.

[0008] Therefore, there is an urgent need for a lightweight brain tumor segmentation method that can dynamically complete missing modalities and adaptively optimize loss, in order to solve the technical problems of low segmentation accuracy, high computational cost and poor robustness in modality missing scenarios. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide an adaptive completion method for brain tumor segmentation based on uncertainty-based missing modalities.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] An uncertainty-based missing modality adaptive completion brain tumor segmentation method includes:

[0012] S1: Construct a UniMAC segmentation network with a single encoder and a single decoder. Input the four MRI modalities FLAIR, T1, T1c, and T2 into independent residual blocks for feature extraction to obtain high-level features of the four modalities. Then, concatenate the high-level features in the channel dimension to form concatenated multimodal features.

[0013] S2: Apply a masking operation to the spliced ​​multimodal features to simulate mode loss. Through the mode adaptive completion MAC mechanism, combined with the learnable module and two masking TM operations, dynamically complete the feature information corresponding to the missing mode to obtain the completed multimodal features.

[0014] S3: Input the completed multimodal features into the 3D-UNet segmentation network and output the brain tumor segmentation results;

[0015] S4: Construct a modality-aware uncertainty loss, combine Dice Loss, weighted Softmax Loss, and FocalLoss, dynamically adjust the contribution of each loss term through learnable weights, and combine deep supervision to perform single-stage end-to-end training on the UniMAC segmentation network;

[0016] S5: Input the MRI multimodal data with missing modalities into the trained UniMAC segmentation network to obtain the final brain tumor segmentation result.

[0017] Furthermore, the modality adaptive completion (MAC) mechanism specifically involves: creating a learnable module with the same size as the input features; retaining valid modal features through two mask TM operations; and replacing the missing modal positions with compensation features obtained by residual fusion of the learnable module and the existing modal.

[0018] Furthermore, the formula for calculating the modality-sensing uncertainty loss is as follows: ,in, It is the number of sub-loss functions participating in the joint optimization. It is the first The value of each sub-loss function, Is with the first The learnable parameters associated with each sub-loss function are initialized to 1.

[0019] Furthermore, the two masking TM operations are as follows: the first mask retains the valid modal features, and the second mask replaces the inverted missing modal positions with learnable module features.

[0020] Furthermore, the UniMAC segmentation network employs single-stage end-to-end training, eliminating the need for pre-trained network generation.

[0021] Compared with the prior art, the technical solution disclosed in this invention has the following beneficial effects:

[0022] By employing a single encoder and single decoder with residual block feature extraction, compared to a multi-encoder architecture, the number of model parameters and computational overhead are significantly reduced, resulting in lower memory usage, faster inference speed, and lightweight deployment.

[0023] Through the MAC mechanism and two masking operations, the correlation between modalities can be dynamically learned, and arbitrary missing modal features can be adaptively completed, avoiding feature distortion caused by zero-filling and improving the segmentation robustness in modal missing scenarios.

[0024] By introducing modality-aware uncertainty loss and dynamically balancing multi-dimensional losses through learnable weights, high-confidence features are prioritized and noise interference is suppressed, significantly improving the segmentation accuracy of small target regions such as tumor core TC and enhanced tumor ET.

[0025] It achieves end-to-end single-stage training, eliminating the need for pre-trained generative networks, resulting in faster model convergence, lower risk of overfitting, and superior segmentation accuracy compared to existing mainstream models. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic diagram of the process of the uncertainty-based missing modality adaptive completion brain tumor segmentation method provided in the embodiments of the present invention;

[0028] Figure 2 This is a diagram of the UniMAC model architecture provided in an embodiment of the present invention;

[0029] Figure 3 This is a diagram of a MAC module provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] like Figure 1 As shown, this embodiment of the invention provides an adaptive completion method for brain tumor segmentation based on uncertain missing modalities, including:

[0033] S1: Construct a UniMAC segmentation network with a single encoder and a single decoder. Input the four MRI modalities FLAIR, T1, T1c, and T2 into independent residual blocks for feature extraction to obtain high-level features of the four modalities. Then, concatenate the high-level features in the channel dimension to form concatenated multimodal features.

[0034] The UniMAC segmentation network employs single-stage end-to-end training, eliminating the need for pre-training to generate the network.

[0035] S2: Apply a masking operation to the spliced ​​multimodal features to simulate mode loss. Through the mode adaptive completion MAC mechanism, combined with the learnable module and two masking TM operations, dynamically complete the feature information corresponding to the missing mode to obtain the completed multimodal features.

[0036] The modality adaptive completion (MAC) mechanism is as follows: a learnable module with the same size as the input feature is created, effective modal features are retained through two mask TM operations, and the missing modal positions are replaced with compensation features obtained by residual fusion of the learnable module and the existing modality.

[0037] The two masking TM operations are as follows: the first mask retains the valid modal features, and the second mask replaces the missing modal positions after inversion with learnable module features.

[0038] S3: Input the completed multimodal features into the 3D-UNet segmentation network and output the brain tumor segmentation results;

[0039] S4: Construct a modality-aware uncertainty loss, combine Dice Loss, weighted Softmax Loss, and FocalLoss, dynamically adjust the contribution of each loss term through learnable weights, and combine deep supervision to perform single-stage end-to-end training on the UniMAC segmentation network;

[0040] The formula for calculating the modality sensing uncertainty loss is as follows: ,in, It is the number of sub-loss functions participating in the joint optimization. It is the first The value of each sub-loss function, Is with the first The learnable parameters associated with each sub-loss function are initialized to 1.

[0041] S5: Input the MRI multimodal data with missing modalities into the trained UniMAC segmentation network to obtain the final brain tumor segmentation result.

[0042] As medical image analysis increasingly demands multi-task collaboration, uncertainty theory demonstrates greater adaptability.

[0043] To better capture and manage the inherent uncertainty in segmentation tasks, Tsai et al. introduced uncertainty into the loss function. This theory dynamically models data reliability through learnable parameters, assigning differentiated weights to prediction results at different spatial levels, enabling the model to adaptively balance the contributions of multi-source information. The essential difference between this theory and traditional methods lies in its dynamic adjustment mechanism, as shown in formulas (1) and (2). Traditional methods use fixed weight combinations, while the new theory achieves end-to-end uncertainty quantification through learnable parameters, enabling the model to autonomously identify reliable data features while suppressing interference from fuzzy or noisy modes. This theoretical framework has significant paradigm shift value.

[0044] (1)

[0045] (2)

[0046] in , These are hyperparameters, which need to be manually set before training. It is the total number of loss functions, for each loss function Each has corresponding learnable parameters. This is used to adjust the contribution of each loss component based on the estimated uncertainty.

[0047] The specific implementation details and principles of the embodiments of the present invention are as follows:

[0048] UniMAC, a segmentation network with modality adaptive completion, such as Figure 2 As shown, it uses a single encoder and single decoder to achieve the segmentation effect of a multi-encoder, single-decoder system. For example... Figure 2 As shown, UniMAC mainly consists of feature extraction, modality adaptive completion (MAC) mechanism, U-shaped segmentation network (3D-UNet), and modality-aware uncertainty loss (…). It consists of four parts. The specific process is as follows.

[0049] In the feature extraction stage, features of the four input modalities are extracted using residual blocks with independent structures. Compared to traditional multi-encoder architectures, this design significantly reduces the number of model parameters while maintaining the ability to distinguish the imaging characteristics of each modality through residual learning. After feature extraction, the multimodal features resulting from the concatenation of features from the four modalities are processed using a masking mechanism to simulate modality loss scenarios in real-world applications. Before being input into the 3D-UNet segmentation network, UniMAC dynamically learns the potential correlations between modalities through a modality adaptive completion (MAC) mechanism to achieve intelligent compensation for missing information. The multimodal features after modality adaptive compensation are then input into the 3D-UNet segmentation network. This network gradually reconstructs refined tumor segmentation results through multi-scale feature encoding and decoding mechanisms. During the training phase, modality-aware uncertainty loss is employed. Using this as the optimization objective, end-to-end model parameter learning is achieved by minimizing the difference between the predicted results and the expert-annotated true values. Compared to the traditional form of summing custom loss weights, this paper proposes... A three-level reliability assessment framework is adopted to achieve a dynamic balance of contributions from multiple information sources.

[0050] To intelligently compensate for multimodal input features and effectively address the problem of missing modalities, this paper proposes a Modality Adaptive Completion (MAC) mechanism. Its core idea is to construct implicit mapping relationships between modalities through a learnable parameter matrix, enabling dynamic reconstruction of missing features. Specifically, a learnable module replaces the original missing modal input. This module, along with the available modalities, engages in deep interaction through residual learning units to generate compensated features. Finally, the reconstructed features are seamlessly embedded into the network backbone. The principle is as follows: Figure 3 As shown.

[0051] First, the entire network has four modalities of image input. ( ), respectively The feature maps of each modality are concatenated (concat) according to their channels to obtain the total feature map. Its formula is as follows:

[0052] (3)

[0053] At the same time, create a learnable module of the same size. The initial values ​​of the learnable modules are randomly set and adaptively adjusted during the training process. Their purpose is to compensate for missing or anomaly-related modal data, thereby enhancing the model's robustness. When certain modal data is missing, the learnable modules can adaptively generate alternative information.

[0054] To simulate the missing modes phenomenon in multimodal data, it is necessary to... Applying a masking operation yields feature representations with partially missing modes. Its formula is as follows:

[0055] (4)

[0056] in The instruction takes a list of boolean values, where each element is a boolean and represents a modality's mask state: True indicates that the original modality data is preserved, and False indicates that the modality is masked (missing). TM represents two masking operations. The result is that available modes are preserved unchanged, while missing modes are changed. This can be used as a substitute; this allows for flexible handling of missing or abnormal data from different modalities, and the detailed mechanism of TM will be explained in detail later.

[0057] To enable learnable modules to better compensate for missing modal information, Feature maps are obtained by feature fusion using residual blocks. Its formula is as follows:

[0058] (5)

[0059] On the other hand, in the network backbone, the four modalities obtain high-level feature representations through the residual blocks of their respective branches, and the high-level feature representations are concatenated according to channels to obtain feature maps. This is to process the information from each modality independently and then integrate it again to extract richer features. Similarly, to maintain consistency in scenarios with missing input data, [the following is also done]... Obtain by masking Its formula is as follows:

[0060] (6)

[0061] Here The instructions are the same as above. Here, The result is that the available modes are preserved. The missing modes remain unchanged, and the compensation features are obtained by fusing learnable modules and existing modes through residual blocks. This is done to flexibly handle missing or outlier cases in different modalities. This is to ensure that the final feature map contains both valid information from existing modalities and supplementary information from learnable modules, thereby improving the robustness and accuracy of the model. The entire process is shown in Algorithm 1.

[0062]

[0063] Taking the first TM operation as an example, the TM module is explained in detail below, and its principle is as follows:

[0064] Create a dataset of all zeros that is exactly the same size as the input dataset. ,right according to The instruction performs a replacement operation twice. First time:

[0065] (7)

[0066] On dimension 0, middle The original tensor will be preserved at the location of True. The corresponding numerical value. This operation uses a selective copying strategy to migrate data only from regions marked as valid by the mask, thus ensuring that the original data not covered by the mask remains undisturbed and maintaining the integrity of the input features. The second operation is the same:

[0067] (8)

[0068] (9)

[0069] ~ indicates inverting the element, originally... In the instruction, elements that were True become False, and elements that were False become True. The negation operation allows us to replace data under different conditions, increasing the flexibility of data processing. middle The position marked as False will be replaced with a learnable module. The corresponding numerical value. Replacing the elements marked as True after inversion with learnable modules can support subsequent computational tasks, thereby compensating for the information lost due to modality missingness. Simply put, as shown in Algorithm 2, if... ,but ,like ,but .

[0070]

[0071] Based on the preceding analysis, the embodiments of this invention exhibit significant uncertainty characteristics, thus proposing a modality-aware uncertainty loss (MDI). ), through a dynamic weight adjustment mechanism, combined with the region-based loss function Dice Loss ( ), category-based loss function Weighted Softmax Loss ( ) and pixel-based loss function Focal Loss ( Optimization was performed to construct a multi-dimensional loss surface. Details are as follows.

[0072] Region-based loss functions: used to locate lesion positions. Dice Loss directly optimizes based on region overlap and is suitable for segmenting irregularly shaped targets commonly found in medical images.

[0073] The category-based loss function, Weighted Softmax Loss, continues the core idea of ​​handling multi-class imbalance in brain tumor segmentation and solves the class bias problem of traditional methods when modalities are missing.

[0074] Pixel-based loss functions are used to address boundary blurring and sample defocusing. Focal Loss's dynamic adjustment mechanism is applicable to feature learning challenges in multiple tasks.

[0075] In summary, the paper proposes... It integrates supervision signals across three dimensions: region, category, and pixel. To illustrate this framework, the following sections will present Dice Loss, Weighted Softmax Loss, Focal Loss, and... The mathematical definition of .

[0076] Modality-aware uncertainty loss uses a learnable weight module to dynamically adjust the contribution of each loss term to the overall objective function. Its formula is as follows:

[0077] (10)

[0078] It is the number of loss functions. It is a learnable parameter, initialized to 1. In order to prevent It can increase or decrease indefinitely. Compared to the formula for uncertainty loss in formula (2), Will Optimized to The reason is Initialize to 1, then compared to A more reasonable approach would be to initialize the sum of the initial weights of each loss function to 1. In this study, we set... ,in , and , respectively corresponding , and .

[0079] Dice Loss is a loss function built on the Dice Similarity Coefficient (DSC). It optimizes the model by directly comparing the overlap between the predicted segmented regions and the ground truth labeled regions. Its formula is as follows:

[0080] (11)

[0081] in, Indicates the number of segmentation categories. Belongs to the The total number of real voxels of the class The model predicts the first Individual elements belong to categories The predicted probability, It is the first Individual elements in categories The real labels on it It is a smoothing factor close to zero, preventing division by zero errors caused by a denominator of zero. When When this happens, the formula degenerates into the standard DiceLoss for binary segmentation.

[0082] Weighted Softmax Loss is an improved weighted cross-entropy loss, but it is still essentially a cross-entropy loss. It simply assigns weights, giving higher weights to categories that appear less frequently. The formula is as follows:

[0083] (12)

[0084] in, , , , , The meaning is the same as above. It is a category The adaptive weights are calculated using the following formula:

[0085] (13)

[0086] in, , , , The meaning is the same as above. It is the first Individual elements in categories The actual label on the page. The numerator in the formula is the category. The number of pixels, where the denominator is the total number of pixels, reflects the category. Frequency of occurrence.

[0087] Focal Loss is an improved version of the standard cross-entropy loss. It uses a dynamic scaling mechanism to effectively handle extreme imbalances between background and foreground pixels, enhances the learning of blurred boundary regions, and solves the problems of class imbalance and distinguishing between easy and difficult samples. The formula is as follows:

[0088] (14)

[0089] in, It is the model's predicted probability for the correct category. It is a custom hyperparameter for adjusting the focus on difficult samples.

[0090] In addition, UniMAC employs deep supervision. Specifically, at the output of each level of the decoder, a prediction branch is constructed in parallel with the final output layer. Each branch uses the modality-aware uncertainty loss function proposed in this paper and is compared with the true label for calculation. The final total loss function is defined as a weighted sum of the losses of each level, as shown in the following formula:

[0091] (15)

[0092] in, It is the number of layers in the 3D-Unet decoder. It is the first The modality-aware uncertainty loss between the prediction results of the layer decoder and the calculated true values.

[0093] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0094] The block diagrams of devices, apparatuses, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0095] It should also be noted that in the apparatus, device, and method of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of the present invention.

[0096] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0097] It should be understood that the qualifying terms "first", "second", "third", "fourth", "fifth" and "sixth" used in the description of the embodiments of the present invention are only used to more clearly illustrate the technical solutions and are not intended to limit the scope of protection of the present invention.

[0098] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for adaptive completion of brain tumor segmentation based on uncertain missing modalities, characterized in that, include: S1: Construct a UniMAC segmentation network with a single encoder and a single decoder. Input the four MRI modalities FLAIR, T1, T1c, and T2 into independent residual blocks for feature extraction to obtain high-level features of the four modalities. Then, concatenate the high-level features in the channel dimension to form concatenated multimodal features. S2: Apply a masking operation to the spliced ​​multimodal features to simulate mode loss. Through the mode adaptive completion MAC mechanism, combined with the learnable module and two masking TM operations, dynamically complete the feature information corresponding to the missing mode to obtain the completed multimodal features. S3: Input the completed multimodal features into the 3D-UNet segmentation network and output the brain tumor segmentation results; S4: Construct a modality-aware uncertainty loss, combine Dice Loss, weighted Softmax Loss, and Focal Loss, dynamically adjust the contribution of each loss term through learnable weights, and combine deep supervision to perform single-stage end-to-end training on the UniMAC segmentation network; S5: Input the MRI multimodal data with missing modalities into the trained UniMAC segmentation network to obtain the final brain tumor segmentation result.

2. The method according to claim 1, characterized in that, The modality adaptive completion (MAC) mechanism is as follows: a learnable module with the same size as the input feature is created, effective modal features are retained through two mask TM operations, and the missing modal positions are replaced with compensation features obtained by residual fusion of the learnable module and the existing modality.

3. The method according to claim 1, characterized in that, The formula for calculating the modality sensing uncertainty loss is as follows: ,in, It is the number of sub-loss functions participating in the joint optimization. It is the first The value of each sub-loss function, Is with the first The learnable parameters associated with each sub-loss function are initialized to 1.

4. The method according to claim 1, characterized in that, The two masking TM operations are as follows: the first mask retains the valid modal features, and the second mask replaces the inverted missing modal positions with learnable module features.

5. The method according to claim 1, characterized in that, The UniMAC segmentation network employs single-stage end-to-end training, eliminating the need for pre-training to generate the network.

Images