A multi-modal MRI brain tumor segmentation method guided by cross-modal super-resolution

Abstract

The application discloses a kind of multi-modal MRI brain tumor segmentation methods of cross-modal super-resolution guide, it is related to image segmentation technical field.The application is first by a multi-modal shared encoder to the four modal MRI of input is characterized learning, for T1 and T2 two low-resolution mode, independently designed super-resolution reconstruction branch is inputed with shared encoder output, using light-weight convolution-attention module is carried out global 2 times reconstruction, and 4 times fine enhancement is applied in candidate lesion area, super-reconstruction result is shared gradient with segmentation branch in optimization process, ensure that the high-frequency information generated meets segmentation semantics, segmentation branch uses decoder structure to predict tumor area, network is according to BraTS annotation standard and output three sub-regions: whole tumor (WT), tumor core (TC) and enhanced tumor (ET), introduce uncertainty estimator on the basis of segmentation output.This module is based on entropy estimation through shared encoder, ensure that super-resolution and segmentation task are consistent on bottom feature.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a cross-modal super-resolution guided multimodal MRI brain tumor segmentation method. Background Technology

[0002] In recent years, diffusion models have shown great application potential in the field of medical image processing. A review by Kazerouni et al. systematically elucidated the broad application prospects of such models in medical image analysis. Building on this, Fei et al. proposed a general medical image enhancement diffusion model to address quality degradation issues in different imaging modalities, aiming to improve medical image quality. Considering the limitations of computational resources in clinical settings, Abdusalomov et al. explored the feasibility of lightweight super-resolution techniques in medical imaging, attempting to find a balance between image quality and computational overhead. For optimization of specific imaging modalities, Li et al. proposed a super-resolution technique suitable for cyclic medical images, further expanding the application of super-resolution in special anatomical structures. The clinical value of these techniques has been gradually validated: Rudie et al. demonstrated the effectiveness of deep learning-based super-resolution in three-dimensional brain MRI through clinical evaluation, while Terzis et al.'s research showed that AI-integrated super-resolution reconstruction can significantly shorten the scanning time of knee MRI to less than five minutes. Despite the significant advantages of deep learning and AI in reducing MRI scan time, Foti et al. also pointed out that it still faces many challenges and potential pitfalls in clinical practice. Therefore, based on existing research, further exploration of medical image super-resolution methods that balance efficiency and robustness has important clinical translational significance.

[0003] In addition, some scholars have explored the joint optimization of super-resolution and segmentation. For example, some scholars have proposed an end-to-end joint framework such as JISS, which can adapt to multiple magnifications with a single model and use the segmentation task signal to inversely constrain the SR module; Zhou et al. proposed a deep learning-based super-resolution method for brain tumor MRI images, which improves image resolution through deep network structure and provides a clearer data foundation for subsequent segmentation tasks; Gao et al. developed the JDSR-GAN joint learning network, which simultaneously achieves super-resolution and structural restoration for facial images under occlusion, and its multi-task learning mechanism has reference value for medical image processing; Wu et al. proposed MMSRNet, which uses a multi-task and multi-scale learning strategy for super-resolution reconstruction of pathological images, and enhances image details and diagnostic relevance through shared representation learning; Qiu et al. rethought the application of dual-stream super-resolution semantic learning methods in medical image segmentation, and improved the accuracy of segmentation boundaries through semantically guided super-resolution optimization.

[0004] Nevertheless, most existing methods still perform full-image super-resolution first, then feed the results into a separate segmentation model, lacking specific optimization for tumor regions. This fragmented approach often leads to blurred lesion boundaries and insufficient detail recovery. The main challenges are:

[0005] Challenge 1: The ability to fuse modal information is limited, often only reaching the level of simple splicing or shallow interaction;

[0006] The second challenge is that the robustness is insufficient and the segmentation results are not stable enough in the face of common clinical modality loss.

[0007] In view of this, this invention proposes a cross-modal super-resolution guided uncertainty-aware segmentation model for multimodal MRI brain tumor segmentation. Summary of the Invention

[0008] The purpose of this invention is to solve the above-mentioned problems by providing a cross-modal super-resolution guided multimodal MRI brain tumor segmentation method.

[0009] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0010] Step 1: Multimodal Feature Extraction and Adaptive Super-Resolution Reconstruction

[0011] Multimodal medical images are acquired, and features are extracted using a shared encoder to obtain a unified high-level feature representation.

[0012] Based on the high-level feature representation, adaptive super-resolution reconstruction is performed on key modalities, and computing resources are adaptively allocated according to the lesion probability. High-magnification fine reconstruction is performed on suspected lesion areas, and low-magnification fast reconstruction is performed on background areas.

[0013] Step 2: Cross-modal attention fusion

[0014] Multimodal features are divided into complementary modal groups, and information exchange between groups is realized through a cross-modal attention interaction module. One set of modal features is used to query another set of modal features, explicitly modeling the complementary relationship between modalities, and dynamically adjusting the interaction intensity by combining uncertainty weights.

[0015] Step 3: Uncertainty-aware segmentation prediction

[0016] An uncertainty estimation mechanism is introduced based on a dual-branch structure. The reliability of each branch is quantified by prediction entropy and prediction variance. Uncertainty is mapped to pixel-level fusion weights. High uncertainty regions are automatically suppressed, while low uncertainty regions are enhanced and preserved. Weighted fusion generates segmentation prediction results.

[0017] Step 4: Multi-task joint optimization

[0018] The network parameters are optimized end-to-end using a joint loss function, which includes: segmentation loss, super-resolution reconstruction loss, auxiliary supervision loss, and uncertainty regularization term.

[0019] Furthermore, the multimodal medical image mentioned in step one is an MRI brain tumor image, including four modalities: T1, T1c, T2, and FLAIR; the shared encoder is based on a hybrid structure of convolution and attention, capturing complementary features between modalities in a unified latent space.

[0020] Furthermore, the adaptive super-resolution reconstruction described in step one is performed only for the two low-resolution modalities T1 and T2, while preserving the original resolution of T1c and FLAIR. A region-of-interest gating strategy is adopted to first generate a global 2x super-resolution result, and then perform an additional 4x upsampling on the suspected tumor region, followed by weighted fusion through an adaptive fusion network.

[0021] Furthermore, the super-resolution ratio mentioned in step one is adaptively adjusted according to the mean tumor probability of the image patch: when the mean tumor probability is greater than a preset threshold, a 4x high super-resolution is used, and when it is less than or equal to the preset threshold, a 2x low super-resolution is used; local super-resolution reconstruction is performed using the SwinIR network based on Swin Transformer, and a dual-branch structure is adopted to balance global consistency and local saliency.

[0022] Furthermore, in step two, the modal groups are divided into two groups, T1 / T1c and T2 / FLAIR, and the mid-level semantic features are extracted by the modal group encoder respectively. The cross-modal attention interaction uses the features of the T1 / T1c group as the query and the features of the T2 / FLAIR group as the key and value. After calculating the attention weight, the values ​​are weighted and summed to obtain the cross-modal interaction features.

[0023] Furthermore, the uncertainty weight matrix mentioned in step two is generated by the uncertainty estimator and is used to enhance cross-modal feature interaction in high-confidence regions, suppress interference in low-confidence regions, upsample the interaction results back to the original resolution, and add them to the input residual.

[0024] Furthermore, the bi-branch uncertainty estimation in step three includes: measuring the branch prediction uncertainty by the entropy of the prediction distribution, with a higher weight assigned to lower entropy; introducing variance-based uncertainty estimation as a supplement, calculating the prediction variance through multiple inferences, with a lower weight assigned to higher branch uncertainty; and weighting and fusing the probabilities of each branch after normalizing the weights.

[0025] Furthermore, in step three, the segmentation prediction outputs three sub-regions—the whole tumor, the tumor core, and the enhanced tumor—according to the BraTS annotation standard, and adaptively assigns different modal weights through a cross-modal attention mechanism.

[0026] Furthermore, the joint loss function described in step four is specifically: segmentation loss combined with cross-entropy loss and Dice loss; super-resolution reconstruction loss combined with tumor region weighted L1 loss, ordinary L1 loss and structural similarity loss; auxiliary supervision loss is the average of the binary cross-entropy losses of the auxiliary branches of the two modality groups; uncertainty regularization term constrains the mean of the prediction entropy of each branch.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] (1) The proposed tumor-aware dynamic super-resolution module predicts tumor probability maps through a lightweight gating network and adaptively assigns different super-resolution ratios to different regions of the image: high-magnification (4×) fine reconstruction is performed in suspected lesion regions to enhance details, while low-magnification (2×) global upsampling is performed in background regions to maintain overall consistency. This "global + local" adaptive strategy significantly improves the representation ability of small-scale tumor structures and blurred boundaries under limited computational resources, providing high-quality input for subsequent accurate segmentation.

[0029] (2) The designed extended cross-modal attention mechanism divides the T1 / T1ce and T2 / FLAIR modalities into two groups based on pathological sensitivity, and constructs an attention interaction with one group as the query and the other group as the key. This mechanism explicitly models the complementary relationship between modalities (such as structural and edema information), realizes dynamic information exchange and collaboration at the feature level, effectively improves the consistency of lesion context and the accuracy of boundary characterization, and avoids the parameter expansion caused by setting an encoder for each modality separately.

[0030] (3) The introduced dual-branch uncertainty calibration weighting module calculates pixel-level uncertainty based on the entropy or variance of each branch prediction and generates dynamic weights accordingly. This module can automatically reduce the modal or feature contribution of high uncertainty regions (such as noisy regions or regions with missing modalities) and enhance the weight of reliable information. This uncertainty-driven adaptive fusion mechanism significantly improves the model's segmentation robustness and the reliability of output results in complex scenarios such as missing modalities and noise interference. Attached Figure Description

[0031] Figure 1 This is an overall block diagram of the method of the present invention;

[0032] Figure 2 This is the tumor sensing dynamic super-resolution module of the present invention;

[0033] Figure 3 This invention relates to cross-modal attention fusion;

[0034] Figure 4 This diagram illustrates the segmentation results of the mainstream method and our method on three different ultrasound image datasets. Detailed Implementation

[0035] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0036] To address the aforementioned technical problems, this invention provides a multimodal MRI brain tumor segmentation method guided by cross-modal super-resolution.

[0037] The proposed cross-modal super-resolution guided uncertainty-aware segmentation model first learns the representation of the input four-modal MRI (T1, T1c, T2, FLAIR) data through a multimodal shared encoder. This encoder, based on a hybrid convolutional and attention structure, captures complementary features between modalities in a unified latent space and provides a consistent high-level representation for downstream tasks. Independent super-resolution reconstruction branches are designed for the two low-resolution modalities, T1 and T2. Standard magnifications of 2x (×2) and 4x (×4) are used, where "×k" represents k-fold upsampling relative to the low-resolution region. This branch takes the shared encoder output as input, performs global 2× reconstruction using a lightweight convolutional-attention module, and applies 4× fine enhancement to candidate lesion regions. This improves the overall grayscale and texture readability while achieving fine-grained restoration at lesion boundaries and small voxel regions. The super-resolution reconstruction results share gradients with the segmentation branch during optimization, ensuring that the generated high-frequency information conforms to segmentation semantics. Based on the shared encoder output, the segmentation branch uses a decoder structure to predict the tumor region. The network outputs three sub-regions according to the BraTS annotation standard: whole tumor (WT), tumor core (TC), and enhanced tumor (ET). This branch adaptively allocates weights for different modalities through a cross-modal attention mechanism, thereby improving the sensitivity of T1c to the enhanced region. To quantify the reliability of the model in the prediction process, we introduce an uncertainty estimator based on the segmentation output. This module, based on entropy estimation and a shared encoder, ensures consistency between the super-resolution and segmentation tasks in terms of low-level features.

[0038] like Figure 1 As shown, the cross-modal super-resolution guided uncertainty-aware segmentation model framework consists of a shared encoder, a task-specific branch, and an uncertainty estimator.

[0039] Figure 1The overall block diagram of the method of the present invention consists of three core parts: (1) a dynamic super-resolution module for tumor perception, in which the input image is used to predict the probability of tumors through a lightweight network and to adaptively select the super-resolution ratio of different regions; (2) an adaptive modal fusion module based on attention mechanism, in which cross-modal attention fusion can realize cross-modal dynamic interaction and feature coordination in the encoding process; and (3) a two-branch uncertainty estimator based on entropy estimation and confidence weighted fusion.

[0040] The overall method and steps are as follows:

[0041] Step 1: Super-resolution reconstruction

[0042] This invention designs a local adaptive super-resolution module and coordinates it with a segmentation network for optimization. The module first dynamically locates the region of interest (ROI) based on the tumor probability map predicted by the segmentation mask. Then, it performs high-magnification super-resolution within the ROI to enhance details, while performing low-magnification super-resolution outside the ROI to save computational resources. Finally, an adaptive fusion network seamlessly integrates the global and local super-resolution results. This design significantly improves the image quality of key regions, providing higher-quality multimodal input for subsequent accurate segmentation, while avoiding the huge computational overhead of global super-resolution.

[0043] This invention performs local super-resolution reconstruction only on the T1 and T2 modalities, while preserving the original resolution of T1c and FLAIR. This is because T1 and T2 provide rich structural and tissue details, but their resolution limits lead to blurred boundaries and difficulty in identifying small lesions. Local super-resolution effectively enhances these spatial details, thereby improving the segmentability of the tumor region. Conversely, T1c primarily relies on the intensity contrast generated by contrast agents to locate enhanced tumors, and its diagnostic value does not lie in spatial resolution; FLAIR is highly sensitive to edematous areas, has high inherent resolution, and requires less SR (spatial resolution). Forcibly performing super-resolution on these two modalities may introduce artifacts or alter the intensity distribution, thus affecting the model's robustness. Therefore, this invention selectively performs local super-resolution reconstruction on T1 and T2 to balance performance improvement with cross-modal consistency.

[0044] To improve the distinguishability of lesions and boundaries while maintaining overall efficiency, a gating strategy based on box-level ROI is adopted. First, a global ×2 super-resolution result is generated. Then, the ×4 super-resolution details of the tumor branches are downsampled to ×2 and adaptively weighted with the global ×2 result through a lightweight fusion network. The adaptive weighting formula for the fusion weights fusion_w∈[0,1] is as follows:

[0045] (1)

[0046] In the formula, This represents the clipping block within the ROI frame representing the global ×2 super-resolution result, sr tumor The ROI local block is obtained by downsampling the ×4 reconstruction result of the tumor branch to ×2. φ is the fusion weight generated by the lightweight fusion network, with a value range of [0,1], which is used to determine the proportion of global and local super-resolution results.

[0047] The ROI gating fusion formula is as follows:

[0048] (2)

[0049] Where M is a binary gated mask for the ROI, aligned with the target on a ×2 scale, with ROI regions having a value of 1 and non-ROI regions having a value of 0. The result is a local block downsampled from the ×4 result to ×2. ⊙ indicates pixel-by-pixel multiplication. Outside the bounding box, the global ×2 result is maintained. This fusion mechanism ensures that the area outside the ROI bounding box maintains the global ×2 super-resolution result, avoiding the introduction of additional artifacts.

[0050] To balance overall speed and lesion detail, the super-resolution ratio is adaptively adjusted based on the tumor probability S(p) of the image patch. The specific strategy is as follows:

[0051] (3)

[0052] In the formula, P(p) represents the mean tumor probability of the current image patch p, τ is a preset threshold, and S high S is a high super-resolution multiplier of ×4. low The super-resolution is 2×2. This mechanism ensures that the model prioritizes the allocation of limited computational resources to suspected tumor regions.

[0053] For each image patch, local super-resolution reconstruction is performed using the SwinIR network based on its corresponding super-resolution ratio S(p), where I LR It is the input low-resolution image patch, I SR This is the reconstructed high-resolution output. SwinIR is an image super-resolution reconstruction network based on SwinTransformer:

[0054] (4)

[0055] The model adopts a two-branch structure, such as Figure 2 As shown, this balances global consistency with local saliency:

[0056] (1) Global super-resolution branch: The overall consistency of the image is maintained by 2x upsampling and residual encoder structure, and the feature representation is enhanced by embedding Residual Swin Block.

[0057] (2) Local lesion perception branch: Perform 4x upsampling on suspected tumor areas, and combine multiple RSTB (Residual Swing Transformer Block) and REU (Residual Enhancement Unit) to enhance local detail reconstruction.

[0058] The outputs of the two branches are integrated by an adaptive fusion module, which significantly improves the visual quality and separability of the lesion area while maintaining the smoothness of the global image structure.

[0059] Step 2: Cross-modal attention fusion

[0060] In multimodal MRI, a single modality often fails to fully characterize the lesion region. However, pairwise combinations based on pathological features can improve lesion information. For example, T1 / T1c is better suited to providing structural and enhancement information, while T2 / FLAIR is more sensitive to lesion extent and edema. To fully utilize the complementarity between modalities, we propose a cross-modal feature interaction mechanism. As shown in the figure, the four MRI modalities (T1ce, T1, T2, FLAIR) are first divided into two modal groups, and then processed through shallow feature extraction modules such as convolution, batch normalization, and activation functions to obtain intra-group feature representations and uncertainty estimates.

[0061] First, the four modalities are divided into two modality groups: T1 / T1c and T2 / FLAIR. Mid-level semantic features are extracted from each modality group using the ModalGroupEncoder, and auxiliary segmentation responses are predicted, thus forming a compact and robust representation within the group.

[0062] To achieve effective information exchange between different modalities, we designed a cross-modal attention interaction module. This module uses one set of modal features as the query and another set of modal features as the key and value, explicitly modeling inter-group correlations through an attention mechanism. Specifically, the input features are first normalized and downsampled to reduce spatial dimensionality and computational overhead; then, Q, K, and V representations are generated through convolutional mapping, where Q is the query matrix, obtained from the T1 / T1c modal group feature mapping, used to "ask" for information from the other set; K is the key matrix, obtained from the T2 / FLAIR modal group feature mapping, used to match similarity with Q; and V is the value matrix, obtained from the T2 / FLAIR modal group feature mapping, carrying the information to be fused. k The scaling factor, representing the dimension of the K matrix, is used to prevent excessively large inner products and stabilize gradients. The attention matrix is ​​obtained by calculating the similarity between Q and K, and then weighted and summed over V to obtain cross-modal interaction features. Subsequently, attention weights are calculated and dynamically adjusted based on the uncertainty weight matrix W(U).

[0063] (5)

[0064] (6)

[0065] (7)

[0066] (8)

[0067] Among them, F (T1,T1c) It is a feature map from two modal groups, T1 and T1ce (as the query side); F (T2,FLAIR) These are feature maps from the T2 / FLAIR group (as key / value sides); Attention* is the attention weight matrix representing the degree of attention T1 / T1c pays to each position of T2 / FLAIR; W(U) is the uncertainty weight matrix, generated by the uncertainty estimator, used to dynamically adjust the contribution of the interaction results.

[0068] Through this mechanism, the model can enhance cross-modal feature interactions in high-confidence regions and suppress interference in low-confidence regions, thereby improving overall robustness and segmentation accuracy. Finally, the interaction results are upsampled back to the original resolution and added to the input residuals to achieve information enhancement. This interaction method allows different modal groups to "question" and "complement" each other at the feature level: for example, the structural and enhancement information of T1 / T1c can guide the edema and lesion characterization of T2 / FLAIR, and vice versa. Through cross-modal attention, the model can capture the complementary relationships between modalities, alleviating the problem of insufficient information from a single modality, thereby improving the contextual consistency and boundary characterization accuracy of lesion regions.

[0069] Step 3: Two-branch uncertainty estimation

[0070] After obtaining two sets of cross-modal features in the cross-modal attention interaction module, we introduce a dynamic fusion module. We transform the uncertainty map generated by the model into a confidence map (the higher the uncertainty, the lower the confidence) through a monotonically decreasing reliability mapping, and use this confidence as a pixel-level weight to adaptively weight and fuse the two sets of modal features. This retains more effective information in high-confidence regions and automatically suppresses interference in low-confidence regions. In this way, smaller weights are automatically assigned to low-quality or high-noise modal regions, reducing their negative contribution to the final reconstruction from the source and improving the model's segmentation robustness in complex scenes.

[0071] We introduce an uncertainty-aware fusion mechanism into the dual-branch structure. Each branch m gives a pixel-level probability P. m (c|x) is the probability of category c at pixel x. The entropy of the prediction distribution is defined as follows.

[0072] (9)

[0073] In the formula, the entropy value H m (x) is used to measure the uncertainty of branch predictions: the higher the entropy, the more uncertain the prediction. To map entropy to non-negative weights (lower entropy, higher weight), the following transformation is used:

[0074] (10)

[0075] (11)

[0076] In the formula, H max The maximum entropy (corresponding to a perfectly uniform distribution) is represented by α, which is a temperature parameter (typically 1–3) used to adjust the sensitivity of the weights to entropy changes. m (x) represents the unnormalized weights; the lower the entropy, the higher the weight. The final weight w m (x) is normalized to ensure that the sum of the weights of each branch is 1.

[0077] We also introduce variance-based uncertainty estimation as a supplement. The predicted variance is calculated through multiple inferences using Monte Carlo Dropout, or the confidence level of the network output is directly used to proxy U. m That is, the uncertainty estimation of the branch is used to construct the weights, using ε as a small constant (10). -6 ~10 -4 To prevent division by zero / numerical instability:

[0078] (12)

[0079] Ultimately, the output probabilities p of different branches m (x) is fused using uncertain weights to generate the final fusion probability P(x) of the pixels:

[0080] (13)

[0081] This improves the robustness and accuracy of segmentation results in situations with blurred boundaries, small targets, or missing modalities.

[0082] Step 4: Use a loss function to control segmentation error

[0083] This invention employs a design based on segmentation loss. Super-resolution reconstruction loss , auxiliary monitoring of losses With uncertainty regularization The joint loss function is used to control and measure the segmentation error of the model. Segmentation loss Combined with cross-entropy loss L CE With multiple types of Dice loss LDice To balance pixel-level classification accuracy and region overlap consistency, its formula is expressed as follows:

[0084] (14)

[0085] (15)

[0086] (16)

[0087] Super-resolution reconstruction loss The tumor region-weighted L1 loss L is calculated by summing three joint loss functions. tumor l1 Ordinary L1 loss L l1 Compared with Structural Similarity (SSIM) loss L SSIM In the formula below, p i g is the foreground probability of pixel i. i λ1 and λ2 are the true values, ε is the smoothing constant to prevent the denominator from being zero when calculating the Dice loss, and λ1 and λ2 are the weights with a default value of 1. The loss terms are defined as follows:

[0088] (17)

[0089] (18)

[0090] (19)

[0091] (20)

[0092] Among them, I SR (i) is the super-resolution reconstructed image output by the network, I HR (i) is the corresponding high-resolution ground truth reference image. The total number of pixels is used to calculate the average loss. The weight ω for the tumor region is... i Lesion areas are assigned a higher weight (1.5 times), while background areas are assigned a weight of 0.5, to highlight the recovery of key structures. α1, α2, and α3 are the weighting coefficients for each loss, with a default setting of 0.4:0.4:0.2.

[0093] An auxiliary supervision branch is introduced after each modality grouping (T1 / T1c and T2 / FLAIR), outputting a single-channel tumor prediction map, and calculating a binary cross-entropy loss with the ground truth mask. The average of the losses from the two auxiliary branches is used as the auxiliary supervision term. :

[0094] (twenty one)

[0095] in and It is the binary cross-entropy loss corresponding to the mode group.

[0096] Furthermore, an uncertainty regularization term is introduced to constrain the mean of the prediction entropy of each branch, preventing the model from escaping learning by increasing uncertainty. This regularization term is defined as:

[0097] (twenty two)

[0098] In the formula, M represents the number of branches participating in the regularization, which can be taken as the spatial mean based on the foreground / full view; H m (x) is the prediction entropy of the branch at the pixel, defined in formula (9); E is the expectation operator, which means taking the average value of the random variable, and here it is the average of the entropy values.

[0099] The final total loss function is the weighted sum of the four losses mentioned above:

[0100] (twenty three)

[0101] In the formula, β1, β2, and β3 are the weighting coefficients of each loss.

[0102] Comparing experimental data and evaluation results

[0103] The experiment was based on the BraTS 2020 (Brain Tumor Segmentation Challenge 2020) and BraTS2021 (Brain Tumor Segmentation Challenge 2021) public datasets, which contain 369 and 1251 multimodal MRI images (T1, T1c, T2, FLAIR) that have been registered to the same space, respectively, along with corresponding expert-annotated true segmentation labels.

[0104] The BraTS (Brain Tumor Segmentation) dataset is a dataset specifically designed for brain tumor segmentation research. It contains multimodal MRI images, including T1, T1c (contrast-enhanced T1), T2, and FLAIR sequences, along with corresponding tumor segmentation labels. Based on this dataset, the performance of different brain tumor segmentation model algorithms can be evaluated and compared. Different MRI modalities can reflect the different pathological features of the tumor and its surrounding tissues.

[0105] (1) T1-weighted imaging (T1): provides details of anatomical structures, but has limited contrast between tumors and normal tissues;

[0106] (2) T1 contrast enhancement (T1c): After injection of contrast agent, the areas where the blood-brain barrier is disrupted and the permeability of new blood vessels is increased will be significantly enhanced, corresponding to the pathological active tumor area (Enhancing Tumor, ET);

[0107] (3) T2-weighted imaging (T2): It is sensitive to the movement of water molecules and can clearly show the edema area (Edema, ED) around the tumor. Pathologically, it corresponds to the tumor's infiltration of surrounding tissues and glial reaction.

[0108] (4) Fluid attenuated inversion recovery sequence (FLAIR): better visualization of diffuse edema and non-enhancing core (NCR / NET) corresponding to necrotic tissue and low-activity tumor components while suppressing cerebrospinal fluid signals.

[0109] To evaluate the performance of the method in this invention, several commonly used image segmentation evaluation metrics were employed. The method was compared with nn-UNet, AttentionUNet, TransBTS, SwinUNet, Switch-UMamba, HOPE-Seg, EoFormer, and GSS on the BraTS dataset.

[0110] Table 1: Quantitative comparison of the method of the present invention with existing medical image segmentation methods. It means the higher the better. The lower the better.

[0111] As shown in Table 1, the comparative methods selected in the experiments of this invention are diverse, covering a variety of mainstream technical routes such as classic CNN, CNN+attention mechanism, CNN-Transformer hybrid model, pure Transformer, boundary modeling, modality fusion, state space-based state segmentation model, and system integration methods. The method of this invention significantly outperforms the aforementioned comparative methods on BraTS2020 and BraTS2021, leading in all metrics (IoU=86.84%, Dice=90.92%, Precision=94.13%, Recall=88.16%, PSNR=30.46 dB, Accuracy=96.55%, HD95=3.78 mm). Furthermore, this invention focuses on WT (clinically closer to "overall lesion burden"), and can be expanded to ET / TC sub-segments in the future. In particular, the PSNR is improved by about 2.1 dB compared to GSS, indicating that the "global + ROI super-resolution" effectively restores the details and textures, and the cross-modal attention mechanism significantly improves the consistency of image structure. This shows that the "global-local super-resolution combination + cross-modal attention fusion + uncertainty perception" framework proposed in this invention can effectively improve the detail restoration of medical images and the segmentation accuracy of tumor regions.

[0112] Figure 4 The results of this invention's model are compared with other image segmentation models. From top to bottom: sub-images (a)-(d) represent the BraTS20 public dataset; sub-images (e)-(h) represent the BraTS21 public dataset. To fully verify the effectiveness of the method of this invention, we selected several representative medical image segmentation methods for comparative experiments. First, NN-UNet, as a recognized strong baseline, is widely used in brain tumor segmentation tasks such as BraTS due to its automated configuration and high robustness; Attention U-Net introduces attention gating in the U-Net skip connections and is a typical representative of CNN+attention mechanism. NN-UNet and Attention U-Net, as classic CNN baselines, have limited ability to characterize small lesions and boundary structures. TransBTS captures local and global dependencies by combining CNN and Transformer, while SwinUNet constructs a hierarchical attention structure based on Swin Transformer. They are representative methods of hybrid architecture and pure Transformer architecture, respectively. TransBTS and SwinUNet have certain improvements over CNN methods, verifying the value of long-range dependencies in modeling the overall tumor structure, but the improvement is limited.

[0113] The method of this invention focuses on segmenting the entire tumor region. Baseline methods such as NN-UNet and GSS generally exhibit varying degrees of blurring, discontinuity, or undersegmentation at the tumor boundary. In contrast, the MARSeg method proposed in this invention produces segmented regions with the highest degree of agreement with the ground truth, exhibiting sharper and more complete boundaries. It also provides the clearest recovery of small tumor structures and complex boundaries, effectively reducing visual errors from oversegmentation and undersegmentation. This directly verifies the advantages of the proposed dynamic super-resolution and cross-modal attention mechanism in improving segmentation precision.

[0114] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0115] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multimodal medical image segmentation method guided by cross-modal super-resolution, characterized in that, Includes the following steps: Step 1: Multimodal Feature Extraction and Adaptive Super-Resolution Reconstruction Multimodal medical images are acquired, and features are extracted using a shared encoder to obtain a unified high-level feature representation. Based on the high-level feature representation, adaptive super-resolution reconstruction is performed on key modalities, and computing resources are adaptively allocated according to the lesion probability. High-magnification fine reconstruction is performed on suspected lesion areas, and low-magnification fast reconstruction is performed on background areas. Step 2: Cross-modal attention fusion Multimodal features are divided into complementary modal groups, and information exchange between groups is realized through a cross-modal attention interaction module. One set of modal features is used to query another set of modal features, explicitly modeling the complementary relationship between modalities, and dynamically adjusting the interaction intensity by combining uncertainty weights. Step 3: Uncertainty-aware segmentation prediction An uncertainty estimation mechanism is introduced based on a dual-branch structure. The reliability of each branch is quantified by prediction entropy and prediction variance. Uncertainty is mapped to pixel-level fusion weights. High uncertainty regions are automatically suppressed, while low uncertainty regions are enhanced and preserved. Weighted fusion generates segmentation prediction results. Step 4: Multi-task joint optimization The network parameters are optimized end-to-end using a joint loss function, which includes: segmentation loss, super-resolution reconstruction loss, auxiliary supervision loss, and uncertainty regularization term.

2. The multimodal medical image segmentation method guided by cross-modal super-resolution according to claim 1, characterized in that, The multimodal medical images mentioned in step one are MRI brain tumor images, including four modalities: T1, T1c, T2, and FLAIR. The shared encoder is based on a hybrid structure of convolution and attention, which captures complementary features between modalities in a unified latent space.

3. The multimodal medical image segmentation method guided by cross-modal super-resolution according to claim 1, characterized in that, The adaptive super-resolution reconstruction described in step one is performed only on the two low-resolution modalities T1 and T2, while preserving the original resolution of T1c and FLAIR. A region-of-interest gating strategy is adopted, first generating a global 2x super-resolution result, then performing an additional 4x upsampling on the suspected tumor region, and finally weighted fusion through an adaptive fusion network.

4. The multimodal medical image segmentation method guided by cross-modal super-resolution according to claim 1, characterized in that, The super-resolution ratio mentioned in step one is adaptively adjusted according to the mean tumor probability of the image patch: when the mean tumor probability is greater than the preset threshold, a high super-resolution of 4 times is used, and when it is less than or equal to the preset threshold, a low super-resolution of 2 times is used; local super-resolution reconstruction is performed using the SwinIR network based on SwinTransformer, and a dual-branch structure is adopted to take into account both global consistency and local saliency.

5. The multimodal medical image segmentation method guided by cross-modal super-resolution according to claim 1, characterized in that, In step two, the modal groups are divided into two groups: T1 / T1c and T2 / FLAIR. The mid-level semantic features are extracted by the modal group encoder. The cross-modal attention interaction uses the features of the T1 / T1c group as the query and the features of the T2 / FLAIR group as the key and value. After calculating the attention weight, the values ​​are weighted and summed to obtain the cross-modal interaction features.

6. The multimodal medical image segmentation method guided by cross-modal super-resolution according to claim 1, characterized in that, The uncertainty weight matrix mentioned in step two is generated by the uncertainty estimator and is used to enhance cross-modal feature interaction in high-confidence regions, suppress interference in low-confidence regions, upsample the interaction results back to the original resolution, and add them to the input residual.

7. The multimodal medical image segmentation method guided by cross-modal super-resolution according to claim 1, characterized in that, The bi-branch uncertainty estimation described in step three includes: measuring the branch prediction uncertainty by the entropy of the prediction distribution, with a higher weight assigned to lower entropy; introducing variance-based uncertainty estimation as a supplement, calculating the prediction variance through multiple inferences, with a lower weight assigned to higher branch uncertainty; and weighting and fusing the probabilities of each branch after normalizing the weights.

8. The multimodal medical image segmentation method guided by cross-modal super-resolution according to claim 7, characterized in that, In step three, the segmentation prediction outputs three sub-regions according to the BraTS annotation standard: the whole tumor, the tumor core, and the enhanced tumor. Different modal weights are adaptively assigned through a cross-modal attention mechanism.

9. The multimodal medical image segmentation method guided by cross-modal super-resolution according to claim 1, characterized in that, The joint loss function described in step four is as follows: segmentation loss combined with cross-entropy loss and Dice loss; super-resolution reconstruction loss combined with tumor region weighted L1 loss, ordinary L1 loss and structural similarity loss; auxiliary supervision loss is the average of the binary cross-entropy losses of the auxiliary branches of the two modality groups; uncertainty regularization term constrains the mean of the prediction entropy of each branch.

Images