A medical image segmentation method, system and device

Abstract

The application discloses a medical image segmentation method, system and device, and relates to the technical field of image segmentation. The method comprises the following steps: generating a missing enhanced modality image based on multi-modality medical image data; performing weighted fusion on the missing enhanced modality image and the multi-modality medical image data based on regional saliency to generate a multi-modality fusion image; extracting local features of different receptive fields of the multi-modality fusion image to generate queries, keys and values; introducing a scale temperature in each attention head to obtain temperature-adjusted attention weights; generating enhanced channel perception features according to the temperature-adjusted attention weights and the values; performing average and maximum pooling on channel dimensions to construct a two-dimensional spatial attention graph; combining the enhanced channel perception features with the two-dimensional spatial attention graph to generate a feature map with strong discriminability and structural perception capability; and gradually upsampling the feature map to generate a segmentation map. The method significantly improves segmentation accuracy and cross-modality generalization capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, in particular to a medical image segmentation method, system and device. BACKGROUND

[0002] Medical image segmentation aims to accurately separate the target organ or tissue structure from the background in medical images. However, the current segmentation performance is still restricted by various factors, especially the problems of low-quality images, missing modalities and blurred boundaries, which will directly affect the model's learning of details and semantic understanding. In addition, single-modality images are difficult to fully reflect the structural characteristics of organ tissues, making the model prone to misclassification or omission when segmenting small-volume organs or adjacent boundary regions.

[0003] Some scholars have proposed a hybrid architecture that combines CNN and Transformer, which takes advantage of the high efficiency of CNN in local feature extraction while introducing the global modeling capability of Transformer, thereby improving the accuracy and robustness of medical image segmentation. Some scholars have proposed a medical image segmentation framework that combines CNN and Transformer. This method uses CNN as an encoder to extract fine-grained local features and introduces Transformer as an intermediate module to model long-range dependencies through self-attention mechanisms, thereby improving overall segmentation performance. On this basis, some scholars have further improved the application of Transformer in visual tasks by introducing a hierarchical window attention mechanism combined with multi-scale features extracted by CNN, which significantly reduces computational overhead while maintaining global perception capabilities, thereby better adapting to the processing needs of high-resolution medical images. Although the above methods that combine CNN and Transformer have made significant progress in medical image segmentation, in actual clinical scenarios, automated segmentation models still face significant challenges. Medical image data has high variability, influenced by factors such as patient individual differences, lesion shape changes, and imaging modality differences, leading to insufficient accuracy of traditional automatic segmentation methods in dealing with complex boundaries, missing modalities, or structural overlaps. Therefore, more and more research has begun to try to introduce interactive mechanisms into the segmentation process to improve the flexibility and controllability of the model. Some scholars have proposed a general-purpose interactive image segmentation framework that guides the segmentation process based on user-provided hints (such as clicks, box selections, or language descriptions) and leverages large-scale pre-training and Transformer architecture to exhibit strong target perception and transfer generalization capabilities. In multiple visual tasks, SAM can achieve fast and high-precision interactive segmentation, especially suitable for application scenarios with high boundary control requirements. However, SAM still has limitations in medical image segmentation, as it heavily relies on user hints and cannot meet the needs of large-scale automated processing or unsupervised environments. In clinical practice, frequent human-computer interaction not only increases operational costs but also may affect segmentation efficiency and annotation consistency. To address this issue, some scholars have proposed an improved architecture that combines automatic and interactive mechanisms, AI-SAM, which generates initial prompt points automatically without user intervention through the introduction of AI-Prompter, and then allows users to further refine, supporting flexible switching from complete automation to semi-interactive. The introduction of AI-SAM significantly improves the practicality of SAM in medical image segmentation tasks, balancing efficiency and robustness and adapting to a wider range of clinical application scenarios.

[0004] Although current research has made significant progress in fusing CNN and Transformer architecture and combining automated and interactive segmentation methods, there are still several key challenges in the actual medical image segmentation scene. Existing interactive mechanisms generally rely on users to actively provide prompt information such as clicks, box selections, etc. to guide the model to perform fine segmentation. However, in practical applications, users often have difficulty providing optimal prompts or need additional human intervention to correct model results, which not only limits the degree of automation of the segmentation process, but also affects the efficiency and consistency of the model in a clinical environment. In addition, most current segmentation models have not fully tapped the potential semantic value of prompt information in their structural design, especially in the collaborative modeling process of the encoder and the decoder, there is a lack of deep utilization of prompt guidance and feature fusion, resulting in interactive clues failing to be truly converted into effective information that can enhance the model's understanding and reconstruction capabilities, thereby affecting the further improvement of segmentation performance. SUMMARY

[0005] In view of the deficiency that the prior art lacks deep utilization of prompt guidance and feature fusion in the collaborative modeling process of the encoder and the decoder, resulting in interactive clues failing to be truly converted into effective information that can enhance the model's understanding and reconstruction capabilities, thereby affecting the segmentation performance, the present application proposes a medical image segmentation method, system and device, which designs an automatic interactive multi-modal super-resolution guided segmentation model. The model introduces an automatic interactive mechanism, which can adaptively adjust the feature extraction and fusion strategy without human intervention, thereby improving the segmentation robustness and generalization ability under different data distribution and structural variation conditions, thereby solving the problems existing in the prior art.

[0006] A medical image segmentation method, comprising the following steps:

[0007] Obtaining multi-modal medical image data to be segmented;

[0008] Generating a missing enhanced modality image based on the multi-modal medical image data; and weighting and fusing the missing enhanced modality image and the multi-modal medical image data based on regional saliency to generate a multi-modal fusion image;

[0009] The local features of different receptive fields of the multi-modal fusion image are extracted, the spatial offset of the local features is predicted, the spatial sampling of the local features of different receptive fields is performed combined with the predicted spatial offset, and a sampling position is obtained; query, key and value features are extracted from a feature map corresponding to the sampling position, the temperature adjusted attention weight is calculated by calculating the attention weight of the query and the key and introducing the scale temperature; the enhanced channel perception feature is generated according to the temperature adjusted attention weight and the value feature; the two-dimensional spatial attention graph is constructed by performing average and maximum pooling on the channel dimension; the enhanced channel perception feature and the two-dimensional spatial attention graph are combined to generate a feature map with strong discriminability and structural perception ability; the segmentation map is reconstructed and generated by gradually upsampling the feature map.

[0010] Further, by constructing an automatic interactive multi-modal super-resolution guided segmentation network AI-MSGNet, the multi-modal medical image data to be segmented is segmented to generate a segmentation map; the automatic interactive multi-modal super-resolution guided segmentation network includes an encoder and a decoder; the encoder includes a multi-layer multi-scale perception attention module MSPA; the decoder includes a plurality of deformable attention mechanisms and neighborhood modeling strategy modules; the multi-scale perception attention module MSPA includes a dynamic channel attention layer and a spatial attention layer; the dynamic channel attention layer is used to generate an enhanced channel perception feature; the spatial attention layer is used to construct a two-dimensional spatial attention graph; the deformable attention mechanism is used to adaptively focus on the target area by dynamically adjusting the receptive field; the neighborhood modeling strategy module supplements the details missed by the deformable attention mechanism through local convolution.

[0011] Further, a multi-modal fusion image is generated by a super-resolution multi-modal generator; the super-resolution multi-modal generator includes a multi-modal generation network, a super-resolution generation network and a cross-modal fusion module.

[0012] Further, the multi-modal generation network is used to generate a missing enhanced modality image based on multi-modal medical image data, and the process is represented as:

[0013] ;

[0014] wherein, represents a missing enhanced modality image, represents known multi-modal medical image data, represents a multi-modal generation network.

[0015] Further, the super-resolution generation network processes the input multi-modal medical image data to generate a resolution image higher than a threshold value based on a residual dense block in a residual error, combined with dense connection and residual learning.

[0016] Further, an adaptive weighted fusion strategy based on region saliency is adopted to weight and fuse the missing enhanced modality image and the multi-modality medical image data based on region saliency, and a multi-modality fusion image is generated, and the process is represented as:

[0017] ;

[0018] wherein, is a fusion weight, is a multi-modality fusion image.

[0019] Further, the AI-MSGNet is trained by combining a cross-entropy loss and a Dice coefficient loss The total loss of the AI-MSGNet training gamma is calculated by assigning a cross-entropy loss lambda 1 and a dice loss lambda 2 weighting factor, and the total loss is represented as:

[0020] .

[0021] The present application also includes a medical image segmentation system, comprising:

[0022] An acquisition module is configured to acquire multi-modality medical image data to be segmented;

[0023] An enhancement module is configured to generate a missing enhanced modality image based on the multi-modality medical image data, and to weight and fuse the missing enhanced modality image and the multi-modality medical image data based on region saliency to generate a multi-modality fusion image;

[0024] A segmentation module is configured to extract local features of different receptive fields of the multi-modality fusion image, to spatially predict the spatial offset of the local features, to spatially sample the local features of different receptive fields in combination with the predicted spatial offset to obtain a sampling position, to extract query, key and value features from a feature map corresponding to the sampling position, to calculate a temperature-adjusted attention weight by calculating the attention weight of the query and the key and introducing a scale temperature, and to generate enhanced channel perception features according to the temperature-adjusted attention weight and the value features. The segmentation module is further configured to average and maximum pool the channel dimension to construct a two-dimensional spatial attention map, to combine the enhanced channel perception features with the two-dimensional spatial attention map to generate a feature map with strong discriminability and structural perception ability, and to reconstruct and generate a segmentation map by progressively upsampling the feature map.

[0025] The present application also includes a medical image segmentation computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the steps of the medical image segmentation method when executing the computer program.

[0026] The application also includes a readable storage medium storing a computer program, the computer program including program instructions for executing the steps of the medical image segmentation method when executed by a processor.

[0027] The application provides a medical image segmentation method, which has the following beneficial effects:

[0028] The application fills the missing information in the multi-modal medical image by generating the missing enhanced modal image, avoids the decline in segmentation performance caused by incomplete modal, adjusts the sampling position through a dynamically learnable offset, efficiently captures the long-distance dynamic context in the input (such as the semantic correlation between objects in the scene or the global layout of complex structures), and introduces a scale temperature adjusted attention weight, which can adaptively focus on key areas and enhance the perception ability of small structures (such as blood vessels and lesion edges). The designed AI-MSGNet can dynamically adjust the receptive field through a deformable attention mechanism and a neighborhood modeling strategy, adapt to structural changes in different organs and different modalities, and enhance the adaptability to noise, low resolution, and fuzzy boundaries. On the basis of maintaining the full-process automation, the model introduces a self-interaction prompt generation mechanism, so that the model can generate high-quality structure guidance information independently, and through the channel-level multi-modal fusion and deep attention guidance mechanism, the role of the prompt information in feature extraction, semantic modeling and boundary restoration is fully released, realizing the full-process optimization from prompt generation to structure recovery, significantly reducing the dependence on manual input, and improving the segmentation accuracy, robustness and cross-modal generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 FIG. 1 is a schematic diagram of the medical image segmentation process in the embodiment of the application;

[0030] Figure 2 FIG. 4 is a schematic diagram of the performance comparison of the non-frozen parameter part of various deep learning architectures on the Synapse dataset in the embodiment of the application;

[0031] Figure 3 FIG. 6 is a schematic diagram of the GradCAM feature visualization result of the multi-dataset in the embodiment of the application and a schematic diagram of the training loss curve of the corresponding dataset;

[0032] Figure 4 FIG. 8 is a schematic diagram of the AI-MSGNet architecture segmentation network in the embodiment of the application;

[0033] Figure 5 FIG. 10 is a schematic diagram of the Multiscale-Perceived Attention module structure in the embodiment of the application;

[0034] Figure 6 Fig. 9 is a schematic diagram of the segmentation effect of GT and each method in the key area in an embodiment of the present application;

[0035] Figure 7 Fig. 10 is a schematic diagram of the segmentation result of different methods on the right ventricle (RV, red), myocardium (Myo, green) and left ventricle (LV, blue) in a cardiac MRI slice in an embodiment of the present application;

[0036] Figure 8 Fig. 11 is a schematic diagram of the segmentation result of different methods on the target area (red) in a medical image in an embodiment of the present application;

[0037] Figure 9 Fig. 12 is a schematic diagram of the global Dice coefficient (left) and global HD95 distance (right) of different model groups (BwDS, BwoDS, TwDS, TwoDS) of the Synapse dataset in an embodiment of the present application;

[0038] Figure 10 Fig. 13 is a schematic diagram of the global Dice coefficient (left) and global HD95 distance (right) of each model group of the ACDC dataset in an embodiment of the present application;

[0039] Figure 11 Fig. 14 is a schematic diagram of multiple indicators of the CVC-ClinicDB dataset in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0041] The present application proposes an image segmentation method, as shown in Figure 1 A new automatic interactive multi-modal super-resolution guided segmentation network (AI-MSGNet) is designed, and the model includes two stages:

[0042] ① Multi-modal super-resolution guided stage, the existing modal is used to generate the missing or enhanced “T1c” modal, and the original modal is used to construct a high-quality fusion input together.

[0043] ② Image segmentation stage, the network structure is composed of multi-scale perception attention (MSPA), deformable attention mechanism and neighborhood modeling strategy, realizing the structure semantic modeling process from perception, screening to restoration.

[0044] The model architecture maintains full-process automation, introduces a self-interaction prompt generation mechanism, enables the model to autonomously generate high-quality structural guidance information, and through channel-level multi-modal fusion and deep attention guidance mechanism, fully releases the role of prompt information in feature extraction, semantic modeling and boundary restoration. Through structured multi-scale perception attention (MSPA), deformable attention mechanism, neighborhood modeling strategy, AI-MSGNet realizes the full-process optimization from prompt generation to structure recovery, significantly reduces the dependence on manual input, and improves the segmentation accuracy, robustness and cross-modal generalization ability.

[0045] The method specifically comprises the following steps:

[0046] S1, collect medical image data: download the public data set from the Synapse challenge website, the ACDC challenge website, and the Kvasir and CVC-ClinicDB data set website, and adjust each input image to 224x224 size through interpolation to ensure consistent input format.

[0047] S2, super-resolution multi-modal generator: for the problem of low resolution and modal missing in medical image processing restricting the performance of the model, especially in the scene of fine structure recognition and soft tissue boundary segmentation, the image quality degradation has a significant impact on the diagnosis efficiency. The present application proposes an innovative super-resolution multi-modal generator (Super-Resolution Multimodal Generator, SR-MG), which is integrated in the AI-MSGNet framework. The core goal of SR-MG is to improve the details of low-resolution images and generate missing modalities, solving the dual challenges of low resolution and modal missing.

[0048] The design concept of SR-MG is based on two core tasks: super-resolution and multi-modal generation. The super-resolution task aims to restore the details in the image by increasing its resolution, while the multi-modal generation generates missing modalities by utilizing existing modal information, making up for the incompleteness of the data. By combining these two tasks, SR-MG can provide more comprehensive and detailed input images, thereby improving the performance and accuracy of the subsequent segmentation network.

[0049] In medical images, there are often multiple modalities of images, each providing different anatomical information. However, due to device or acquisition limitations, certain modalities (such as T1c enhanced images) may not be available. To address this situation, SR-MG introduces the concept of multi-modal generation, aiming to generate missing modalities from existing ones, thereby supplementing key information in the image. Given the known modalities , SR-MG uses a generative network to generate super-resolution modalities This process is represented by the following equation:

[0050] ;

[0051] In SR-MG, the implementation of the super-resolution task relies on the Residual-in-Residual Dense Block (RRDB) architecture in the residual, which has strong feature extraction and detail recovery capabilities. This architecture extracts image features at multiple scales, effectively removes artifacts in low-resolution images through residual learning, and restores details. The RRDB architecture enhances the learning ability of the network through skip connections and residual structures, making the generated image maintain high resolution while having more detailed features, which is suitable for texture detail recovery in medical images.

[0052] Based on the prompt inspiration of SAM, an efficient multi-modal generation algorithm is designed, which uses labels as mask prompt information to establish spatial constraints, thereby accurately simulating the enhancement effect of contrast agents in the lesion area in the clinical T1c sequence. In the cross-modal fusion stage, an adaptive weighted fusion strategy based on regional saliency is designed in the mask mapper:

[0053] ;

[0054] where is the fusion weight, which controls the proportion of the original image modalities and the generated modalities in the final fused image. The formula shows that the fused image is a weighted combination of the original modalities and the generated modalities. This nonlinear mapping mechanism can effectively maintain the edge sharpness of the lesion area. Experiments show that this fusion method successfully transplants the pathological features of the generated T1c modality while maintaining the integrity of the anatomical structure of the original T1 sequence. SR-MG not only improves the image resolution, but also complements the missing information in the case of modality loss, ensuring that the segmentation network can fully utilize multi-modal information and ultimately improve the accuracy and robustness of segmentation. Subsequently, the multi-modal parser recombines the multi-modal fusion image and the label to adapt to the input of the segmentation network.

[0055] S3, in the encoder stage of AI-MSGNet, the primary task of the model is to extract features from the multi-modal input The ability to proactively and accurately perceive and extract discriminative structural features, especially when processing high-dimensional medical image data, plays a crucial role in whether the model can capture soft boundaries, texture boundaries, and intermodal differences. To enhance the model's ability to perceive fine-grained structural differences, a Multiscale-Perceived Attention Block (MSPA Block) is proposed as the backbone structure of the AI-MSGNet encoder. Its goal is to achieve automatic feature modeling and adaptive perception of structural saliency. MSPA combines dynamic attention mechanisms, multiscale convolutional structures, and spatial awareness, achieving deep fusion of intermodal context while maintaining local sensitivity.

[0056] The core concept of the MSPA module is to enable the network to actively adapt to salient regions at different scales through the construction of multi-scale receptive fields and the guidance of attention mechanisms. Compared to CBAM, MSPA introduces a more flexible dynamic perception structure, which can automatically focus on key regions by introducing guiding information. Its implementation can be abstracted into two stages: dynamic channel attention modeling and spatial saliency reconstruction.

[0057] S3.1 First, deformable attention adjusts the key / value sampling position through dynamic learnable offsets. Combined with grouping mechanisms and stride downsampling, it efficiently captures long-distance dynamic context in the input (such as semantic relationships between objects in the scene or the global layout of complex structures). Its multimodal position encoding (such as logarithmic coordinates or grid sampling) further enhances the flexibility of position information expression. Neighborhood attention, on the other hand, focuses on early enhancement of local fine-grained structures (such as detailed features of edges, textures, or small objects) with a fixed local neighborhood (kernel_size) + dilation mechanism. Its relative position bias (RPB) encodes the precise spatial relationship of pixels in the neighborhood, avoiding the dilution of local information that may be caused by the dynamic sampling of deformable attention blocks (DAB).

[0058] (1) Multi-scale deformable position coding:

[0059] The input features first enter the deformable positional encoding module (including multi-scale residual convolution). Through convolution with different kernel sizes such as 3×3 and 5×5, local spatial features of different receptive fields are extracted (such as small kernels capturing fine-grained textures and large kernels capturing global structures). Multi-scale information is then fused through residual connections to enhance the spatial positional representation of the features (solving the problem of fuzzy spatial information in the original features).

[0060] (2) Offset network coordinate prediction:

[0061] Based on the deformable position-encoding features, a learnable offset convolution composed of convolution, layer normalization, and Gaussian error linear units is used to generate offset networks and matrix convolutions to jointly predict spatial offsets (two-dimensional vectors, such as [batch x group, 2, height, width]). The offsets are used to dynamically adjust the position of subsequent feature sampling, so that attention can be adaptively focused on key areas in the image (such as lesions).

[0062] (3) Dynamic attention modeling:

[0063] Combined with the offset predicted by the offset network, the input features (local spatial features) are adaptively spatially sampled by the dynamic modeling attention module: first, the reference point coordinates (grid points normalized to [-1, 1], representing the spatial position of the original feature) are generated; the offset is added to the reference point to obtain the final sampling position (adjusted spatial coordinates); the key (K) and value (V) features are extracted according to the sampling position using bilinear interpolation (dynamically adjusting the spatial distribution of K / V), and the Q feature is directly generated from the original position feature through linear projection; finally, the query (Q) and the dynamically sampled K are used to calculate the attention weight, realizing the dynamic attention distribution of "focusing on key areas" (solving the problem of fixed window attention that cannot adapt), and standardization of the index and random dropout commonly used in standard attention are also used to maintain the performance, and finally the extracted features are sent to the next step through projection and depth separable convolution.

[0064] S3.2, step S3.1 makes the features output by the encoder have both global semantic consistency and local structure recognition, providing a richer feature basis for subsequent decoding. For this purpose, the MDA attention mechanism is designed, which together with the original attention mechanism forms a new MSPA attention.

[0065] Multi-head attention feature enhancement: the dynamically sampled features are input into the multi-head attention module to further enhance the channel-level expression of the features: 1x1 convolution is used to map the input features into query (Q), key (K), and value (V) (the number of channels remains unchanged); deep separable convolution (groups=channel numberx3) is used to extract the local spatial features of Q / K / V (reducing the amount of calculation while preserving local information); the dot product attention is calculated after normalizing Q / K, and a temperature parameter is introduced (adjusting the sharpness of the attention distribution: the lower the temperature, the more the attention is concentrated on the key area); V is weighted by the attention weight, and the multi-head features are spliced to generate enhanced channel-aware features (capturing feature information in different subspaces, such as texture and shape).

[0066] The channel attention part in MSPA no longer uses the traditional fully connected structure or average pooling method, but replaces it with a more dynamic modeling ability attention module, namely MDA. This structure is derived from the spatial expansion of self-attention mechanism, which makes the calculation of attention have the interpretability of spatial convolution by encoding the query (Q), key (K) and value (V) into local perception blocks, and is compatible with multi-scale feature interaction. Its core calculation method is as follows:

[0067] First, the input image X is obtained by channel-by-channel convolution Q , K , V :

[0068]

[0069] Then, standardization is performed and the attention weight is calculated:

[0070]

[0071]

[0072]

[0073] wherein is the learnable temperature coefficient of each attention head, which is used to control the discreteness of the attention matrix. This mechanism adaptively adjusts the perception range by introducing a scale temperature in each attention head, so that the channel dimension attention has adjustable local receptive ability, and the focus on different regions is strengthened.

[0074] The final weighted value: is obtained, forming the enhanced channel perception feature.

[0075] S3.3, spatial attention map construction: by performing channel dimension statistical fusion on the channel perception feature, a spatial attention map is constructed: average pooling (extracting channel global statistical information, such as regional brightness mean) and maximum pooling (extracting channel local extreme value information, such as lesion boundary high response) are performed respectively; the pooling results are spliced, the channel is compressed to 1 by a small kernel convolution (such as 7x7), and then a two-dimensional spatial attention map (highlighting the spatial key area in the feature, such as the position and boundary of the lesion) is generated by using Sigmoid activation.

[0076] Secondly, MSPA further introduces a spatial attention module to make up for the deficiency of channel dimension attention. This module constructs a two-dimensional spatial attention map by performing average and maximum pooling on the channel dimension, and extracts the spatial saliency feature by a 7x7 convolution kernel. Its formalization process is as follows:

[0077]

[0078]

[0079]

[0080] where, denotes the Sigmoid activation function, and the concatenated tensor generates a spatial attention map after convolution Finally, the original feature map is multiplied by to achieve weighted enhancement of spatial saliency.

[0081] S3.4, Channel-Spatial Feature Fusion:

[0082] The channel and spatial attention information are fused using a gating mechanism (element-level multiplication): first, the channel perception feature is multiplied by the channel attention map (to enhance the expression of key channels such as the texture channel of the lesion); second, the result of the first step is multiplied by the spatial attention map (to enhance the expression of key spatial regions such as the boundary of the lesion); finally, a feature map with strong discriminability and structure perception ability is generated (simultaneously capturing key information at the channel level and the spatial level).

[0083] The overall feature enhancement process of the MSPA module can be represented as:

[0084]

[0085] where A C and A S denote the channel attention and spatial attention maps, respectively, denotes element-wise multiplication. This mechanism, based on multi-scale receptive fields, considers both global semantics and local structures through joint attention guidance, ultimately forming a feature expression with strong discriminability and structure perception ability.

[0086] In the multi-modal scenario, MSPA not only extracts information from salient regions, but also dynamically models the similarities and differences between different modalities, enhancing the information flow between modalities, thereby effectively improving the ability to identify fuzzy boundaries, fine-grained textures, and lesion regions in segmentation tasks.

[0087] S3.5 Output:

[0088] The fused enhanced feature map (shape consistent with the input) is output to the subsequent module (such as the step-by-step upsampling module) for reconstruction of the segmentation map (by upsampling to restore the spatial resolution and generate the final lesion segmentation result).

[0089] The layer-by-layer progression of these components and mechanisms make the encoder output features possess global dynamic context, local fine-grained structure, channel semantic relevance and spatial saliency. This rich feature representation provides strong input for the decoder, which fuses high-resolution features from each stage of the encoder through skip connections, aligns dynamic context with deformable attention (e.g., adaptively aggregates relevant semantic information in the encoder according to the decoding position), restores fine-grained details with neighborhood attention (e.g., corrects the accuracy of segmentation boundaries), and finally maps the features to segmentation masks through the output head (1x1 convolution + Softmax).

[0090] S4、In the training process of AI-MSGNet, a loss function is adopted which strategically combines the cross-entropy loss and the Dice coefficient loss . Specifically, the total loss (L) of AI-MSGNet model training is calculated by assigning weighting factors to the cross-entropy and Dice losses gamma 1 and lambda 2, respectively. The sum of these losses is expressed as: lambda

[0091] .

[0092] Experiments prove:

[0093] In the "Multi-Atlas Abdomen Annotation Challenge" of the 2015 MICCAI conference, a dataset containing 30 abdominal CT images was proposed, with a total of 3779 axial CT layers, image matrix of 512x512 pixels, and voxel size of (0.54x0.98x2.5) mm³ to (0.54x0.98x5.0) mm³. The dataset provides evaluation indicators based on Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD) for eight abdominal organs (aorta, gallbladder, spleen, kidney, liver, pancreas, stomach). The dataset is randomly divided into 18 training sets (2,212 axial layers) and 12 validation sets.

[0094] ACDC: The dataset constructed in the Automatic Cardiac Disease Challenge (ACDC) collects multi-center cardiac dynamic MRI images. The dataset includes cardiac cine magnetic resonance sequences collected under breath-hold conditions, covering the anatomical region from the left ventricular base to the apex, containing consecutive short-axis imaging layers. The image layer thickness is between 5-8mm, and the short-axis image resolution is 0.83-1.75mm² / pixel. All cases are manually annotated by professional physicians for left ventricular (LV), right ventricular (RV) and myocardial (MYO) anatomical structures. The dataset is divided into a training set of 1,902 axial images and a test set of 100 independent validation cases to verify the generalization ability and clinical reliability of the algorithm.​

[0095] Comparison with state-of-the-art methods:

[0096] Experimental results of Synapse dataset: Table 1 compares the performance of the AI-MSGNet series models proposed in the present application and the existing mainstream methods in the multi-organ segmentation task. Two different scale model variants are designed: AI-MSGNet-Tiny (T) and AI-MSGNet-Base (B). Among them, T uses a channel dimension of , and the corresponding number of attention heads is ; B is configured as , and the number of heads is to support higher expression capability. Overall, AI-MSGNet performs best in both DSC and HD indicators. Especially AI-MSGNet-B w / DS, with DSC reaching 95.19 and HD as low as 5.52, significantly better than other methods. In terms of segmentation accuracy of each organ, AI-MSGNet-B w / DS achieves the highest Dice coefficient in most organs (such as liver, SP, SM, etc.), showing its strong ability in multi-modal fusion and spatial detail modeling. This performance benefits from the SR-MG module proposed in the present application, which effectively enhances the reconstruction quality in low resolution and missing modalities, as well as the multi-scale perception ability of the MSPA module in key structural regions. Compared with the latest large model methods such as SAM and nnFormer, AI-MSGNet not only has better overall performance, but also has higher robustness in multiple small organs (such as gallbladder, pancreas, etc.), verifying its wide adaptability and practical value in complex medical image scenarios.

[0097] Table 1 Performance comparison of AI-MSGNet series models and existing mainstream methods in multi-organ segmentation task

[0098]

[0099] Experimental Results on the ACDC Dataset: To further verify the versatility and superiority of AI-MSGNet in organ-level segmentation tasks, this invention compared several state-of-the-art methods on cardiac MRI segmentation. Table 2 lists the Dice coefficient (DSC) performance of different models on the right ventricle (RV), myocardium (Myo), and left ventricle (LV). Traditional models such as TransUnet, SwinU-Net, and SegFormer3D showed stable performance, but their accuracy in myocardial region segmentation was limited. Recent MERIT series methods (such as Cascaded MERIT and Parallel MERIT) and AI-SAM have achieved high overall performance. In particular, LHU-Net achieved a peak DSC of 96.26 in the RV region. In contrast, the AI-MSGNet proposed in this invention outperformed AI-MSGNet in all four metrics. AI-MSGNet-B w / DS achieved an overall DSC of 96.21, and 97.23, 94.23, and 97.16 in the RV, Myo, and LV sub-regions, respectively, demonstrating excellent region recognition and boundary modeling capabilities. The lightweight AI-MSGNet-T w / DS also achieved competitive performance, with an overall DSC of 96.13 and 97.18 and 94.16 on RV and Myo, respectively, surpassing existing state-of-the-art methods on several key architectures. Experimental results further validate the broad adaptability and high reliability of AI-MSGNet in high-precision tasks such as cardiac segmentation, highlighting its potential for practical clinical applications.

[0100] Table 2. Dice coefficient performance of different models on RV, Myo, and LV.

[0101]

[0102] Experimental results on the CVC-ClinicDB dataset: Table 3 shows the quantitative comparison results of AI-MSGNet and several state-of-the-art methods on the CVC-ClinicDB dataset. The experiments used six authoritative metrics: mean Dice coefficient (mDice) measures the overlap between the segmented region and the ground truth annotation; mean intersection-union ratio (mIoU) evaluates segmentation consistency; structural similarity (…). Quantify boundary structure integrity; weighted F-value ( Strengthen boundary pixel weights; enhance alignment metrics. ) Evaluate mask alignment quality; mean absolute error (MAR) Calculate pixel-level error.

[0103] With the multi-modal super-resolution guided architecture and adaptive interaction mechanism, AI-MSGNet breaks the record with a decisive advantage: AI-MSGNet-T w / DS achieves mDice 0.9875 (↑4.2% compared to the best SOTA method Polyp-PVT), as low as 0.0014 (↓76%). This breakthrough is attributed to two innovations: the multi-modal super-resolution module improves image quality consistency and achieves sub-pixel level positioning in the polyp-mucosa fuzzy boundary area by up to 0.9875, an increase of 3.8% over Polyp-PVT); the adaptive interaction mechanism dynamically fuses features, resulting in a performance difference of <0.1% among the four configurations (T / B w / or w / o DS) (e.g., AI-MSGNet-B w / o DS still achieves mDice 0.9874). It is particularly noteworthy that the model's dominant performance on soft boundary indicators: 0.9786 (↑2.7%) verifies its ability to model low-contrast medical images, while approaching the theoretical limit of 0.9988 demonstrates its segmentation consistency in complex anatomical environments to a new height.

[0104] Table 3 Comparison of various advanced methods on the CVC-ClinicDB dataset

[0105]

[0106] Ablation experiments: Table 4 shows the performance of different model configurations in terms of Dice coefficient (DSC) and Hausdorff distance (HD). The baseline model (*Baseline) performs poorly with a DSC of 83.26 and an HD of 19.22. After introducing the MSPA module, the DSC improves to 85.12 and the HD decreases to 14.52, indicating its significant effect on modeling multi-scale semantic features of key structures in multi-modal input. The proposed SR-MG significantly improves performance, with a DSC of 93.52 and an HD of only 4.85, effectively addressing the problems of low resolution and missing modalities. When SR-MG is used in conjunction with MSPA, the model achieves optimal performance (DSC of 95.19 and HD of 5.52), verifying the synergistic advantages of the two.

[0107] Table 4 Ablation study of different single-branch segmentation performance

[0108]

[0109] SR-MG ablation study:

[0110] To further explore the potential of the super-resolution multi-modal generator, extensive research on different paradigms of the super-resolution synthesis module was conducted. As shown in Table 5, various possibilities of T1c-based contrast injection were simulated. Experiments show that classic algorithms such as edge-optimized bilateral filtering and direct application of a highlight filter to the foreground region introduce significant interference information. In contrast, the super-resolution modal obtained using the real-ESRGAN weight as a pre-training parameter is proven to be effective. Therefore, a super-resolution modal guided T1c modal generation method was adopted to design the implementation of the super-resolution multi-modal generator of the present application. The experimental results confirm the effectiveness of the design concept.

[0111] Table 5 Influence of different paradigm strategies on segmentation performance

[0112]

[0113] Network layer ablation experiment:

[0114] This experiment studies the influence of different numbers of skip connections on the segmentation performance of the model. As shown in Table 6, the results show that as the number of skip connection layers increases, the DSC index of the model continuously improves, rising from 88.76 (without skip connection) to 95.19 (with 3 skip connection layers). At the same time, the HD metric overall shows a downward trend, indicating that the boundary prediction accuracy has improved. The addition of skip connections enables the fusion of low-level detail information and high-level semantic features, thereby enhancing the model's ability to recognize target boundaries and shapes. When there are insufficient skip connections, the model's ability to capture fine details is still limited. However, an appropriate number of skip connections can effectively improve feature representation and segmentation results, thereby significantly improving performance.

[0115] Table 6 Influence of different skip connection depths on segmentation performance

[0116]

[0117] Weight ablation experiment:

[0118] This experiment conducts ablation analysis on the weight ratio of cross-entropy loss and Dice loss in the loss function, evaluating the influence of different lambda 1and lambda 2combinations on the segmentation performance of the model. As shown in Table 7, the results show that when lambda 1=4and lambda2=6, the model achieves the best performance on both DSC and HD indicators, reaching 95.19 and 5.52, respectively, indicating that the configuration of the loss function has the best synergistic effect. The reason is that the cross-entropy loss is more sensitive to pixel-level classification and is suitable for optimizing overall classification accuracy, while the dice loss pays more attention to class imbalance and regional structural integrity. When the weights are properly balanced, the model can achieve accurate classification and structural integrity at the same time. On the contrary, unbalanced weights may lead to excessive attention to one aspect, thereby affecting overall performance. Therefore, setting appropriate weight ratios for the loss function is of great significance to improve the segmentation effect of the model.

[0119] Table 7 Influence of loss function weight ratio on segmentation performance

[0120]

[0121] Table 8 AI-MSGNet-B w / DS on Synapse batch_size hyperparameter ablation, DSC, HD represent “batch-dependent training behavior”

[0122]

[0123] Table 9 AI-MSGNet-B w / DS on Synapse learning rate hyperparameter ablation, DSC, HD explore “gradient descent stability”

[0124]

[0125] Table 10 AI-MSGNet-B w / DS on Synapse random seed hyperparameter ablation, DSC, HD evaluate “initialization-induced performance variability”

[0126]

[0127] Figure 1 The overall processing flow of AI-MSGNet is shown, and the model includes two stages: ① multi-modal super-resolution guidance stage, using existing modalities to generate missing or enhanced “T1c” modalities, and constructing high-quality fusion input with the original modalities; ② image segmentation stage, through multi-scale perception attention (MSPA), deformable attention mechanism, and neighborhood modeling strategy to form the network structure, realizing the structure semantic modeling process from perception, screening to restoration. The overall model has the ability of automation, multi-modal, structure sensitivity and resolution recovery.

[0128] Figure 2The performance of the non-frozen parameter part of various deep learning architectures on the Synapse dataset is shown. The Dice similarity coefficient is used as the vertical axis, and the model parameters (Params, M) are used as the horizontal axis. The size of the marker represents the amount of calculation (FLOPs, G).

[0129] Figure 3 (a) is the GradCAM feature visualization result of the multi-dataset (Synapse / ACDC / CVC-ClinicDB). GradCAM, as an explainable AI technology, can intuitively present the attention distribution of the neural network to the key areas of the medical image (such as the organ boundary) through a heat map. The multi-modal module of AI-MSGNet makes the heat map more accurate and closely matches the target organ, strengthening the feature consistency at the soft boundary. Figure 3 The training loss (Loss) of the three datasets (b), (c), and (d) in (b), (c), and (d) changes with the iteration period (Epoch). The convergence trend verifies the optimization ability of the model automatic interaction mechanism for feature fusion: the loss continues to decrease and the fluctuation is smoothed, ensuring the stability of the training.

[0130] Figure 4 is the AI-MSGNet architecture segmentation network. The model uses an encoding-decoding structure, the encoder extracts multi-scale features through multiple layers of Multiscale-Perceived Attention (MSPA), and the bottleneck part further strengthens the deep semantics. The decoder combines Deformable and Neighborhood modules for feature reconstruction, and introduces deep supervision to improve the training effect. Multi-level skip connection realizes efficient feature fusion, enhancing the segmentation accuracy.

[0131] Figure 5 is the Multiscale-Perceived Attention module structure. The left side of the module captures dynamic spatial context through deformable attention mechanism (offset prediction + deformable position encoding), and the right side models local-global feature interaction through neighborhood attention branch (deep convolution + multi-scale multi-head attention). The dual branches are fused through a gating mechanism, realizing the effective combination of deformable spatial adaptation and neighborhood multi-scale perception, enhancing the feature representation ability.

[0132] Figure 6 shows the segmentation effect of GT and each method in the key area, highlighting the differences in boundary accuracy and organ structure preservation. The first three rows show the local details of multiple organs, and the last row shows the overall 3D effect. This method (Ours) is closer to the true label in multiple organs, showing better segmentation consistency and detail restoration ability.

[0133] Figure 7Segmentation results of different methods on the right ventricle (RV, red), myocardium (Myo, green) and left ventricle (LV, blue) in the cardiac MRI slice. The first column is the ground truth (GT), and the rest are the prediction results of each method and the enlarged detail map.

[0134] Figure 8 Segmentation results of different methods on the target region (red) in medical images. The first column is the ground truth (GT), and the rest are the prediction results of each method (Ours, AgileFormer, Swin-UNet, TransUNet), some of which contain enlarged detail maps.

[0135] Figure 9 The global Dice coefficient (left) and global HD95 distance (right) of different model groups (BwDS, BwoDS, TwDS, TwoDS) in the Synapse dataset are shown. Statistical significance (p<0.05) marks the key performance differences, helping to evaluate the trade-off between efficiency and accuracy of the model.

[0136] Figure 10 The global Dice coefficient (left) and global HD95 distance (right) of each model group in the ACDC dataset are presented. By visualizing the central tendency and dispersion, the influence of deep supervision design strategy (with or without DS) on segmentation performance is revealed, and statistical significance provides a reliable basis for result interpretation.

[0137] Figure 11 (a), (b), (c), (d), (e), (f) respectively show the CVC-ClinicDB dataset's multiple indicators mDice, mIoU, , , E , M, a schematic diagram of comprehensive comparison of model groups, reveals the performance difference and significant difference (p<0.05), which is crucial for understanding the robustness of the model in medical image segmentation tasks.

[0138] The application provides an automatic interactive multi-modal super-resolution guided segmentation model AI-MSGNet, which introduces an automatic interaction mechanism and can adaptively adjust a feature extraction and fusion strategy without human intervention, thereby improving segmentation robustness and generalization ability under different data distribution and structural variation conditions. In particular, the AI-MSGNet integrates a multi-modal super-resolution generation module, which not only improves the consistency of image quality between modes, but also generates a missing guide mode, thereby fundamentally alleviating the problem of incomplete medical image data modes. There is a great improvement on Synapse and ACDC, especially in the fuzzy boundary area of the liver and spleen. Research shows that the AI-MSGNet has better segmentation accuracy and stability than existing methods under low resolution, soft boundary and mode missing conditions, thereby verifying the effectiveness and advancement of the AI-MSGNet under the automatic and interactive fusion modeling framework.

[0139] Based on the same inventive concept, the application further provides a medical image segmentation system, which comprises:

[0140] An acquisition module is configured to acquire multi-modal medical image data to be segmented.

[0141] An enhancement module is configured to generate a missing enhanced mode image based on the multi-modal medical image data, and to perform weighted fusion on the missing enhanced mode image and the multi-modal medical image data based on regional saliency to generate a multi-modal fusion image.

[0142] A segmentation module is configured to extract local features of different receptive fields of the multi-modal fusion image, to perform spatial prediction on the spatial displacement of the local features, to perform spatial sampling on the local features of different receptive fields in combination with the predicted spatial displacement to obtain a sampling position, to extract query, key and value features from a feature map corresponding to the sampling position, to calculate temperature-adjusted attention weights by calculating the attention weights of the query and key and introducing a scale temperature, to generate enhanced channel perception features according to the temperature-adjusted attention weights and the value features, to construct a two-dimensional spatial attention map by performing average and maximum pooling on the channel dimension, to combine the enhanced channel perception features and the two-dimensional spatial attention map to generate a feature map with strong discriminability and structural perception ability, and to reconstruct and generate a segmentation map by gradually upsampling the feature map.

[0143] The application further provides an image segmentation computer device, which comprises a memory, a processor and a computer program stored in the memory, and the processor implements the steps of the image segmentation method when executing the computer program.

[0144] The application further provides a readable storage medium, which stores a computer program, and the computer program comprises program instructions, and the program instructions are executed by a processor to implement the steps of the image segmentation method.

[0145] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can make equivalent substitutions or changes according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A medical image segmentation method, characterized in that, Includes the following steps: Acquire multimodal medical image data to be segmented; The missing enhanced modality image is generated based on multimodal medical image data; the missing enhanced modality image and the multimodal medical image data are weighted and fused based on regional saliency to generate a multimodal fused image; wherein the multimodal fused image is generated through a super-resolution multimodal generator; the super-resolution multimodal generator includes a multimodal generation network, a super-resolution generation network and a cross-modal fusion module; Local features of different receptive fields in a multimodal fusion image are extracted, and spatial offsets of local features are spatially predicted. The spatial offsets are combined with spatial sampling of local features of different receptive fields to obtain sampling positions. Query, key, and value features are extracted from the feature map corresponding to the sampling positions. Temperature-modulated attention weights are calculated by calculating the attention weights of queries and keys and introducing scale temperature. Based on the temperature-adjusted attention weights and value characteristics, enhanced channel perception features are generated. A two-dimensional spatial attention map is constructed by averaging and max pooling the channel dimensions. The enhanced channel-aware features are combined with the two-dimensional spatial attention map to generate a feature map with strong discriminative and structural awareness capabilities. A segmentation map is reconstructed by progressively upsampling the feature map. Specifically, an automated interactive multimodal super-resolution guided segmentation network (AI-MSGNet) is constructed to segment the multimodal medical image data to be segmented, generating a segmentation map. The AI-MSGNet includes an encoder and a decoder. The encoder includes a multi-layer multi-scale perceptual attention module (MSPA). The decoder includes multiple deformable attention mechanisms and neighborhood modeling strategy modules. The MSPA includes a dynamic channel attention layer and a spatial attention layer. The dynamic channel attention layer generates enhanced channel-aware features. The spatial attention layer constructs the two-dimensional spatial attention map. The deformable attention mechanism dynamically adjusts the receptive field to adaptively focus on the target region. The neighborhood modeling strategy module supplements details missed by the deformable attention mechanism through local convolution.

2. The medical image segmentation method according to claim 1, characterized in that, The multimodal generation network is used to generate missing enhanced modal images based on multimodal medical image data, and the process is represented as follows: ； in, Indicates the missing augmented modality image. This represents known multimodal medical imaging data. This represents a multimodal generative network.

3. The medical image segmentation method according to claim 1, characterized in that, The super-resolution generation network is based on dense residual blocks in the residuals, and combines dense connections and residual learning to process the input multimodal medical image data to generate images with a resolution higher than a threshold.

4. The medical image segmentation method according to claim 2, characterized in that, An adaptive weighted fusion strategy based on region saliency is used to weight and fuse the missing enhanced modality image with multimodal medical image data based on region saliency, generating a multimodal fused image. The process is represented as follows: ； in, It is a fusion weight. It is a multimodal fused image.

5. The medical image segmentation method according to claim 1, characterized in that, By combining cross-entropy loss and Dice coefficient loss The AI-MSGNet is trained; the total loss of the AI-MSGNet training is... γ By assigning cross-entropy loss respectively λ 1 and Dice Loss λ The total loss is calculated using a weighted factor of 2, and is expressed as follows: 。 6. A medical image segmentation system, characterized in that, include: The acquisition module is used to acquire the multimodal medical image data to be segmented; An enhancement module is used to generate missing enhanced modal images based on multimodal medical imaging data; The missing enhanced modal image is weighted and fused with multimodal medical image data based on regional saliency to generate a multimodal fused image; wherein the multimodal fused image is generated by a super-resolution multimodal generator; the super-resolution multimodal generator includes a multimodal generation network, a super-resolution generation network and a cross-modal fusion module; The segmentation module is used to extract local features of different receptive fields of the multimodal fusion image, perform spatial prediction on the spatial offset of the local features, and perform spatial sampling on the local features of different receptive fields in combination with the predicted spatial offset to obtain the sampling position; extract query, key and value features from the feature map corresponding to the sampling position, calculate the attention weight of query and key and introduce scale temperature to calculate the temperature regulation attention weight. Based on the temperature-adjusted attention weights and value characteristics, enhanced channel perception features are generated. A two-dimensional spatial attention map is constructed by averaging and max pooling the channel dimensions. The enhanced channel-aware features are combined with the two-dimensional spatial attention map to generate a feature map with strong discriminative and structural awareness capabilities. A segmentation map is reconstructed by progressively upsampling the feature map. Specifically, an automated interactive multimodal super-resolution guided segmentation network (AI-MSGNet) is constructed to segment the multimodal medical image data to be segmented, generating a segmentation map. The AI-MSGNet includes an encoder and a decoder. The encoder includes a multi-layer multi-scale perceptual attention module (MSPA). The decoder includes multiple deformable attention mechanisms and neighborhood modeling strategy modules. The MSPA includes a dynamic channel attention layer and a spatial attention layer. The dynamic channel attention layer generates enhanced channel-aware features. The spatial attention layer constructs the two-dimensional spatial attention map. The deformable attention mechanism dynamically adjusts the receptive field to adaptively focus on the target region. The neighborhood modeling strategy module supplements details missed by the deformable attention mechanism through local convolution.

7. A medical image segmentation computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the medical image segmentation method according to any one of claims 1-5.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which includes program instructions that, when executed by a processor, perform the steps of the medical image segmentation method according to any one of claims 1-5.

Images