Multi-organ medical image segmentation method based on multi-feature fusion Swinin-Unet architecture

Abstract

The invention discloses a multi-organ medical image segmentation method based on a multi-feature fusion Swindow-Unet architecture, and belongs to the field of medical image processing. The core of the method is that CT and MRI images are input into a pre-trained CMFSA-UNet model for segmentation, and the model comprises an encoder, an MAFR module, an MFDF module, a decoder and a jump connection layer. CNN-Swin Transform double branches are adopted by the encoder, local details and long-range semantics are extracted, and Attention Gate reinforcement is carried out; the MAFR module widens a receptive field through double branches, combines an attention mechanism with residual connection, reduces the calculated amount and gives consideration to local and global features; and the MFDF module fuses multi-scale dense connection and frequency domain processing, so that feature loss is reduced. The decoder extracts features through Swin Transform Block, resolutions are recovered through 4 times of up-sampling, and the segmentation precision is optimized in combination with depth supervision and a mixed loss function. According to the method, local and long-range feature modeling is efficiently cooperated, precision and efficiency are balanced, segmentation global consistency, boundary accuracy and training stability are improved, the method is suitable for multi-modal multi-organ segmentation, and reliable support is provided for clinical diagnosis and the like.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically a multi-organ medical image segmentation method based on the multi-feature fusion Swin-Unet architecture. Background Technology

[0002] In clinical diagnosis and medical image analysis, multi-organ medical image segmentation is a core component in transforming imaging technologies such as CT and MRI into clinical decision support. It requires precisely delineating the boundaries between target organs such as the liver, pancreas, and kidneys and surrounding tissues, providing quantitative data for critical scenarios such as radiotherapy target delineation, surgical navigation planning, and disease progression monitoring. The accuracy of these segmentation results directly impacts the reliability of clinical diagnosis: segmentation errors can lead to misjudgments of radiotherapy dosage and mistakes in surgical path planning, thus significantly affecting patient treatment outcomes and safety. Therefore, high-precision, high-efficiency multi-organ medical image segmentation technology plays an irreplaceable role in improving the quality of diagnosis and treatment and reducing medical risks.

[0003] In recent years, deep learning-based medical image segmentation technologies (such as the UNet series and Transformer architecture) have made significant progress in single-organ segmentation tasks due to their powerful feature modeling capabilities. However, in multi-organ collaborative segmentation scenarios, existing algorithms still face many technical bottlenecks. On the one hand, medical image data has inherent problems of "difficult annotation and few samples": multi-organ annotation requires professional physicians to spend hours annotating pixel by pixel, and large-scale public datasets are scarce due to privacy protection regulations; at the same time, there are significant individual differences in organ morphology (such as liver size and pancreas location) among different patients, and the proportion of defective samples (such as organ deformation after tumor invasion) is low, which makes the model training prone to overfitting and poor segmentation accuracy stability. On the other hand, traditional segmentation models (especially 3D architectures) have a large number of parameters and high computational complexity: multi-organ segmentation requires processing high-resolution 3D image data (such as CT images with millions of voxels), and existing models have slow inference speeds in real-time clinical scenarios (such as rapid intraoperative image segmentation), making it difficult to meet the clinical demand for "second-level response".

[0004] Furthermore, current multi-organ segmentation algorithms have significant shortcomings in feature fusion and detail capture: First, multi-scale feature integration is insufficient, and shallow features (such as organ boundary texture) are difficult to effectively combine with deep semantic features (such as organ category information), leading to blurred boundaries and segmentation confusion of neighboring organs (such as the pancreas and duodenum); Second, the ability to model features of small organs (such as the gallbladder and adrenal glands) is weak—these organs account for a low proportion in images and have limited feature information, making existing models prone to omissions or incomplete segmentation. Therefore, how to reduce model computation costs and improve inference efficiency while ensuring multi-organ segmentation accuracy (especially boundary accuracy and small organ integrity), and adapting to scenarios where medical data is scarce, is a core problem that urgently needs to be solved in the field of multi-organ medical image segmentation. The disclosure of the above background technology is only for assisting in understanding the concept and technical solution of this invention, and it does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content was disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0005] This application provides a multi-organ medical image segmentation method based on the Swin-Unet architecture with multi-feature fusion, which can reduce the computational load of the model and improve the segmentation efficiency while ensuring segmentation accuracy.

[0006] To achieve the above objectives, the embodiments of this application disclose the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a multi-organ medical image segmentation method based on the multi-feature fusion Swin-Unet architecture, comprising the following steps:

[0008] Receive CT and MRI image data of organs;

[0009] The received CT and MRI image data are input into the pre-trained CMFSA-UNet model for image segmentation.

[0010] The CMFSA-UNet model consists of an encoder layer, a MAFR module layer, a skip connection layer (MFDF module), and a decoder layer. The steps of the CMFSA-UNet model for segmenting multi-organ medical images include:

[0011] CT and MRI image data are input into the encoder layer for multi-scale feature extraction, generating a set of feature maps containing semantic information at different levels.

[0012] The feature map set is input into the MAFR module layer, and global-local features are fused through parallel convolution and attention mechanism to output an enhanced aggregated feature map;

[0013] During the feature transfer process between the encoder and decoder layers, cross-scale feature fusion and frequency domain enhancement operations are performed through the skip connection layer (MFDF module) to output a multi-scale feature map adapted to the decoder.

[0014] The fused feature map is input into the decoder layer, upsampled to restore spatial resolution, and combined with a deep supervision mechanism to generate a segmentation result map containing pixel-level categories of each organ.

[0015] In some possible implementations of the first aspect, the encoder layer includes a CNN-Swin Transformer dual-branch structure and an Attention Gate fusion module. The CNN branch contains a 7×7 initial convolutional module and a ResBlock module with residual connections. The Swin Transformer branch contains a Patch Partition module, a Double Swin Transformer Block module, and a Patch Merging module. The multi-scale feature extraction steps include:

[0016] The medical image is first extracted using the 7×7 initial convolutional module of the CNN branch, generating an initial feature map containing the low-level local texture;

[0017] The initial feature map is input into the ResBlock module, and the feature transfer is enhanced by 3×3 convolution-Batch Norm-ReLU operation and residual connection. At the same time, downsampling is performed to compress the spatial dimension and expand the channel dimension, and the middle-layer local feature map is output.

[0018] The medical image is synchronously input into the Patch Partition module of the Swin Transformer branch, segmented into 4×4 non-overlapping image patches, and linearly projected onto the embedding space to generate initial embedding features.

[0019] The initial embedded features are input into the Double Swin Transformer Block module, which alternately models long-range dependencies through W-MSA and SW-MSA, and then performs three downsampling and feature aggregations through the Patch Merging module to output a high-level global feature map.

[0020] The mid-level local feature map and the high-level global feature map are input into the Attention Gate fusion module. After feature reshaping and element addition, complementary fusion is achieved. The data is then fed back to the dual branches for further downsampling, and finally outputs a set of feature maps that fuse multi-scale semantics and details.

[0021] In some possible implementations of the first aspect, the MAFR module layer includes parallel heterogeneous convolutional branches (AKConv module, ScConv module), SEMAConv module, CBAM module, and residual connection units, and the feature enhancement steps include:

[0022] The feature map output by the encoder is synchronously input into the AKConv module and the ScConv module. AKConv captures multi-scale local features through adaptive convolution kernels, and ScConv outputs AKConv features and ScConv features respectively through channel splitting and interactive optimization of channel association.

[0023] The weight coefficient 'a' is generated by the sigmoid function. Then, 'a' is multiplied by the ScConv feature element, and '(1-a)' is multiplied by the AKConv feature element and then added together to obtain the preliminary fused features.

[0024] The initial fused features are input into the SEMAConv module, and the channel-space interaction is optimized a second time by the ScConv submodule. Then, the spatial features are dynamically weighted by the EMA attention submodule to output enhanced multi-scale features.

[0025] The enhanced multi-scale features are input into the CBAM module, and channel dependence is learned through global average pooling and fully connected layers to strengthen the key organ feature channels and output channel-enhanced features.

[0026] The initial input features are added to the channel-enhanced features by the residual connection unit, and the output is an aggregated feature map that takes into account both global semantics and local details.

[0027] In some possible implementations of the first aspect, the skip connection layer (MFDF module) includes a multi-fusion dense connection unit, a VggBlock convolutional unit, an FSAS frequency domain processing module, and a feature adaptation component. The cross-scale feature fusion step includes:

[0028] Receive feature maps from each stage of the encoder, and perform adaptive upsampling on feature maps of different scales through multiple fusion dense connection units (matching the resolution of the corresponding stage of the decoder), and concatenate them with the current layer features to generate multi-scale aggregated initial features;

[0029] The initial features of multi-scale aggregation are input into the VggBlock convolutional unit, and the hierarchical features are integrated through continuous convolution and ReLU activation, outputting a densely fused feature map;

[0030] The densely fused feature map is input into the FSAS frequency domain processing module. The dimensionality compatibility is adjusted by dynamic channel adaptation. The feature structure is optimized by 1×1 convolution and depthwise separable convolution. Then, it is transformed into the frequency domain by FFT to capture long-range dependencies.

[0031] In the frequency domain, the feature map is mapped back to the spatial domain by frequency domain multiplication and inverse IFFT transformation. Key features are selected by gating mechanism, and then fused with spatial features through residual connection to output frequency domain enhanced feature map.

[0032] The frequency domain enhanced feature map is adjusted and normalized by the feature adaptation component to match the corresponding stage feature map of the decoder, and the adaptive fusion feature is output.

[0033] In some possible implementations of the first aspect, the decoder layer includes a Swing Transformer Block unit, a 4x upsampling unit, a Deep Supervision module, and a normalization layer, and the segmentation result generation steps include:

[0034] The aggregated feature map output by the MAFR module is input into the Swing Transformer Block unit, and the semantic association is strengthened through the self-attention mechanism to output a deep semantic feature map.

[0035] The deep semantic feature map is input into the 4x upsampling unit, and the spatial resolution is increased to 4x by bilinear interpolation while the channel dimension is compressed to 1 / 4, and the feature map with the appropriate resolution is output.

[0036] The adaptive resolution feature map is concatenated with the adaptive fusion feature map output by the skip connection layer, integrating details and semantic information to output the decoder fusion feature;

[0037] The decoder's fused features are input into the Deep Supervision module, adjusted to the number of organ categories by 1×1 convolution, upsampled to the original image size by bilinear interpolation, and then activated by sigmoid to generate the stage-side output segmentation map.

[0038] The decoder's fused features are normalized to stabilize training. The above feature extraction, upsampling, fusion, and supervision steps are repeated to gradually restore the complete spatial resolution.

[0039] In the final stage, the output feature map is processed by 1×1 convolution and sigmoid activation to generate a segmentation result map containing pixel-level category labels for each organ. The cross-entropy-Dice hybrid loss function is then used to optimize the organ boundary accuracy.

[0040] Secondly, embodiments of this application provide a multi-organ medical image segmentation system, including:

[0041] The first receiving unit is used to receive CT and MRI image data of organs;

[0042] The first analysis unit is used to input the received CT and MRI image data into the pre-trained CMFSA-UNet model for multi-organ segmentation.

[0043] The CMFSA-UNet model includes an encoder layer, a MAFR module layer, a skip connection layer (MFDF module), and a decoder layer. The steps of segmenting multi-organ images using the CMFSA-UNet model include:

[0044] CT and MRI image data are input into the encoder layer for multi-scale feature extraction, generating a set of feature maps containing semantic information at different levels.

[0045] The feature map set is input into the MAFR module layer, and global-local features are fused through parallel convolution and attention mechanism to output an enhanced aggregated feature map;

[0046] During the feature transfer process between the encoder and decoder layers, cross-scale feature fusion and frequency domain enhancement operations are performed through the skip connection layer (MFDF module) to output a multi-scale feature map adapted to the decoder.

[0047] The fused feature map is input into the decoder layer, upsampled to restore spatial resolution, and combined with a deep supervision mechanism to generate a segmentation result map containing pixel-level categories of each organ.

[0048] In some possible implementations of the second aspect, the encoder layer includes a CNN-Swin Transformer dual-branch structure and an Attention Gate fusion module, wherein the CNN branch includes a 7×7 initial convolution module and a ResBlock module with residual connections, and the Swin Transformer branch includes a Patch Partition module, a Double Swin Transformer Block module, and a Patch Merging module, and the first analysis unit is specifically used for:

[0049] The medical image is first extracted using the 7×7 initial convolutional module of the CNN branch, generating an initial feature map containing the low-level local texture;

[0050] The initial feature map is input into the ResBlock module, and the feature transfer is enhanced by 3×3 convolution-Batch Norm-ReLU operation and residual connection. At the same time, downsampling is performed to compress the spatial dimension and expand the channel dimension, and the middle-layer local feature map is output.

[0051] The medical image is synchronously input into the Patch Partition module of the Swin Transformer branch, segmented into 4×4 non-overlapping image patches, and linearly projected onto the embedding space to generate initial embedding features.

[0052] The initial embedded features are input into the Double Swin Transformer Block module, which alternately models long-range dependencies through W-MSA and SW-MSA, and then performs three downsampling and feature aggregations through the Patch Merging module to output a high-level global feature map.

[0053] The mid-level local feature map and the high-level global feature map are input into the Attention Gate fusion module. After feature reshaping and element addition, complementary fusion is achieved. The data is then fed back to the dual branches for further downsampling, and finally outputs a set of feature maps that fuse multi-scale semantics and details.

[0054] In some possible implementations of the second aspect, the MAFR module layer includes parallel heterogeneous convolutional branches (AKConv module, ScConv module), SEMAConv module, CBAM module, and residual connection unit, wherein the first analysis unit is further specifically used for:

[0055] The feature map output by the encoder is synchronously input into the AKConv module and the ScConv module. AKConv captures multi-scale local features through adaptive convolution kernels, and ScConv outputs AKConv features and ScConv features respectively through channel splitting and interactive optimization of channel association.

[0056] The weight coefficient 'a' is generated by the sigmoid function. Then, 'a' is multiplied by the ScConv feature element, and '(1-a)' is multiplied by the AKConv feature element and then added together to obtain the preliminary fused features.

[0057] The initial fused features are input into the SEMAConv module, and the channel-space interaction is optimized a second time by the ScConv submodule. Then, the spatial features are dynamically weighted by the EMA attention submodule to output enhanced multi-scale features.

[0058] The enhanced multi-scale features are input into the CBAM module, and channel dependence is learned through global average pooling and fully connected layers to strengthen the key organ feature channels and output channel-enhanced features.

[0059] The initial input features are added to the channel-enhanced features by the residual connection unit, and the output is an aggregated feature map that takes into account both global semantics and local details.

[0060] In some possible implementations of the second aspect, the skip connection layer (MFDF module) includes a multi-fusion dense connection unit, a VggBlock convolutional unit, an FSAS frequency domain processing module, and a feature adaptation component, wherein the first analysis unit is further specifically used for:

[0061] Receive feature maps from each stage of the encoder, and perform adaptive upsampling on feature maps of different scales through multiple fusion dense connection units (matching the resolution of the corresponding stage of the decoder), and concatenate them with the current layer features to generate multi-scale aggregated initial features;

[0062] The initial features of multi-scale aggregation are input into the VggBlock convolutional unit, and the hierarchical features are integrated through continuous convolution and ReLU activation, outputting a densely fused feature map;

[0063] The densely fused feature map is input into the FSAS frequency domain processing module. The dimensionality compatibility is adjusted by dynamic channel adaptation. The feature structure is optimized by 1×1 convolution and depthwise separable convolution. Then, it is transformed into the frequency domain by FFT to capture long-range dependencies.

[0064] In the frequency domain, the feature map is mapped back to the spatial domain by frequency domain multiplication and inverse IFFT transformation. Key features are selected by gating mechanism, and then fused with spatial features through residual connection to output frequency domain enhanced feature map.

[0065] The frequency domain enhanced feature map is adjusted and normalized by the feature adaptation component to match the corresponding stage feature map of the decoder, and the adaptive fusion feature is output.

[0066] In some possible implementations of the second aspect, the decoder layer includes a Swing Transformer Block unit, a 4x upsampling unit, a Deep Supervision module, and a normalization layer, wherein the first analysis unit is further configured to:

[0067] The aggregated feature map output by the MAFR module is input into the Swing Transformer Block unit, and the semantic association is strengthened through the self-attention mechanism to output a deep semantic feature map.

[0068] The deep semantic feature map is input into the 4x upsampling unit, and the spatial resolution is increased to 4x by bilinear interpolation while the channel dimension is compressed to 1 / 4, and the feature map with the appropriate resolution is output.

[0069] The adaptive resolution feature map is concatenated with the adaptive fusion feature map output by the skip connection layer, integrating details and semantic information to output the decoder fusion feature;

[0070] The decoder's fused features are input into the Deep Supervision module, adjusted to the number of organ categories by 1×1 convolution, upsampled to the original image size by bilinear interpolation, and then activated by sigmoid to generate the stage-side output segmentation map.

[0071] The decoder's fused features are normalized to stabilize training. The above feature extraction, upsampling, fusion, and supervision steps are repeated to gradually restore the complete spatial resolution.

[0072] In the final stage, the output feature map is processed by 1×1 convolution and sigmoid activation to generate a segmentation result map containing pixel-level category labels for each organ. The cross-entropy-Dice hybrid loss function is then used to optimize the organ boundary accuracy.

[0073] The one or more technical solutions provided in the first aspect of this application have at least the following technical effects or advantages:

[0074] The CMFSA-UNet model, built on the traditional UNet architecture, is improved by introducing a CNN-Swin Transformer dual-branch encoder to enhance the collaboration between local and global features. The MAFR module is used to balance feature capture and computational efficiency. MFDF skip connections are designed to enhance multi-scale feature transfer. Combined with deep supervision, the segmentation accuracy is optimized, effectively solving the problems of local detail loss, small organ omission, and excessive computational cost in multi-organ segmentation.

[0075] The technical effects of any design method in the second aspect can be found in the technical effects of different design methods in the first aspect, and will not be repeated here. Attached Figure Description

[0076] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0077] Figure 1 Flowcharts of the CMFSA-UNet model provided for some embodiments of this application;

[0078] Figure 2 Schematic diagram of the MAFR module provided for some embodiments of this application;

[0079] Figure 3 This is a schematic diagram of the CNN Encoder Branches module provided in some embodiments of this application.

[0080] Figure 4 This is a schematic diagram of a skip connection (MFDF) module provided for some embodiments of this application. Detailed Implementation

[0081] Specific embodiments of the invention will now be described in detail. Although the invention is described in conjunction with these specific embodiments, it should be understood that it is not intended to limit the invention to these specific embodiments. Rather, these embodiments are intended to cover alternative, modified, or equivalent embodiments that may be included within the spirit and scope of the invention as defined by the claims. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. The invention may be practiced without some or all of these specific details.

[0082] When used in conjunction with the terms "comprising," "method comprising," or similar language in this specification and appended claims, the singular forms "a," "some," and "the" include plural references unless the context clearly indicates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0083] Application Overview: In clinical diagnosis and medical image analysis, multi-organ medical image segmentation is a core component in transforming imaging technologies such as CT and MRI into clinical decision support. It requires accurately delineating the boundaries between target organs such as the liver, pancreas, and kidneys and surrounding tissues, providing quantitative data for critical scenarios such as radiotherapy target delineation, surgical navigation planning, and disease progression monitoring. The accuracy of these segmentation results directly impacts the reliability of clinical diagnosis: segmentation errors can lead to misjudgments of radiotherapy dosage and errors in surgical path planning, thus significantly affecting patient treatment outcomes and safety. Therefore, high-precision, high-efficiency multi-organ medical image segmentation technology is crucial for improving the quality of diagnosis and treatment and reducing medical risks.

[0084] In recent years, deep learning-based medical image segmentation technologies (such as the UNet series and Transformer architecture) have made significant progress in single-organ segmentation tasks due to their powerful feature modeling capabilities. However, existing algorithms still have some problems in multi-organ collaborative segmentation scenarios. On the one hand, medical image data has inherent problems of "difficult annotation and few samples": multi-organ annotation requires professional physicians to spend hours annotating pixel by pixel, and large-scale public datasets are scarce due to privacy protection regulations; at the same time, there are significant individual differences in organ morphology (such as liver size and pancreas location) among different patients, and the proportion of lesion samples (such as organ deformation after tumor invasion) is low, which makes the model prone to overfitting during training, resulting in poor segmentation accuracy stability. On the other hand, traditional segmentation models (especially 3D architectures) have a large number of parameters and high computational complexity: multi-organ segmentation requires processing high-resolution 3D image data (such as CT images with millions of voxels), which is difficult to meet the "second-level response" requirements in real-time clinical scenarios (such as rapid segmentation of intraoperative images).

[0085] For the above technical issues, please refer to [link / reference]. Figure 1and Figure 2 This application provides a multi-organ medical image segmentation method based on the Swin-Unet architecture with multi-feature fusion, including the following steps:

[0086] The CMFSA-UNet model consists of an encoder layer (CNN-Swin Transformer dual-branch structure), MAFR module layers, skip connection layers (MFDF modules), and a decoder layer. The steps of the CMFSA-UNet model for multi-organ medical image segmentation include:

[0087] The first step is to input multi-organ medical image data (such as abdominal CT images with a resolution of 512×512) into the encoder layer for multi-scale feature extraction, generating a set of feature maps containing semantic information at different levels;

[0088] Specifically, in some embodiments, the encoder layer includes a CNN branch (containing a 7×7 initial convolutional module and a ResBlock module with residual connections) and a Swin Transformer branch (containing a Patch Partition module, a Double Swin Transformer Block module, and a Patch Merging module), and an Attention Gate fusion module is provided between the two branches. The multi-scale feature extraction steps include:

[0089] The first sub-step involves inputting the medical image into the 7×7 initial convolutional module of the CNN branch, performing spatial compression (stride 2) and channel expansion (expanding from 3 channels to 64 channels) through convolution operations, generating a first-scale feature map with a size of 256×256 and 64 channels (denoted as P1 / 2, where "1 / 2" represents the resolution scaling ratio relative to the input image).

[0090] The second sub-step involves inputting the P1 / 2 feature map into the ResBlock module, enhancing the feature representation through a "3×3 convolution-BatchNorm-ReLU" operation, and simultaneously performing downsampling with a stride of 2 to generate a mid-level local feature map (denoted as P2 / 4) with a size of 128×128 and 128 channels.

[0091] The third sub-step involves synchronously inputting the medical image into the Patch Partition module of the Swing Transformer branch, dividing it into 4×4 non-overlapping image blocks (each block contains 16 pixels), and mapping it to a 96-dimensional embedding space via linear projection to generate initial embedding features of size 128×128 and 96 channels.

[0092] The fourth sub-step involves inputting the initial embedded features into the Double Swin Transformer Block module, which then models long-range dependencies using W-MSA (window multi-head self-attention) and SW-MSA (shift window multi-head self-attention) in sequence. Finally, the Patch Merging module downsamples the data (step size 2) to generate a high-level global feature map (denoted as P3 / 8) with a size of 64×64 and 192 channels.

[0093] The fifth sub-step involves inputting the P2 / 4 feature maps from the CNN branch and the P3 / 8 feature maps from the Swin Transformer branch into the Attention Gate fusion module. First, the P3 / 8 maps are upsampled by a factor of 2 (to match the P2 / 4 size of 128×128), then reshaped (all adjusted to 128×128×128 dimensions) and added element-wise to output a set of feature maps that fuse multi-scale semantics and details. The set of feature maps includes, but is not limited to, P1 / 2 (256×256×64), P2 / 4 (128×128×128), and P3 / 8 (64×64×192) level feature maps.

[0094] The second step involves inputting the high-level feature maps (P3 / 8) from the feature map set into the MAFR module layer, fusing global and local features through parallel convolution and attention mechanisms, and outputting an enhanced aggregated feature map.

[0095] Specifically, in some embodiments, the MAFR module layer includes parallel heterogeneous convolutional branches (AKConv module, ScConv module), SEMAConv module (including ScConv sub-module and EMA attention sub-module), CBAM module and residual connection unit, and the feature fusion step includes:

[0096] The first sub-step involves synchronously inputting the P3 / 8 feature map (64×64×192) output by the encoder into the AKConv and ScConv modules: AKConv captures multi-scale local features through adaptive convolution kernels (dynamically adjusting the kernel size) and outputs a 64×64×192 AKConv feature map; ScConv strengthens the correlation between channels through channel splitting (dividing into 2 groups) and interactive convolution, and outputs a 64×64×192 ScConv feature map.

[0097] The second sub-step involves generating learnable weight coefficients a (dimension 1×1×192) using the sigmoid function, multiplying a element-wise with the ScConv feature map, and multiplying (1-a) element-wise with the AKConv feature map. The two are then added together to obtain a preliminary fused feature map of 64×64×192.

[0098] The third sub-step involves inputting the preliminary fused feature map into the SEMAConv module: first, the channel-space interaction is optimized twice by the ScConv sub-module, and then the feature space distribution is dynamically weighted by the EMA attention sub-module (exponential moving average to calculate spatial weights), outputting an enhanced multi-scale feature map of 64×64×192.

[0099] The fourth sub-step involves inputting the enhanced multi-scale feature map into the CBAM module. Through global average pooling (compressed to 1×1×192) and learning the channel dependency relationship with the fully connected layer, the channel weights are adaptively adjusted, and a 64×64×192 channel enhanced feature map is output.

[0100] The fifth sub-step involves adding the initial input P3 / 8 feature map with the channel-enhanced feature map elements through the residual connection unit, preserving the original feature details, and finally outputting a 64×64×192 MAFR aggregated feature map.

[0101] The third step involves inputting the multi-scale feature maps (P1 / 2, P2 / 4) output from the encoder layer and the MAFR aggregated feature map into the skip connection layer (MFDF module) to perform cross-scale feature dense fusion and frequency domain enhancement operations, and outputting feature maps adapted to the decoder.

[0102] Specifically, in some embodiments, the MFDF module includes multi-scale dense connection units, VggBlock convolution units, an FSAS frequency domain processing module (including a dynamic channel adaptation submodule, an FFT transform submodule, and an IFFT inverse transform submodule), and a feature adaptation component. The cross-scale feature processing steps include:

[0103] The first sub-step involves receiving the P1 / 2 (256×256×64) and P2 / 4 (128×128×128) feature maps output from the encoder layer, upsampling the P2 / 4 feature map by a factor of 2 using a multi-scale dense connection unit (bilinear interpolation to match the P1 / 2 size of 256×256), and then concatenating the upsampled P2 / 4 feature map with the P1 / 2 feature map to generate a multi-scale initial aggregated feature with a size of 256×256 and 192 channels.

[0104] The second sub-step inputs the multi-scale initial aggregated features into the VggBlock convolutional unit (containing two "3×3 convolution-Batch Norm-ReLU" structures), integrates feature information at different levels through continuous convolution operations, strengthens the non-linear expression of features, and outputs a dense fused feature map of 256×256×192.

[0105] The third sub-step involves inputting the densely fused feature map into the FSAS frequency domain processing module: first, the number of channels is adjusted to 64 through the dynamic channel adaptation sub-module (1×1 convolution), and then mapped to the frequency domain space through the FFT transform sub-module to capture long-range contextual dependencies; after performing frequency domain multiplication (with preset frequency domain weights) in the frequency domain, it is mapped back to the spatial domain through the IFFT inverse transform sub-module, and key features are selected by combining the gating mechanism (sigmoid activation);

[0106] The fourth sub-step involves fusing the original spatial features (densely fused feature map) with the frequency-domain processed features through residual connections, outputting a 256×256×192 frequency-domain enhanced feature map; then, through a feature adaptation component (1×1 convolution + normalization), the number of channels is adjusted to 64, generating an adaptive fused feature map (denoted as F1 / 2) with a size of 256×256×64, which is used for fusion with the corresponding stage features of the decoder.

[0107] The fourth step involves inputting the MAFR aggregated feature map and the MFDF adaptive fusion feature map into the decoder layer. After upsampling to restore the spatial resolution and combining with depth supervision, a multi-organ pixel-level segmentation result map is generated.

[0108] Specifically, in some embodiments, the decoder layer includes a Swin Transformer Block module, a 4x upsampling unit (bilinear interpolation), a Deep Supervision module (containing a 1×1 convolutional submodule and a sigmoid activation submodule), and a cross-entropy-Dice hybrid loss calculation unit. The segmentation result generation steps include:

[0109] The first sub-step involves inputting the MAFR aggregated feature map (64×64×192) into the Swin TransformerBlock module, strengthening semantic associations through a self-attention mechanism, and outputting a deep semantic feature map of 64×64×192.

[0110] The second sub-step involves performing a 4x upsampling on the deep semantic feature map (restoring it from 64×64 to 256×256), and simultaneously compressing the number of channels to 64 through a 1×1 convolution to generate an intermediate feature map of the decoder with a size of 256×256×64.

[0111] The third sub-step involves concatenating the intermediate feature map from the decoder with the F1 / 2 feature map (256×256×64) output from the MFDF module to generate a fused feature map of 256×256×128, which is then input into the Deep Supervision module. The module performs a 1×1 convolution to adjust the number of channels to correspond to the number of organ categories (e.g., 5 categories: liver, pancreas, kidney, spleen, gallbladder, corresponding to 5 channels), then performs a 2x upsampling (restoring the input image size to 512×512), and finally activates it with a sigmoid function to generate the stage-side output segmentation map (pixel values ​​representing organ category probabilities).

[0112] The fourth sub-step repeats the above "feature extraction-upsampling-feature fusion-side output" process: the 256×256 fused feature map is processed again by the Swing Transformer Block and upsampled by 2 times, and finally restored to a resolution of 512×512. After 1×1 convolution and sigmoid activation, the final multi-organ segmentation result map is generated.

[0113] The fifth sub-step involves calculating the loss value between the final segmentation result image and the labeled image (pixel-level category ID, such as liver 1, pancreas 2, background 0) based on the cross-entropy-Dice hybrid loss calculation unit. For example, the cross-entropy loss accounts for 40% and the Dice loss accounts for 60% in the hybrid loss, which is used to optimize the organ boundary segmentation accuracy. At the same time, the stage-side output segmentation image and the labeled image are used to calculate the loss, and the model parameters are updated synchronously through backpropagation.

[0114] The sixth sub-step is to use the highest probability category ID of each pixel in the final segmentation result image as the organ category label of that pixel, and generate a segmentation result containing the outline of multiple organ regions and category identifiers. For example, it is stored in the form of pixel coordinates (x,y) corresponding to organ category ID, such as (120,150)→1 (liver) and (280,320)→2 (pancreas).

[0115] The specific training process is as follows: the model is not initialized using pre-trained parameters, but is trained from scratch.

[0116] Set hyperparameters: initial learning rate of 0.05, batch size of 24, and number of training epochs of 150; use AdamW optimization algorithm to update model parameters;

[0117] Training Process: During training, training set data (including pixel-level labeled CT / MRI images) is input into the model in batches. The model calculates the cross-entropy-Dice mixed loss function value based on the input images and labeled information. The gradient is calculated using the backpropagation algorithm, and the model parameters are updated using the AdamW optimization algorithm to gradually reduce the loss function value. Every 10 rounds, the model performance is evaluated on the validation set (multi-organ image data independent of the training set) using Dice similarity coefficient and Hausdorff distance as evaluation metrics. If the Dice coefficient on the validation set does not improve for 20 consecutive rounds, training is stopped early to prevent overfitting. After training, the weights of the model with the best performance on the validation set are saved as the final inference model.

[0118] The experimental results comparing the CMFSA-UNet model of this invention with other advanced models are shown in the table below:

[0119] method aorta gallbladder Left kidney Right kidney liver pancreas spleen Stomach Dess similarity coefficient 95% Hausdorf Distance Swin-Unet 85.99 65.02 83.28 76.54 94.20 56.72 88.83 75.06 78.21 23.25 U-Net 87.36 59.14 81.16 76.23 92.96 52.81 85.68 73.38 76.09 37.56 TransUNet 85.26 64.03 75.93 76.59 92.82 51.39 81.87 72.02 74.99 29.74 SCUNET++ 87.04 67.64 84.24 79.60 94.70 60.65 88.14 77.54 79.89 25.41 MissFormer 81.69 60.49 75.87 68.62 93.26 43.42 86.33 69.81 72.44 31.59 CMFSA-UNet 86.41 67.67 84.59 82.44 94.48 55.31 88.66 79.00 79.90 20.06

[0120] The experimental results show that the new model has achieved a significant improvement in overall accuracy.

[0121] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A multi-organ medical image segmentation method based on the Swin-Unet architecture with multi-feature fusion, characterized in that, Includes the following steps: Receive CT and MRI image data of organs; The received CT and MRI image data are input into a pre-trained CMFSA-UNet model for medical image segmentation. The CMFSA-UNet model includes an encoder layer, a MAFR module layer, a skip connection layer, and a decoder layer. The image segmentation steps performed by the CMFSA-UNet model include: The CT and MRI image data of the organs are input into the encoder layer for multi-scale feature extraction, generating a set of feature maps containing semantic information at different levels; The feature map set is input into the MAFR module layer, and the receptive field is broadened through parallel dual branches of AKConv and ScConv. Combined with SEMAConv, CBAM attention mechanisms and residual connections, the computational cost is reduced while capturing both local and global features. The fused feature map is input into the skip connection layer. Based on frequency domain processing and multi-scale feature dense connection, feature maps of different scales are fused to enhance the original features and then output to the decoder layer.

2. The fused feature map is input into the decoder layer, and features are extracted through the Swing Transformer Block, the spatial resolution is restored by 4x upsampling, and a deep supervision mechanism and a cross-entropy-Dice hybrid loss function are introduced to optimize the organ boundary segmentation accuracy and training stability.

3. The multi-organ medical image segmentation method based on a multi-feature fusion Swin-Unet architecture according to claim 1, characterized in that, The encoder layer includes a CNN-Swin Transformer dual-branch structure and an Attention Gate fusion module. The CNN-Swin Transformer dual-branch structure comprises CNNEncoder Branches and a Swing Transformer branch. The CNN Encoder Branches have a 7×7 initial convolutional module and a ResBlock module with residual connections (using 3×3 convolution-Batch Norm-ReLU operation). The Swing Transformer branch has a Patch Partition module, a Double Swing Transformer Block module (embedding window-based multi-head self-attention W-MSA and shift-window-based multi-head self-attention SW-MSA), and a Patch Merging module. The multi-scale feature extraction steps include: performing initial feature extraction on the multi-organ medical input image using the 7×7 initial convolutional module of the CNN Encoder Branches to initially capture local texture details of the organs and generate an initial feature map containing low-level local features; and inputting the initial feature map into the ResBlock. The module ensures feature transfer integrity through residual connections, while downsampling to compress spatial dimensions and expand channel dimensions, outputting a mid-level local feature map that enhances local detail representation. The multi-organ medical input image is synchronously input to the Patch Partition module of the Swin Transformer branch, segmented into 4×4 non-overlapping image blocks, and linearly projected onto a preset embedding space dimension to generate initial embedding features. The initial embedding features are input to the Double Swin Transformer Block module, which alternately models long-range image dependencies using W-MSA and SW-MSA, and then performs three rounds of downsampling and feature aggregation through the Patch Merging module to generate a high-level global feature map containing global semantic information. The mid-level local feature map output from the CNN Encoder Branches and the high-level global feature map output from the Swin Transformer branch are input together to the Attention Gate fusion module, where complementary fusion of local and global features is achieved through feature reshaping and element-wise addition, and then fed back to the dual branches for further downsampling operations, finally outputting a feature map set integrating multi-scale semantic and detail information.

4. The multi-organ medical image segmentation method based on a multi-feature fusion Swin-Unet architecture according to claim 2, characterized in that, The MAFR module includes parallel heterogeneous convolutional branches (AKConv module, ScConv module), SEMAConv fusion unit (including ScConv sub-module and EMA attention sub-module), CBAM channel-spatial attention module, and residual connection component. The parallel heterogeneous convolutional branches broaden the receptive field through different convolutional mechanisms; the SEMAConv fusion unit integrates convolutional operations and attention mechanisms; the CBAM module enhances key feature channels and spatial regions; and the feature processing steps of the MAFR module include: The AKConv module receives the aggregated features X output by the encoder through parallel heterogeneous convolution branches, captures multi-scale local textures with adaptive convolution kernels, and the ScConv module outputs AKConv features and ScConv features respectively after channel splitting and interactive optimization of feature channel association. The AKConv and ScConv features are dynamically weighted and fused: a weight coefficient a is generated by the sigmoid function, a is multiplied by the ScConv feature element, (1-a) is multiplied by the AKConv feature element, and the two are added together to obtain the preliminary fused features; The preliminary fusion features are input into the SEMAConv fusion unit: first, the ScConv submodule optimizes the channel and spatial interaction, and then the EMA attention submodule dynamically weights the feature spatial distribution to highlight key organ regions and output enhanced multi-scale features. The enhanced multi-scale features are input into the CBAM module: channel dependencies are learned through global average pooling and fully connected layers, and organ boundary features are enhanced by spatial attention mechanism, outputting channel-spatial dual-enhanced features; The initial aggregated feature X is added to the channel-space dual-enhanced feature element by the residual connection component, preserving the original feature details and fusing the processed features, and finally outputting a MAFR module feature map that takes into account both global semantics and local details.

5. The multi-organ medical image segmentation method based on the multi-feature fusion Swin-Unet architecture according to claim 3, characterized in that, The skip connection module is an MFDF (Multi-Fuse Dense Frequency Domain Skip Connection) module, comprising a multi-fusion dense connection unit (including a multi-scale feature upsampling submodule), a VggBlock convolutional integration unit, an FSAS frequency domain enhancement module (including a dynamic channel adaptation submodule, a 1×1 convolutional projection submodule, a depthwise separable convolutional splitting submodule, and a frequency domain transformation submodule), and a feature adaptation component. The multi-fusion dense connection unit aggregates multi-stage features from the encoder, the VggBlock unit enhances the nonlinear representation of features, the FSAS module captures long-range dependencies through frequency domain operations, and the feature fusion and transfer steps of the MFDF module include: The feature map set output from each stage of the encoder is received, and the feature maps at different scales are adaptively upsampled through multiple fusion dense connection units (matching the resolution of the corresponding stage of the decoder). The upsampled features are then concatenated with the features of the current level to generate multi-scale aggregated initial features. The multi-scale aggregated initial features are input into the VggBlock convolutional integration unit. Through continuous convolution operations and nonlinear transformation (ReLU), feature information at different levels is integrated to enhance feature expression capabilities and output a densely fused feature map. The densely fused feature map is input into the FSAS frequency domain enhancement module: the feature dimension is adjusted by the dynamic channel adaptation submodule to be compatible with multi-stage input, the feature structure is optimized by the 1×1 convolution projection submodule and the depthwise separable convolution splitting submodule, and then mapped to the frequency domain space by the frequency domain transformation submodule (FFT) to capture long-distance contextual dependencies. In the frequency domain space, global information is mapped back to the spatial domain through frequency domain multiplication and inverse frequency transform (IFFT). Key features are selected by gating mechanism, and then fused with the original features in the spatial domain through residual connection. Local details are preserved and feature transfer is stabilized, and frequency domain enhanced feature map is output. The frequency domain enhanced feature map is dimensionally adjusted and normalized by the feature adaptation component to match the resolution and number of channels of the corresponding stage feature map of the decoder. Finally, it is passed to the decoder to realize multi-scale feature fusion, reduce the loss of downsampling information, and enhance the expression of organ boundaries and global consistency.

6. The multi-organ medical image segmentation method based on the multi-feature fusion Swin-Unet architecture according to claim 4, characterized in that, The decoder module includes a Swing Transformer Block feature extraction unit, a 4x upsampling unit (including a bilinear interpolation submodule), a Deep Supervision unit (including a 1×1 convolution submodule and a sigmoid activation submodule), a normalization layer, and a feature fusion component. The Swing Transformer Block unit captures long-range feature dependencies, the upsampling unit restores spatial resolution, and the Deep Supervision unit provides multi-stage supervision signals. The feature processing and segmentation steps of the decoder module include: The system receives the global-local fusion feature map output by the MAFR module, inputs it into the Swing Transformer Block feature extraction unit, models the long-distance dependency between features through the self-attention mechanism, strengthens the semantic information association, and outputs a deep semantic feature map. The deep semantic feature map is input into a 4x upsampling unit, and the spatial resolution is increased to 4 times the original size through a bilinear interpolation submodule. Simultaneously, the feature channel dimension is compressed to 1 / 4 through a convolution operation, and a mid-level feature map adapted to the resolution is output. The mid-level feature map is spliced ​​and fused with the multi-scale feature map processed by the MFDF module at the corresponding stage of the encoder by the feature fusion component, integrating low-level details and high-level semantic information, and outputting a fused feature map. The fused feature map is input into the Deep Supervision unit: first, the channels are adjusted to the number of segmentation categories by a 1×1 convolution submodule, then it is upsampled to the input image size by bilinear interpolation, and finally a stage-side output segmentation map is generated by a sigmoid activation submodule, and training is supervised by the real labels. The fused feature map is processed by a normalization layer (Batch Norm) to stabilize the feature distribution and optimize the training process. Then, the above steps of Swing Transformer Block extraction, upsampling, feature fusion and deep supervision are repeated to gradually restore the complete spatial resolution of the image. In the final stage, the output feature map is processed by 1×1 convolution and sigmoid activation to generate the final segmentation map. The cross-entropy loss and Dice loss are combined to form a hybrid loss function, which balances the class weights and strengthens the organ boundary constraints, and outputs accurate multi-organ segmentation results.

7. The first receiving unit is used to receive multi-organ medical image data (such as CT and MRI image data); The first analysis unit is used to input the received multi-organ medical image data into a pre-trained CMFSA-UNet model for multi-organ segmentation. The CMFSA-UNet model includes an encoder module, a MAFR module, a decoder module, and a skip connection module. The steps of segmenting multi-organ medical images using the CMFSA-UNet model include: The multi-organ medical image data is input into the encoder module, and features are extracted through the CNN-Swin Transformer dual-branch structure: the CNN branch captures local detail features through 7×7 initial convolution and ResBlock, and the Swin Transformer branch models long-range dependencies through Patch Partition and Double Swin Transformer Block. The features of the two branches are fused through Attention Gate to generate a multi-scale feature map set. The multi-scale feature map set is input into the MAFR module, the receptive field is broadened by the parallel branches of AKConv and ScConv, key features are enhanced by the SEMAConv module and CBAM attention mechanism, and global-local fusion features are output by combining residual connections. The global-local fusion features are input into the decoder module, and deep semantic features are extracted by the Swin Transformer Block. Spatial resolution is gradually restored by 4x upsampling, and a deep supervision mechanism is introduced to generate multi-stage side output feature maps. During the feature transfer process between the encoder and decoder, cross-scale feature fusion is achieved through the MFDF module: encoder features are aggregated through multiple fusion dense connection units, long-range context is captured through the FSAS frequency domain processing module, and enhanced skip connection features are output and fused with corresponding stage features of the decoder. The feature map output by the decoder is processed by 1×1 convolution and sigmoid activation function to generate a segmentation result map containing pixel-level classification labels for each organ region, where each pixel corresponds to the organ category identifier.

Images