A 3D abdominal multi-organ image segmentation method based on improved EM-Net

Abstract

The application discloses a 3D abdominal multi-organ image segmentation method based on an improved EM-Net, and comprises the following steps: firstly, a data set is extracted and preprocessed; then, an improved network model is constructed based on an EM-Net model, an adaptive dynamic state space modulation module ADSSM is used to replace a feature enhancement module in an original EM-Net encoder; and an adaptive multi-scale feature interaction module AMFI is integrated in each stage of the EM-Net encoder to replace an original multi-scale fusion component; finally, an adaptive multi-modal context modulation module AMCM is used to replace a feature fusion module in an EM-Net decoder; secondly, the improved model is trained; the trained model is verified; finally, the improved network model is used for segmenting BTCV abdominal multi-organ images, and a segmented image is output. The application has strong feature expression capability and high segmentation precision, and is suitable for an abdominal CT multi-organ segmentation scene.

Description

Technical Field

[0001] This invention relates to the technical field of medical image segmentation, specifically to a 3D abdominal multi-organ image segmentation method based on an improved EM-Net. Background Technology

[0002] EM-Net, an improved medical image segmentation model based on the Mamba architecture, achieves medical image segmentation through feature extraction and multi-scale fusion. It has been applied in image segmentation scenarios such as abdominal CT and MRI. However, it still has significant shortcomings in practical applications and cannot meet the clinical demand for high-precision segmentation.

[0003] The core shortcomings of the existing EM-Net model are mainly reflected in three aspects: First, the feature enhancement module in the encoder only uses a single spatial feature extraction method and does not effectively fuse sequence features, resulting in insufficient ability to capture fine structures (such as small organs like the pancreas and adrenal glands) in medical images and blurred segmentation boundaries; Second, the multi-scale fusion component uses a simple feature stitching method, lacks deep interaction of cross-scale features, has too many redundant features, and cannot effectively filter out features useful for segmentation, further affecting segmentation accuracy; Finally, the feature fusion module in the decoder uses a single fusion method and does not fully utilize the complementarity of multimodal features, resulting in insufficient utilization of multimodal features, leading to insufficient feature restoration ability and a large deviation between the segmentation results and the actual organ contours.

[0004] In abdominal CT image segmentation scenarios, the BTCV dataset includes 13 types of abdominal organs such as the spleen, liver, pancreas, and adrenal glands. These targets exhibit significant scale differences, blurred boundary features, and adhesion to surrounding tissues, placing higher demands on the feature representation and fusion capabilities of segmentation models. Existing EM-Net models reveal several technical shortcomings when processing this type of data: First, the original feature enhancement module cannot effectively fuse sequential and spatial features, essentially capturing only information from a single perspective, failing to fully capture the subtle structural features of organs, resulting in low segmentation accuracy for small organs; second, the original multi-scale fusion component lacks cross-scale feature interaction, failing to effectively filter effective features, and redundant features interfere with the segmentation results, leading to poorly aligned organ boundaries; third, the original decoder's feature fusion method is simplistic, insufficiently utilizing multimodal features, and inadequate feature reconstruction capabilities, failing to accurately reconstruct the three-dimensional structure of organs, thus affecting the accuracy of clinical diagnosis.

[0005] Current improvements to the EM-Net model are mostly simple optimizations of single functions. There is no end-to-end module improvement scheme for the fusion of sequence features and spatial features, multi-scale feature interaction, and multi-modal feature modulation. It is difficult to adapt to the segmentation requirements of abdominal CT images in the BTCV dataset, and it cannot meet the high-precision requirements of clinical medical image segmentation. Summary of the Invention

[0006] To address the technical problems of insufficient feature fusion, low precision in fine structure segmentation, and inadequate utilization of multimodal features in existing EM-Net models, this technical solution provides a 3D abdominal multi-organ image segmentation method based on an improved EM-Net. It systematically reconstructs the key processes of the original EM-Net, and through the coordinated operation of three modules—enhanced encoder features, cross-scale interaction, and optimized decoder features—it achieves strong feature representation capabilities and high segmentation precision, effectively solving the technical problems.

[0007] This invention is achieved through the following technical solution:

[0008] A 3D abdominal multi-organ image segmentation method based on an improved EM-Net includes the following steps: constructing an improved network model based on the EM-Net model, training and validating the improved network model, and finally using the improved network model to segment BTCV abdominal multi-organ images; the operation mode of the improved network model includes:

[0009] (1) The original feature enhancement module in the EM-Net encoder is replaced by the Adaptive Dynamic State Space Modulation (ADSSM) module, which is used to extract sequence features and spatial features from BTCV abdominal multi-organ image data to achieve adaptive fusion of sequence modeling and spatial features.

[0010] (2) The adaptive multi-scale feature interaction module AMFI is integrated into each stage of the EM-Net encoder to replace the original multi-scale fusion component;

[0011] (3) The feature fusion module in the EM-Net decoder is replaced by the adaptive multimodal context modulation module AMCM.

[0012] Furthermore, the Adaptive Dynamic State Space Modulation (ADSSM) module is a parallel dual-branch architecture. It takes 3D features (B, C, D, H, W) as direct input, with one path used for feature complexity evaluation and the other for main feature processing. The module's operation flow is as follows:

[0013] First, the features are fed into the State Adaptor, and after 1×1×1 convolution, (B, 3, D, H, W) is obtained. Then, after Softmax normalization and global average pooling, a weight vector of (B, 3) is obtained. The optimal state space dimension d_state is adaptively selected.

[0014] Next, the backbone input features are flattened into a sequence form (B, C, T), where T = D×H×W, and converted into a sequence modeling format (B, T, C);

[0015] Then the data is fed into the Mamba branch and the standard attention branch in parallel for processing, with both paths maintaining the same dimensions (B, T, C).

[0016] Then, the two outputs are fused by attention through learnable weights λ to recover the spatial feature shape (B, C, D, H, W);

[0017] Then, multi-scale extraction is performed on the feature: feature learning is performed at the original scale, 1 / 2 scale, and 1 / 4 scale respectively, and then all scale features are uniformly upsampled back to the original size to obtain three sets of features of the same size (B, C, D, H, W);

[0018] Subsequently, the three sets of multi-scale features are dynamically weighted and fused using learnable scale weights, and the fused features (B, C, D, H, W) are output.

[0019] Finally, the fused features are sequentially processed through an MLP layer and a second Mamba process, and then residually connected with the original input features to output the enhanced features (B, C, D, H, W), providing more robust feature support for subsequent feature interactions.

[0020] Furthermore, the adaptive multi-scale feature interaction module AMFI takes the enhanced features (B, C, D, H, W) output by ADSSM as input. AMFI takes the enhanced features output by ADSSM as input and performs multi-scale feature extraction through three extractors with different time lengths. After upsampling the downsampled features to unify the size, it performs multi-scale averaging fusion, cross-scale attention interaction, fusion gating filtering, and then performs residual connection with the original input features.

[0021] Furthermore, the operation flow of the adaptive multi-scale feature interaction module AMFI is as follows:

[0022] First, multi-scale feature extraction is performed by parallel processing using three extractors with different stride lengths of stride=1, stride=2, and stride=4: the extractor with stride=1 keeps the original feature size unchanged, while the extractors with stride=2 and stride=4 downsample the features by 1 / 2 and 1 / 4 respectively, outputting features of the corresponding scales (B, C, D, H, W), (B, C, D / 2, H / 2, W / 2), and (B, C, D / 4, H / 4, W / 4).

[0023] Subsequently, an upsampling operation is performed on the two downsampled feature sets to restore them to the original feature size (B, C, D, H, W). These features are then fed into the Multi-scale AverageFusion module along with the original scale features to complete the initial fusion and output features with dimensions (B, C, D, H, W).

[0024] The fused features enter the Cross-scale Attention module, where they are processed by 1×1 convolution, BN batch normalization, ReLU activation function and double 1×1 convolution to enhance cross-scale feature interaction.

[0025] Next, the data is fed into the 3D Fusion Gate module, where spatial feature mapping is performed on the cross-scale fusion features through 1×1×1 3D convolution. Then, a 3D spatial weight matrix (B,C,D,H,W) with the same dimension as the features is generated by the Sigmoid activation function. The cross-scale features are then adaptively weighted and filtered according to spatial location, which strengthens the feature response of key locations of organ boundaries and fine structures, suppresses the interference of invalid features from background and redundant regions, and outputs the filtered effective 3D cross-scale features.

[0026] Finally, the gated features are residually connected with the original input features, and after adaptive weighted fusion, the enhanced features (B, C, D, H, W) are output, improving the model's ability to express features of organ boundaries and fine structures.

[0027] Furthermore, the adaptive multimodal context modulation module AMCM takes the enhanced features (B, C, D, H, W) output by AMFI as input and processes them in parallel through two paths: one path extracts multimodal features, and the other path generates global modulation parameters. The module's operation flow is as follows:

[0028] First, the global modulation branch feeds the input features into the global pooling layer, where they are then processed by global average pooling to obtain... The channel description vector, then through two layers Convolution with Sigmoid activation generates a learnable scaling factor. With bias factor This is used to model channel-level global context information;

[0029] Next, the multimodal feature extraction branch applies multi-scale 3D convolutional kernels to the input features in parallel: such as... , , Feature extraction is performed on different receptive fields to generate multiple sets of complementary feature maps, which are then concatenated through channels to obtain fused features. ;

[0030] Then, this feature is sent to the interactive gating compression module, and then... Convolutional dimensionality reduction and channel attention weights generated using the Sigmoid algorithm are applied to adaptively filter and enhance multimodal features, outputting interactively enhanced features. ;

[0031] Finally, the global modulation parameters , By performing affine transformation modulation on the interactive enhancement features and then outputting optimized features through residual connection, the precise fusion and channel-level optimization of multimodal features are achieved, thereby improving the feature restoration capability and organ boundary reconstruction accuracy of the decoder.

[0032] Furthermore, in the multimodal extraction branch, the input features are divided into three parallel modal channels. Each channel is subjected to feature extraction through a process of 1×1 convolution, BN batch normalization, GELU activation, and 1×1 convolution, ensuring that the output dimension of each modality remains unchanged (B, C, D, H, W).

[0033] Subsequently, the features of the three modalities are concatenated using Channel Concat to obtain concatenated features with dimensions (B, 3C, D, H, W). Then, the number of channels is compressed by the Interaction Gate module through 1×1 convolution, BN, and SiLU activation to output interactive features of (B, C, D, H, W).

[0034] In the global modulation branch, the input features are processed by AdaptiveAvgPool 3D to compress the spatial dimension and extract channel-level global semantic features (B,C,1,1,1).

[0035] The data is then fed into a 3D modulation generator, where global semantic features are mapped through 1×1×1 3D convolutions. The sigmoid activation function then generates channel-level 3D global modulation parameters (B,C,1,1,1). These parameters assign independent scaling factors to each feature channel, enabling channel-wise amplitude scaling and distribution calibration of multimodal interactive features. This eliminates distribution differences between different modal features and improves the consistency and accuracy of multimodal feature fusion.

[0036] Finally, the interaction features and modulation parameters are modulated to produce Out = Input × Modulation + Interaction. After adjusting the feature dimensions through a 1×1 convolution of the output projection, the final multimodal fusion features (B, C, D, H, W) are output, achieving accurate fusion and optimization of multimodal features and improving the feature restoration capability of the decoder.

[0037] Based on EM-Net, the above three major modules are improved to obtain an improved medical image segmentation architecture model dominated by three major modules: sequence modeling enhancement, multi-scale interaction, and multi-modal modulation.

[0038] Furthermore, prior to the step of constructing an improved network model based on the EM-Net model, there are steps of extracting the dataset and preprocessing the dataset.

[0039] Furthermore, the dataset used is the Abdominal Organs Dataset (BTCV) provided by Vanderbilt University Medical Center. This dataset contains 30 cases of abdominal CT volume data in the portal venous phase, covering pixel-level precise annotations of 13 abdominal anatomical structures, including the spleen, right / left kidney, gallbladder, esophagus, liver, stomach, aorta, and pancreas. The number of samples for each case is 130-170 images, and the cross-sectional slice resolution is 512×512. To facilitate the model's rapid reading of 3D volume data, the .nii / .nii.gz format 3D volume data of the dataset is converted to NPY format, and the pixel values ​​of the 3D volume data are normalized.

[0040] Furthermore, the dataset is re-divided into training, validation, and test sets in an 8:1:1 ratio.

[0041] Furthermore, the training of the improved network model was performed on the AutoDL computing cloud. The test hardware platform environment was Python 3.10, CUDA 11.8, PyTorch 2.1.2, NVIDIA RTX4090D GPU, and 60G of memory. The training hyperparameters were set as follows: 500 training rounds, BatchSize set to 1, and SGD optimizer used.

[0042] Beneficial effects

[0043] This invention proposes a 3D abdominal multi-organ image segmentation method based on an improved EM-Net, which has the following advantages compared with existing technologies:

[0044] (1) In the encoder, the original fixed-structure feature enhancement module is replaced with an adaptive dynamic state space modulation module (ADSSM). A mechanism is introduced to dynamically evaluate feature complexity and automatically select the optimal state space dimension. Through parallel modeling with Mamba and attention branches, the limitations of traditional sequence modeling in capturing fine structures in medical images are overcome. In the skip connection between the encoder and decoder, an adaptive multi-scale feature interaction module (AMFI) is embedded. Through cross-scale attention modeling and gated interactive compression, dynamic association and accurate fusion between high and low level features are achieved, which is different from the simple splicing method of the original model. In the decoder, the original feature fusion module is replaced with an adaptive multimodal context modulation module (AMCM). Through multimodal parallel extraction, channel-level global modulation parameter generation and affine transformation modulation, adaptive calibration and optimization of multimodal features are achieved, which significantly improves the feature restoration capability of the decoder. This improves the segmentation accuracy and robustness of the model on abdominal CT images of the BTCV dataset, reduces clinical diagnostic errors, and meets the actual application needs of clinical medicine.

[0045] (2) This invention replaces the original feature enhancement module with an Adaptive Dynamic State Space Modulation (ADSSM) module. The original model's feature enhancement module uses fixed-dimensional sequence modeling, which cannot dynamically adjust the modeling capability according to the complexity of the input features, resulting in the dilution of features of fine organs with high information density. This module evaluates the complexity of the input features through a state adapter and adaptively selects the optimal state space dimension, so that Mamba remains lightweight when processing simple regions and has stronger modeling capabilities when processing complex organ regions. At the same time, a standard attention branch is introduced to process in parallel with Mamba. Mamba is good at capturing global sequence dependencies, while the attention mechanism strengthens key local features. The two are fused through learnable weights to achieve complementary enhancement of sequence context and spatial local features. Subsequent multi-scale dynamic weighted fusion further aggregates feature information from different receptive fields, enabling the model to capture the boundaries and morphological details of small organs such as the pancreas and gallbladder more accurately. Finally, it achieves deep fusion of sequence features and spatial features, effectively enhancing the model's ability to capture the fine structure of medical images, and fundamentally solving the problems of insufficient feature fusion and low segmentation accuracy of small organs in the original model.

[0046] (3) In this invention, an adaptive multi-scale feature interaction module (AMFI) is integrated into each stage of the encoder. The original model's multi-scale fusion only uses simple channel splicing, which lacks effective information interaction between features at different levels. Low-level spatial details cannot be effectively transmitted to high-level features, and high-level semantic information is also difficult to guide low-level features, resulting in blurred organ boundary segmentation. This module first extracts features of different receptive fields through multi-scale convolution, and then calculates the correlation weights between high and low-level features through a cross-scale attention mechanism to establish a bidirectional association between semantic information and spatial details. Subsequently, the interaction features are filtered through fusion gating to suppress redundant features that are irrelevant to the segmentation task and strengthen the key information of organ boundaries. This approach realizes deep interaction and adaptive filtering of multi-scale features, reduces redundant feature interference, significantly improves the segmentation fit of organ boundaries, and solves the defects of insufficient interaction and blurred boundaries in the original multi-scale fusion components.

[0047] (4) This invention replaces the original decoder feature fusion module with an adaptive multimodal context modulation module (AMCM). The original decoder feature fusion module lacks global calibration and adaptive modulation of multimodal information, and only performs simple addition or splicing of features at different levels, which cannot effectively utilize the complementary information of different modalities, resulting in the loss of organ contour details reconstructed by the decoder. On the one hand, this module extracts multimodal features in parallel through multi-scale 3D convolution, and then achieves adaptive screening of information from different receptive fields through channel splicing and interactive gating compression. On the other hand, it generates channel-level modulation parameters through global pooling and fully connected layers, performs affine transformation modulation on the interactively enhanced features, and injects global context information into each channel. This globally guided multimodal fusion method achieves accurate fusion and optimization of multimodal features, enabling the decoder to make full use of multi-scale and multimodal complementary information, accurately restore the edge details and morphological features of organs, significantly improve the feature restoration capability of the decoder, and solve the problems of insufficient utilization of multimodal features and weak detail reconstruction capability of the original model.

[0048] (5) Through the above three modular improvements, this invention achieves three core capability enhancements—sequence feature enhancement, multi-scale interactive optimization, and multi-modal modulation fusion—while retaining the overall architecture and interface compatibility of the EM-Net model. It can accurately adapt to the segmentation requirements of 13 abdominal organs in the BTCV dataset, and its overall segmentation accuracy and robustness are significantly better than the original model, demonstrating good practicality and clinical application value. At the same time, this network structure retains some of the architecture and input / output interfaces of the original EM-Net model, possessing strong compatibility and applicability, providing reliable technical support for the accurate diagnosis and treatment of abdominal diseases. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the overall segmentation process of the present invention.

[0050] Figure 2 This is a diagram of the improved EM-Net network model architecture in this invention.

[0051] Figure 3 This is a structural diagram of the ADSSM in this invention.

[0052] Figure 4 This is a structural diagram of AMFI in this invention.

[0053] Figure 5 This is a structural diagram of the AMCM in this invention.

[0054] Figure 6 The figure shows the experimental results of the improved structural model in this invention. Detailed Implementation

[0055] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. The described embodiments are merely some embodiments of the present invention, and not all embodiments. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the design concept of the present invention should fall within the protection scope of the present invention.

[0056] Example 1:

[0057] like Figure 1 As shown, a 3D abdominal multi-organ image segmentation method based on an improved EM-Net is described, with the following specific implementation steps:

[0058] Step 1: Extract the dataset. The dataset used is the abdominal organ dataset BTCV, provided by Vanderbilt University Medical Center and publicly available at the MICCAI2015 Workshop. This dataset contains 30 cases of abdominal CT images during the portal venous phase, covering pixel-level precise annotations of 13 abdominal anatomical structures, including the spleen, right / left kidney, gallbladder, esophagus, liver, stomach, aorta, and pancreas. The number of samples for each case is 130-170 images, with the image format being .nii / .nii.gz and the cross-sectional slice resolution being 512×512.

[0059] Step 2: Preprocess the dataset; re-divide the dataset into training, validation, and test sets in an 8:1:1 ratio; the training set contains 3840 images, the validation set contains 480 images, and the test set contains 480 images. To facilitate the model's rapid reading of 3D volume data, convert the .nii / .nii.gz format 3D volume data of the dataset to NPY format, and normalize the pixel values ​​of the 3D volume data.

[0060] Step 3: Using EM-Net as the baseline model, improve the network architecture of the model; the improved EM-Net network model architecture is as follows. Figure 2 As shown, its overall architecture still adopts an encoder-decoder structure. First, the 3D abdominal CT body data is input into the network and mapped to shallow features by the Patch Embedding module. This feature is directly passed to the end of the decoder through the Conv Block, while another path enters the STEM module to complete initial downsampling and feature enhancement. Subsequently, the data enters the encoder, which contains four cascaded stages. Each stage consists of an ADSSM module and an AMFI module connected in series. ADSSM achieves adaptive fusion of sequence and spatial features, and AMFI completes multi-scale feature interaction enhancement. As the layers deepen, the feature resolution decreases step by step, and the number of channels increases step by step. The multi-scale features output by each stage are sent to the corresponding layer of the decoder through skip connections. The decoder is composed of an upsampling module and an AMCM module alternately. The feature resolution is gradually restored through the Up-Sample Block, while the AMCM module performs multimodal context modulation and fusion of skip connection features and upsampling features. Finally, the features output by the decoder are fused across layers with the shallow features passed from the PatchEmbedding stage, and then the Conv Block module generates an organ segmentation mask of the same size as the input volume data to complete the 3D abdominal multi-organ segmentation task.

[0061] Step 3.1: The original feature enhancement module in the EM-Net encoder is replaced by the Adaptive Dynamic State Space Modulation (ADSSM) module. This module is used to extract sequence features and spatial features from the BTCV abdominal 3D CT body data, achieving adaptive fusion of sequence modeling and spatial features. This solves the problems of insufficient fusion of sequence features and spatial features, inadequate capture of fine structures, and low segmentation accuracy in the original model.

[0062] The architecture of the Adaptive Dynamic State Space Modulation Module (ADSSM) is as follows: Figure 3 As shown. ADSSM takes 3D features (B, C, D, H, W) as direct input and is executed synchronously by two parallel branches: a feature complexity evaluation branch and a main feature processing branch. First, the features are fed into a state adapter, and after a 1×1×1 convolution, the channels are mapped to three candidate state dimensions, and the output feature shape becomes (B, 3, D, H, W). Then, after Softmax normalization and global average pooling, a (B, 3) dimensional state weight vector is obtained, thereby achieving adaptive selection of the state space dimension. The specific calculation method is as follows:

[0063] ;

[0064] Where X is the input 3D feature of ADSSM, for 1. Three-dimensional convolution, The output state weight vector is used to characterize the importance of different state spaces.

[0065] Based on the aforementioned state weight vector, the three preset candidate state space dimensions are weighted and summed to obtain the optimal state space dimension that best suits the features of the current batch. :

[0066] ;

[0067] in, The first element in the state weight vector represents the... The weight value of the i-th component is used to characterize the i-th component. The importance of the dimensions of the candidate state space. The preset candidate dimensions are {32, 64, 128}. As a core modeling parameter of the Mamba branch, it enables dynamic matching between the state space and organ features.

[0068] In the main feature processing flow, the input 3D features are flattened into a sequence form (B,C,T) (where T=D×H×W is the sequence length), and converted into a sequence modeling format (B,T,C). These are then fed in parallel into the Mamba branch and the standard attention branch for feature learning, with both paths maintaining the same dimensions (B,T,C). The Mamba branch adapts the state space dimension. Modeling long-distance sequences for parameters:

[0069] ;

[0070] In the formula, The input features are serialized. For the aforementioned adaptive state space dimension, For the convolution kernel dimension, These are channel expansion coefficients used to capture sequence length dependencies, thereby obtaining the sequence features output by the Mamba branch. .

[0071] The standard attention branch simultaneously models the spatial correlation of features in the same sequence to enhance the spatial local details and regional associations:

[0072] ;

[0073] In the formula, Spatial augmentation features for the output of the standard attention branch. For multi-head self-attention operations, using sequence features Using the input as input, spatial feature correlation calculation is completed through query, key, and value matrix mapping, which is used to enhance the spatial correlation and detailed response of local areas of organs.

[0074] Then the two outputs are passed through learnable weights. Weighted fusion yields a hybrid feature that combines sequence length dependency and spatial detail:

[0075] ;

[0076] in, These are learnable, dynamically fused weights used to automatically balance the proportions of sequence features and spatial features. This is a weighted fusion of the Mamba branch and the attention branch, resulting in a hybrid feature.

[0077] The fused sequence features were then restored to their 3D spatial shape. Subsequently, multi-scale extraction was performed on the feature, with feature learning conducted at the original scale, half-scale, and quarter-scale. All scale features were then uniformly upsampled back to the original size, resulting in three sets of features of the same size. Dynamic weighted fusion was achieved through learnable scale weights to fuse organ structure information at different scales.

[0078] ;

[0079] in, The output features are those obtained after multi-scale weighted fusion. For scale indexing, 0 = original scale, 1 = 1 / 2 scale, 2 = 1 / 4 scale. For the first The learnable weight parameters corresponding to each scale, and These are the normalized scale weights; For the first The 3D convolutional features extracted by each scale branch are kept consistent with the upsampled size.

[0080] Finally, the fused features are sequentially processed through an MLP layer and a second Mamba process, and then residually connected with the original input features to output the enhanced 3D features (B, C, D, H, W). These features combine adaptive state space modeling capabilities, sequence-space joint features, and multi-scale detail information, providing more robust and accurate feature support for subsequent multi-scale interaction modules.

[0081] Step 3.2: Integrate the Adaptive Multi-Scale Feature Interaction Module (AMFI) into each Stage of the EM-Net encoder to replace the original multi-scale fusion component, thereby achieving efficient interaction and effective feature selection of multi-scale features and solving the technical defects of insufficient feature interaction, excessive redundant features, and blurred boundary segmentation of the original multi-scale fusion component.

[0082] Using multi-scale features from different levels of the encoder as input, features from different receptive fields are first extracted through multi-scale convolution. Then, a cross-scale attention mechanism is used to establish the correlation between high- and low-level features. Subsequently, fusion gating is used to adaptively filter the interacting features and suppress redundant information. Finally, the interactive enhancement features are output through residual connections, achieving efficient interaction of multi-scale features and enhancement of boundary details. The specific structure is as follows: Figure 4 As shown.

[0083] AMFI Enhanced Features Output by ADSSM As input, multi-scale feature extraction is first achieved through three parallel branches with different stride lengths (stride=1, stride=2, stride=4): the stride=1 branch maintains the original size, the stride=2 branch outputs 1 / 2 scale features, and the stride=4 branch outputs 1 / 4 scale features. The multi-scale parallel extraction formula is as follows:

[0084] ;

[0085] Where X is the AMFI input feature, , , These are features at the original scale, half scale, and quarter scale, respectively, used to capture the global structure, local contours, and subtle textures of organs.

[0086] To achieve scale alignment, the process will be... Downsampling , through Upsample back to the original size and with Multi-scale averaging fusion is performed to obtain the basic features of the fused multi-scale information. :

[0087] ;

[0088] The fused features enter the cross-scale attention module, where they undergo 1×1 convolution, BN, ReLU, and a second 1×1 convolution to strengthen the inter-scale correlation, outputting cross-scale enhanced features. Highlighting organ boundaries and key structures:

[0089] ;

[0090] In the formula, The features enhanced by cross-scale attention retain their dimensions (B, C, D, H, W), exhibiting stronger inter-scale correlations and more prominent key region features. BN(·) is a batch normalization operation used to standardize the feature distribution, accelerate model training, and avoid gradient vanishing. ReLU(·) is a linearly modified activation function used to introduce non-linearity, enhance the expressive power of the features, and suppress negative feature responses. Furthermore, both Conv(·) operations here are 3D convolution operations. The first convolution reduces the number of feature channels and extracts core features, while the second convolution restores the number of channels, ensuring compatibility with subsequent modules.

[0091] Then, by using 3D fusion gating to generate a spatial weight matrix of the same dimension as the features, adaptive weighting of key regions is achieved, strengthening organ boundaries and suppressing background redundant noise.

[0092] ;

[0093] In the above formula, The weight matrix is ​​a 3D spatial matrix, with dimensions corresponding to the input features. Consistent, i.e. (B,C,D,H,W), each element of this matrix takes values ​​in the range [0,1]. The closer the value is to 1, the more important the feature at the corresponding spatial location is; the closer the value is to 0, the more redundant the feature at the corresponding spatial location is. Sigmoid(·) is an S-type activation function used to map the convolutional features to the interval [0,1] to generate a spatial matrix that meets the weight requirements. This is a 1×1×1 3D convolution operation used for... Perform channel mapping to generate a weight matrix with the same dimensions as the input features.

[0094] Finally, the gated features are residually fused with the original input to output enhanced multi-scale features:

[0095] ;

[0096] Final output feature dimension Keep While remaining unchanged, this feature possesses stronger boundary representation and detail capture capabilities, significantly improving subsequent segmentation accuracy.

[0097] Step 3.3: Adaptive Multimodal Context Modulation (AMCM) module is used to replace the feature fusion module in the EM-Net decoder, which solves the problems of the original decoder having a single feature fusion method, insufficient utilization of multimodal features, and insufficient segmentation accuracy.

[0098] The original feature fusion module of the decoder is replaced with an adaptive multimodal context modulation module (AMCM). Taking the decoder's input features as input, it consists of two branches: multimodal feature extraction and global modulation parameter generation. The multimodal branch extracts features from different receptive fields in parallel through multi-scale 3D convolutions, and then obtains enhanced features through channel concatenation and interactive gating compression. The global modulation branch generates learnable channel-level modulation parameters through pooling and convolution. Finally, the modulation parameters and interactive enhanced features are subjected to an affine transformation to output optimized features, achieving accurate fusion and detail restoration of multimodal features. The specific structure is as follows: Figure 5 As shown.

[0099] This module achieves accurate multimodal feature fusion and global context calibration, addressing the issues of the original decoder's simplistic fusion method and insufficient utilization of modal information. AMCM uses decoder input features... The input is divided into a multimodal feature extraction branch and a global modulation parameter generation branch for parallel processing. In the multimodal extraction branch, the features are divided into three parallel modal channels, and differential complementary features are extracted through a unified convolutional structure.

[0100] ;

[0101] in, The output feature of the m-th modality is used to mine semantic and detailed information of organs from different dimensions. m is the modality index, with values ​​of 1, 2, and 3, corresponding to three parallel modal channels; GELU is the Gaussian error linear unit activation function. Compared with the traditional ReLU activation function, GELU can introduce a random regularization effect, improve the generalization ability of the model, and retain more feature information.

[0102] Then, the three modal feature channels are concatenated, and the modal information is compressed and fused through interactive gating to output multimodal interactive features. To achieve deep coupling of information between modes:

[0103] ;

[0104] In the above formula, This is a multimodal interaction fusion feature with dimensions (B, C, D, H, W). This feature achieves deep coupling of information from three modalities, integrating complementary semantics from different modalities. `Concat` is a channel concatenation operation that concatenates along the channel dimension. They are concatenated together; and the SiLU activation function is used to introduce non-linear factors, while also possessing self-normalization properties to enhance the expressive power of features.

[0105] In the global modulation branch, the same input feature X is fed into an adaptive 3D average pooling layer to compress the spatial dimension and obtain channel-level global semantic features, which are used to characterize the overall semantic distribution and provide global guidance for modality calibration.

[0106] ;

[0107] In the formula, The channel-level global semantic feature has a size of (B,C,1,1,1). This feature can characterize the overall semantic distribution of the input features, reflect the global structural information of the organ, and provide global guidance for subsequent modality calibration. The AdaptiveAvgPool3d(·) is an adaptive three-dimensional average pooling operation that can automatically adjust the pooling kernel size according to the size of the input features to ensure that the spatial dimension of the output features is (1,1,1), and only retain the global information of the channel dimension.

[0108] Then global features Feeding in a 1×1×1 convolution and sigmoid activation generates channel-level global modulation parameters, which are used to unify the distribution of multimodal features and enhance the effective channel weights.

[0109] ;

[0110] Finally, the modulation parameters and interaction features are subjected to channel-by-channel multiply-accumulate joint modulation:

[0111] ;

[0112] Final output features The dimensions (B, C, D, H, W) remain unchanged, ensuring full compatibility with subsequent modules of the decoder. The symbol ⊙ represents a channel-by-channel multiplication operation, enabling channel-by-channel calibration of modulation parameters and interactive features. Furthermore, residual connections are achieved using addition operations to mitigate feature information loss during modulation. This feature integrates global context and multimodal detail information, significantly improving the decoder's ability to reconstruct organ contours, edges, and fine structures. The structural diagram is shown in Figure 6.

[0113] Based on EM-Net, the above three major modules are improved to obtain an improved medical image segmentation architecture model dominated by three major modules: sequence modeling enhancement, multi-scale interaction, and multi-modal modulation.

[0114] Step 4: Model Training. By setting training hyperparameters, the improved model from Step 3 is trained using the training set from Step 2. Training is conducted on the AutoDL computing cloud platform. The test hardware platform environment is Python 3.10, CUDA 11.8, PyTorch 2.1.2, NVIDIA RTX4090D GPU with 24GB of VRAM, and 60GB of RAM. The training hyperparameters are set as follows: 500 training iterations, BatchSize set to 1, and SGD optimizer used.

[0115] Step 5: Model Validation. Set validation parameters to validate the model trained in Step 4. Validation results show that compared to EM-Net, the model's DSC is improved by 1.94%, and the HD index is reduced by 4.33.

[0116] Step 6: Using the validated model, perform feature extraction, multi-scale interaction, and multi-modal fusion processes on the BTCV abdominal multi-organ images to achieve accurate segmentation of abdominal organs.

[0117] To verify the innovative improvements of ADSSM, AMFI, and AMCM in this invention, an ablation experiment was conducted. The experiment was carried out in a unified environment based on Python 3.10, using the PyTorch 2.1.2 deep learning framework and CUDA 11.8. The SGD optimizer was used uniformly during the training process, with the initial learning rate set to 1e-4 and the batch size set to 1, for a total of 500 epochs.

[0118] Using EM-Net as the baseline model, the experimental results are shown in Table 1:

[0119]

[0120] Table 1. Model ablation experiments

[0121] Any deep learning experiment has randomness. To improve the credibility of the experimental results, the improved model is compared with other models. Evaluation metrics include DSC and HD indices. DSC is used to evaluate the segmentation overlap accuracy of a single organ, and HD is used to measure the maximum deviation between the predicted boundary and the true boundary. The combination of the two can comprehensively reflect the segmentation accuracy and boundary fit of the model, and can comprehensively evaluate the segmentation performance of 13 types of abdominal organs. The calculation formulas for DSC and HD are shown in (17) and (18):

[0122] ;

[0123] ;

[0124] DSC is a core metric for measuring the degree of overlap between the predicted region and the ground truth labeled region. In the formula, A represents the segmented region predicted by the model, and B represents the ground truth labeled region (gold standard). This represents the size of the intersection region between the two. The range of DSC values ​​is... The closer the value is to 1, the higher the segmentation accuracy.

[0125] HD is the Hausdorff distance, used to evaluate the matching degree of the segmentation boundary, where... HD represents the spatial Euclidean distance, reflecting the maximum boundary error between two sets of points in the region. The smaller the HD value, the more accurate the boundary segmentation and the smaller the deviation.

[0126] The improved EM-Net model was compared with other models, and the experimental results are shown in Table 2.

[0127] EM-Net 78.97 25.24 UNETR 68.88 49.55 Swin UNETR 75.34 48.45 3D Trans UNet 70.59 41.01 U-Mamba 77.98 30.21 Diff-UNet 69.79 62.46 This invention 82.59 20.92

[0128] Table 2. Model Comparison Experiment

[0129] Experimental data show that the improved EM-Net model proposed in this invention achieves a DSC score of 82.59%, an improvement of 3.62 percentage points compared to the original EM-Net model, while reducing the HD score to 20.92. Both metrics are the best among all compared models. These results directly demonstrate that this invention, through the synergistic improvement of ADSSM, AMFI, and AMCM modules, significantly enhances the model's segmentation accuracy and boundary localization ability for abdominal organs, fully validating the effectiveness of the technical solution of this invention.

[0130] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent transformations or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A 3D abdominal multi-organ image segmentation method based on an improved EM-Net, characterized in that: The steps include: constructing an improved network model based on the EM-Net model, training and validating the improved network model, and finally using the improved network model to segment BTCV abdominal multi-organ images; the operation mode of the improved network model includes: (1) The original feature enhancement module in the EM-Net encoder is replaced by the Adaptive Dynamic State Space Modulation (ADSSM) module, which is used to extract sequence features and spatial features from BTCV abdominal multi-organ image data to achieve adaptive fusion of sequence modeling and spatial features. (2) The adaptive multi-scale feature interaction module AMFI is integrated into each stage of the EM-Net encoder to replace the original multi-scale fusion component; (3) The feature fusion module in the EM-Net decoder is replaced by the adaptive multimodal context modulation module AMCM.

2. The 3D abdominal multi-organ image segmentation method based on improved EM-Net according to claim 1, characterized in that: The improved network model adopts an encoder-decoder structure in its overall architecture, and its specific process is as follows: First, the 3D abdominal CT body data is input into the network and mapped into shallow features by the Patch Embedding module. These features are then passed directly to the decoder via the Conv Block, while another path enters the STEM module to complete initial downsampling and feature enhancement. Subsequently, the data enters an encoder containing four cascaded stages. Each stage consists of an ADSSM module and an AMFI module connected in series. The ADSSM module enables adaptive fusion of sequence and spatial features, while the AMFI module performs multi-scale feature interaction enhancement. As the layers deepen, the feature resolution decreases and the number of channels increases. The multi-scale features output from each stage are sent to the corresponding layers of the decoder through skip connections. The decoder consists of an upsampling module and an AMCM module, which gradually restore the feature resolution through the Up-Sample Block. At the same time, the AMCM module performs multimodal context modulation and fusion on the skip connection features and the upsampled features.

3. Finally, the features output by the decoder are fused across layers with the shallow features passed from the Patch Embedding stage, and then the Conv Block module generates an organ segmentation mask of the same size as the input data to complete the 3D abdominal multi-organ segmentation task.

4. The 3D abdominal multi-organ image segmentation method based on improved EM-Net according to claim 1, characterized in that: The Adaptive Dynamic State Space Modulation (ADSSM) module is a parallel dual-branch architecture. It takes 3D features (B, C, D, H, W) as direct input, with one branch used for feature complexity evaluation and the other for main feature processing. The module's operation flow is as follows: First, the features are fed into the State Adaptor, and after 1×1×1 convolution, (B, 3, D, H, W) are obtained. Then, after Softmax normalization and global average pooling, a weight vector of (B, 3) is obtained. The optimal state space dimension d_state is adaptively selected. Next, the backbone input features are flattened into a sequence form (B, C, T), where T = D×H×W, and converted into a sequence modeling format (B, T, C); Then the data is fed into the Mamba branch and the standard attention branch in parallel for processing, with both paths maintaining the same dimensions (B, T, C). Then, the two outputs are fused by attention through learnable weights λ to restore the spatial feature shape (B, C, D, H, W); Then, multi-scale extraction is performed on the feature: feature learning is performed at the original scale, 1 / 2 scale, and 1 / 4 scale respectively, and then all scale features are uniformly upsampled back to the original size to obtain three sets of features of the same size (B, C, D, H, W); Subsequently, the three sets of multi-scale features are dynamically weighted and fused using learnable scale weights, and the fused features (B, C, D, H, W) are output. Finally, the fused features are sequentially processed through an MLP layer and a second Mamba process, and then residually connected with the original input features to output the enhanced features (B, C, D, H, W), providing more robust feature support for subsequent feature interactions.

5. A 3D abdominal multi-organ image segmentation method based on an improved EM-Net according to claim 1 or 3, characterized in that: The adaptive multi-scale feature interaction module AMFI takes the enhanced features (B, C, D, H, W) output by ADSSM as input. AMFI takes the enhanced features output by ADSSM as input and performs multi-scale feature extraction through three extractors with different time lengths. After upsampling the downsampled features to unify the size, it performs multi-scale averaging fusion, cross-scale attention interaction, fusion gating filtering, and then performs residual connection with the original input features.

6. A 3D abdominal multi-organ image segmentation method based on an improved EM-Net according to claim 3, characterized in that: The operation flow of the adaptive multi-scale feature interaction module AMFI is as follows: First, multi-scale feature extraction is performed by parallel processing using three extractors with different stride lengths of 1, 2, and 4: the extractor with stride=1 keeps the original feature size unchanged, while the extractors with stride=2 and stride=4 downsample the features by 1 / 2 and 1 / 4 respectively, outputting features of the corresponding scales (B, C, D, H, W), (B, C, D / 2, H / 2, W / 2), and (B, C, D / 4, H / 4, W / 4). Subsequently, an upsampling operation is performed on the two downsampled feature sets to restore them to the original feature size (B, C, D, H, W). These features are then fed into the Multi-scale Average Fusion module along with the original scale features to complete the initial fusion and output features with dimensions (B, C, D, H, W). The fused features enter the Cross-scale Attention module, where they are processed by 1×1 convolution, BN batch normalization, ReLU activation function and double 1×1 convolution to enhance cross-scale feature interaction. Next, the data is fed into the 3D Fusion Gate module, where spatial feature mapping is performed on the cross-scale fusion features through 1×1×1 3D convolution. Then, a 3D spatial weight matrix (B,C,D,H,W) with the same dimension as the features is generated by the Sigmoid activation function. The cross-scale features are then adaptively weighted and filtered according to spatial location, which strengthens the feature response of key locations of organ boundaries and fine structures, suppresses the interference of invalid features from background and redundant regions, and outputs the filtered effective 3D cross-scale features. Finally, the gated features are residually concatenated with the original input features, and after adaptive weighted fusion, the enhanced features (B, C, D, H, W) are output.

7. A 3D abdominal multi-organ image segmentation method based on an improved EM-Net according to claim 5, characterized in that: The adaptive multimodal context modulation module AMCM takes the enhanced features (B, C, D, H, W) output by AMFI as input and processes them in parallel through two paths: one path extracts multimodal features, and the other path generates global modulation parameters. The module's operation flow is as follows: First, the global modulation branch feeds the input features into the global pooling layer, where they are then processed by global average pooling to obtain... The channel description vector, then through two layers Convolution with Sigmoid activation generates a learnable scaling factor. With bias factor This is used to model channel-level global context information; Next, the multimodal feature extraction branch applies multi-scale 3D convolutional kernels to the input features in parallel: such as... , , Feature extraction is performed on different receptive fields to generate multiple sets of complementary feature maps, which are then concatenated through channels to obtain fused features. ; Then, this feature is sent to the interactive gating compression module, and then... Convolutional dimensionality reduction and channel attention weights generated using the Sigmoid algorithm are applied to adaptively filter and enhance multimodal features, outputting interactively enhanced features. ; Finally, the global modulation parameters , The interactive enhancement features are modulated by affine transformation and then output as optimized features via residual connection.

8. A 3D abdominal multi-organ image segmentation method based on an improved EM-Net according to claim 6, characterized in that: In the multimodal extraction branch, the input features are divided into three parallel modal channels. Each channel is processed by 1×1 convolution, BN batch normalization, GELU activation, and 1×1 convolution to extract features, ensuring that the output dimensions of each modality remain unchanged (B, C, D, H, W). Subsequently, the features of the three modalities are concatenated using Channel Concat to obtain concatenated features with dimensions (B, 3C, D, H, W). Then, the number of channels is compressed by the Interaction Gate module through 1×1 convolution, BN, and SiLU activation to output interactive features of (B, C, D, H, W). In the global modulation branch, the input features are processed by AdaptiveAvgPool 3D to compress the spatial dimension and extract channel-level global semantic features (B,C,1,1,1). The data is then fed into a 3D modulation generator, where global semantic features are mapped through 1×1×1 3D convolutions. The Sigmoid activation function is then used to generate channel-level 3D global modulation parameters (B,C,1,1,1), which assign an independent scaling factor to each feature channel. Finally, the interaction features and modulation parameters are modulated to produce Out = Input × Modulation + Interaction. After adjusting the feature dimensions through a 1×1 convolution of the output projection, the final multimodal fusion features (B, C, D, H, W) are output.

9. A 3D abdominal multi-organ image segmentation method based on an improved EM-Net according to claim 1, characterized in that: Before the step of constructing an improved network model based on the EM-Net model, there are steps of extracting the dataset and preprocessing the dataset.

Images