Abdomen multi-organ image segmentation method fusing multi-scale features

Abstract

The invention provides an abdomen multi-organ image segmentation method fusing multi-scale features, and the method comprises the steps: S1, constructing a space detail fusion module extraction network, which comprises a space attention module, a channel attention module, an up-sampling operation, a down-sampling operation and a feature smoothing operation; s2, designing an encoder, and connecting the encoder with the space detail fusion module; s3, transmitting a plurality of features with different resolutions obtained by the encoder to a decoder through jump connection for information recovery; and S4, arranging the abdominal multi-organ medical image data set to perform model training, and performing multi-organ segmentation prediction on the abdominal multi-organ medical image. According to the method, local and global information is fully utilized, multi-scale feature information is combined, the image feature extraction and expression capability of the model is improved, and the segmentation precision of multiple organs of the abdomen is improved.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and more specifically, to a method for segmenting abdominal multi-organ images by fusing multi-scale features. Background Technology

[0002] Multi-organ image segmentation is a key task in the field of medical image analysis, focusing on accurately distinguishing and delineating the boundaries of multiple organs from complex medical images such as CT, MRI, or ultrasound images. This technology is crucial for early disease diagnosis, treatment planning, and navigation during surgery. With the advancement of medical imaging technology, doctors can obtain increasingly detailed images of the human body's interior, but this also brings challenges to data processing and analysis. The development of multi-organ image segmentation technology can not only improve diagnostic accuracy but also provide richer information for clinical decision-making.

[0003] Before the advent of deep learning, multi-organ image segmentation primarily relied on a series of classic image processing techniques. These techniques were typically based on intuitive features of images such as grayscale, texture, and shape, evolving from simple methods based on thresholding and morphological operations to more complex model-based methods, such as those based on graph cuts and level sets. While these traditional methods could provide useful segmentation results in some cases, they often required extensive manual adjustments and domain knowledge, and were not robust enough to noise and variability in images. Furthermore, these methods often depended on simplified models and assumptions, making them ill-suited to the complexity and diversity of medical images.

[0004] The emergence of deep learning technology, especially the development of convolutional neural networks (CNNs), has revolutionized the field of medical image segmentation. Deep learning methods significantly improve segmentation accuracy and robustness by automatically learning features from massive amounts of data, rather than relying on manually designed image processing operations. These models are able to capture complex patterns and structures in images, including subtle differences between organs, resulting in more precise segmentation. Furthermore, deep learning models are adaptable to new data and variations, exhibiting better generalization capabilities, making them more effective in processing medical images from different patients, using different imaging devices, and under different pathological conditions.

[0005] Convolutional neural network (CNN)-based methods, such as U-Net and its various variants, have become mainstream in this field. The success of these methods stems from their encoder-decoder architecture and skip connections, designs that facilitate the integration of features at different scales. However, the small receptive domain of CNNs limits their ability to establish long-range dependencies between pixels, which is crucial for segmentation tasks.

[0006] In recent years, with the introduction of the visual Transformer, which effectively constructs long-range dependencies within image sequences through a multi-head self-attention mechanism, new opportunities have been brought to the field of multi-organ medical image segmentation. However, the pure Transformer model is insufficient in capturing low-level features, which are crucial for accurate boundary localization and detail restoration.

[0007] To overcome this challenge, researchers have begun exploring hybrid models that combine CNNs and Transformers, such as TransUNet, HiFormer, and TransCeption, aiming to leverage the strengths of both to achieve more accurate medical image segmentation. These models have already shown significant performance improvements in multi-organ medical image segmentation. Specifically, TransUNet, as a medical image segmentation model combining CNNs and Transformers, while contributing to the integration of local features and global context, employs a CNN-based decoder with skipped connections for simple upsampling feature recovery, failing to effectively handle multi-scale feature fusion. Furthermore, its performance remains insufficient in medical image segmentation tasks due to the limited number of annotated medical images.

[0008] While these Transformer-based methods have achieved state-of-the-art results on benchmarks, these hybrid models consume significant computational resources and are inefficient when processing large-scale datasets. Furthermore, the structure of abdominal multi-organs is relatively complex, with significant differences in size, shape, and structure among different organs, as well as variations in the number of organs in each category. Therefore, improving the accuracy of abdominal multi-organ image segmentation while reducing the number of model parameters remains a challenge for current technologies. Summary of the Invention

[0009] In view of this, this application provides a method for segmenting abdominal multi-organ images by fusing multi-scale features, so as to accurately locate and extract key features of organs and achieve accurate segmentation of multiple organs.

[0010] To achieve the above objectives, according to one aspect of this application, a method for segmenting abdominal multi-organ images by fusing multi-scale features is provided, comprising: S1: Construct a spatial detail fusion module, which includes a spatial attention module, a channel attention module, upsampling, downsampling, and feature smoothing operations. The spatial detail fusion module is used to fuse the spatial details of the feature map. S2: Design an encoder and connect the encoder to the spatial detail fusion module to form an abdominal multi-organ image segmentation model; S3: For multiple features of different resolutions obtained by the encoder, they are passed to the decoder through skip connections for information restoration; S4: Organize the abdominal multi-organ medical image dataset, train the model on the dataset, and then perform multi-organ segmentation prediction on the abdominal multi-organ medical images.

[0011] Furthermore, the construction of the spatial detail fusion module in step S1 specifically includes: S11: Input an abdominal medical image of size H*W*C into the spatial detail fusion module, where H represents the image length, W represents the image bandwidth, and C represents the number of channels; S12: The abdominal medical image passes through the spatial attention module and the channel attention module in sequence to simultaneously obtain local spatial information and global channel information; The spatial attention module uses max pooling to capture local salient features of the image and average pooling to capture global statistical features of the image, and fuses these features through a learnable convolutional layer to finally generate an attention map. Spatial attention is represented as: Where x is the input feature map with shape (N, C, H, W), where N is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively. This indicates that max pooling is applied to the i-th channel. This indicates that average pooling is applied to the i-th channel. This represents an operation that connects different features. This represents a 2D convolution operation, where kernel is the kernel size and padding is the padding length. The output channel count is half the input channel count. This represents the Sigmoid activation function; Channel attention is calculated using average pooling and max pooling to enhance the model's focus on important features and obtain global information in the spatial dimension. It can be represented as: Here, FC represents a fully connected layer, which is implemented using convolutional layers in the code. AvgPool and MaxPool represent average pooling and max pooling operations, respectively. This represents the Sigmoid activation function; FC consists of two convolutional layers, the first of which compresses the number of channels from C to... After passing through the ReLU activation function, it is then restored to C by a second convolutional layer; S13: Pass the feature maps of each layer that have passed through the attention module through a convolutional layer to smooth and unify the number of channels; S14: The feature maps with a uniform number of channels are upsampled or downsampled to unify the size of the feature maps, and then these feature maps are fused by Hada code product to obtain more feature details.

[0012] Furthermore, the specific steps in step S2 are as follows: S21: Construct a CNN branch as a branch of the parallel encoder; The CNN branch uses ResNet34 as its architecture, employing the first four residual blocks for feature extraction. Each residual block contains several convolutional layers and shortcut connections. The first layer consists of a 7×7 convolutional kernel with a stride of 2, used for initial feature extraction, followed by a 3×3 max-pooling layer with a stride of 2. This first layer halves the height and width of the input image, significantly reducing the number of parameters and computational cost. The second layer contains three residual blocks, each containing two 3×3 convolutional layers with a stride of 1. This second layer further extracts features and uses residual connections to facilitate gradient flow during training. The third layer contains four residual blocks, each containing two 3×3 convolutional layers with a stride of 1. In this third layer, the network begins to capture more complex features. The fourth layer contains six residual blocks, each containing two 3×3 convolutional layers with a stride of 1. This fourth layer further enhances the network's feature extraction capabilities. S22: Construct a Transformer branch as another branch of the parallel encoder; The Transformer branch consists of three modules, each consisting of a patch embedding module and two consecutive context-interacting SwingTransformer modules; The patch embedding module extracts features from the input image through a convolutional layer and controls downsampling by stride to reduce spatial dimensions, thus preparing for subsequent self-attention mechanism processing. Each SwingTransformer module consists of layer normalization, a multi-head self-attention module, residual connections, and a two-layer multilayer perceptron (MLP) with GELU nonlinearity; two consecutive Transformer blocks employ a conventional window-based multi-head self-attention (W-MSA) module and a moving window-based multi-head self-attention (SW-MSA) module, respectively; based on this window partitioning mechanism, a SwingTransformer block can be represented as: Where LN represents layer normalization, MLP represents multilayer perceptron, W-MSA represents multi-head self-attention based on a regular window, and SW-MSA represents multi-head self-attention module based on a moving window. This represents the output of the (l-1)th layer MLP. This indicates the output of W-MSA. This represents the output of the MLP at layer l. This represents the output of the SW-MSA at layer l+1. The output of the (l+1)th layer MLP is represented by the following method for calculating self-attention: in This indicates a query within a partial window. The key that represents a local window. A matrix representing the values ​​of a local window; This indicates the number of patches in a window. This represents the dimension of the query or the dimension of the key; and the values ​​in B are taken from the bias matrix. ; S23: Connect with the spatial detail integration module; Specifically, this involves concatenating the output feature maps of two branches on the same layer, and stacking feature maps of the same size from two branches together in the channel dimension, thereby increasing the number of channels in the feature map.

[0013] Furthermore, in step S3, the specific process of passing multiple features of different resolutions obtained by the encoder to the decoder for information restoration through skip connections is as follows: A decoder is constructed to restore information. The decoder includes an upsampling layer and a decoding bottleneck layer. The upsampling layer uses transposed convolutions to progressively increase the spatial resolution of the feature map. Each decoding bottleneck layer contains two sets of convolution, batch normalization, and ReLU activation function sequences. The decoding bottleneck layer is used to further refine and process the fused feature map to adapt it to the current resolution. The decoder processes the multi-scale feature maps from the encoder layer by layer. Through upsampling and feature fusion, it progressively restores the detailed information of the image and finally outputs a feature map with the same resolution as the original input image.

[0014] Furthermore, the specific steps of S4 are as follows: S41: Collect open-source abdominal multi-organ medical image datasets and preprocess the datasets; The preprocessing includes dividing the dataset into a training set and a test set, and performing data augmentation on the dataset during training; wherein, the data augmentation uses random rotation and flipping operations; S42: Design a loss function and an adaptive moment estimation (AdamW) optimizer with decoupled weight decay, use a cosine annealing strategy to adjust the learning rate, and then input the preprocessed data into the abdominal multi-organ image segmentation model for training. S43: After training is complete, obtain the trained model weight file; S44: Use the model weight file to perform multi-organ segmentation on the abdominal medical image to be predicted, and obtain the abdominal multi-organ prediction mask.

[0015] Furthermore, the expression for the loss function is as follows: in Indicates Dice loss; Represents cross-entropy loss; and These represent different weighting coefficients; Furthermore, the Dice loss is a metric function used to measure the similarity between two samples, and it is defined as follows: in and These represent the predicted value and the true label value at the i-th pixel, respectively.

[0016] Furthermore, the cross-entropy loss is a function used to measure the difference between the probability distribution predicted by the model and the true labels. For multi-class classification problems, it is defined as follows: Where M is the number of categories. This represents the value of the i-th class in the actual label, which can be either 0 or 1; This represents the probability that the model predicts for the i-th positive class.

[0017] The beneficial effects of this application are: 1. The parallel CNN and Transformer encoder designed in this application, compared with the standard Transformer, the Swin Transformer effectively reduces the computational complexity by introducing a hierarchical structure and window mechanism, while maintaining the ability to model long-distance dependencies. 2. The Swing Transformer introduces a hierarchical structure and window mechanism, which not only improves computational efficiency but also enhances the model's ability to capture global and local features in medical images. 3. In the skip connection part of the codec, an advanced multi-scale feature fusion strategy is adopted. By fusing multi-level feature maps, information from different branches is integrated. The feature maps obtained by the accompanying channel attention and spatial attention methods can retain and enhance useful features during the decoder upsampling process, while suppressing irrelevant information. 4. The feature fusion strategy of this application enhances the model's ability to capture multi-scale and multi-view features, and improves the richness of feature representation; 5. The method in this application makes full use of local and global information, combined with multi-scale feature information, to improve the model's ability to extract and represent image features, and improve the segmentation accuracy of multiple organs in the abdomen. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 : Flowchart of the abdominal multi-organ image segmentation method that integrates multi-scale features in this application; Figure 2 : A schematic diagram of the network model for abdominal multi-organ image segmentation that integrates multi-scale features in this application; Figure 3 : A structural schematic diagram of the spatial detail fusion module of this application; Figure 4 : A schematic diagram of the spatial attention module of the spatial detail fusion module in this application; Figure 5 : A schematic diagram of the channel attention module of the spatial detail fusion module in this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0021] like Figure 1 As shown, a method for segmenting abdominal multi-organ images by fusing multi-scale features includes the following steps: S1: Construct a spatial detail fusion module, including a spatial attention module, a channel attention module, upsampling, downsampling, and feature smoothing operations; S2: Design an encoder and connect it with the spatial detail fusion module to form an image segmentation model of multiple organs in the abdomen; S3: For multiple features of different resolutions obtained by the encoder, they are passed to the decoder through skip connections for information restoration; S4: Organize the abdominal multi-organ medical image dataset, train the model on the dataset, and then perform multi-organ segmentation prediction on the abdominal multi-organ medical images.

[0022] Furthermore, such as Figure 3 As shown, the steps for constructing the spatial detail fusion module in S1 are as follows: S11: Input an abdominal medical image of size H*W*C into the spatial detail fusion module; S12: Pass through the spatial attention module and the channel attention module to simultaneously obtain local spatial information and global channel information; like Figure 4 As shown, the spatial attention module uses max pooling and average pooling to capture global and local features of the image, and fuses these features through a learnable convolutional layer to finally generate an attention map. Spatial attention is represented as: Where x is the input feature map with shape (N, C, H, W), where N is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively. This indicates that max pooling is applied to the i-th channel. This indicates that average pooling is applied to the i-th channel. This represents an operation that connects different features. This represents a 2D convolution operation, where kernel is the kernel size and padding is the padding length. The output channel count is half the input channel count. This represents the Sigmoid activation function; like Figure 5 As shown, channel attention is calculated using average pooling and max pooling to enhance the model's focus on important features and obtain global information in the spatial dimension. It can be represented as: Where FC represents a fully connected layer (implemented as a convolutional layer in the code), and AvgPool and MaxPool represent average pooling and max pooling operations, respectively. This represents the Sigmoid activation function; FC consists of two convolutional layers, the first of which compresses the number of channels from C to... After passing through the ReLU activation function, it is then restored to C by a second convolutional layer.

[0023] S13: Then pass the feature maps of each layer that have passed through the attention module through a convolutional layer to smooth and unify the number of channels; S14: The feature maps with a uniform number of channels are upsampled or downsampled to unify their size, and then these feature maps are fused by Hada code product to obtain more feature details.

[0024] The decision to perform the upsampling or downsampling operation depends on the size of the feature map. If the size of the feature map is smaller than the target size, upsampling is performed; if the size of the feature map is larger than the target size, downsampling is performed; if the size of the feature map is equal to the target size, no processing is performed. The upsampling operation uses bilinear interpolation, and the downsampling operation uses adaptive average pooling.

[0025] like Figure 2As shown, the specific steps in S2 are as follows: S21: Construct a CNN branch as a branch of the parallel encoder; The CNN branch uses ResNet34 as its architecture, employing the first four residual blocks for feature extraction. Each residual block contains several convolutional layers and shortcut connections. The first layer consists of a 7×7 convolutional kernel with a stride of 2, used for initial feature extraction, followed by a 3×3 max-pooling layer with a stride of 2. This first layer halves the height and width of the input image, significantly reducing the number of parameters and computational cost. The second layer contains three residual blocks, each containing two 3×3 convolutional layers with a stride of 1. This second layer further extracts features and uses residual connections to facilitate gradient flow during training. The third layer contains four residual blocks, each containing two 3×3 convolutional layers with a stride of 1. In this third layer, the network begins to capture more complex features. The fourth layer contains six residual blocks, each containing two 3×3 convolutional layers with a stride of 1. This fourth layer further enhances the network's feature extraction capabilities. S22: Construct a Transformer branch as another branch of the parallel encoder; The Transformer branch consists of three modules, each consisting of a patch embedding module and two consecutive context-interacting SwingTransformer modules; The patch embedding module extracts features from the input image through a convolutional layer and controls downsampling by stride to reduce spatial dimensions, thus preparing for subsequent self-attention mechanism processing. Each SwingTransformer module consists of an LN (LayerNorm) layer, a multi-head self-attention module, residual connections, and a 2-layer MLP (Multilayer Perceptron) with GELU nonlinearity; two consecutive Transformer blocks employ a window-based multi-head self-attention (W-MSA) module and a moving-window-based multi-head self-attention (SW-MSA) module, respectively; based on this window partitioning mechanism, a SwingTransformer block can be represented as: Where LN represents layer normalization, MLP represents multilayer perceptron, W-MSA represents multi-head self-attention based on a regular window, and SW-MSA represents multi-head self-attention module based on a moving window. This represents the output of the (l-1)th layer MLP. This indicates the output of W-MSA. This represents the output of the MLP at layer l. This represents the output of the SW-MSA at layer l+1. The output of the (l+1)th layer MLP is represented by the following method for calculating self-attention: in This indicates a query within a partial window. The key that represents a local window. A matrix representing the values ​​of a local window; This indicates the number of patches in a window. This represents the dimension of the query or the dimension of the key; and the values ​​in B are taken from the bias matrix. ; The specific implementation of the window shifting operation is to move the features within each window to the right and down by half the window size; this operation enables each window to interact with its neighboring windows, thereby achieving global self-attention calculation.

[0026] Specifically, we randomly discard 50% of the neurons after each SwinTransformer to improve the model's generalization ability.

[0027] S23: Connect with the spatial detail integration module; Specifically, this involves concatenating the output feature maps of two branches on the same layer, and stacking feature maps of the same size from two branches together in the channel dimension, thereby increasing the number of channels in the feature map.

[0028] like Figure 2 As shown, in step S3, the specific process of passing multiple features of different resolutions obtained by the encoder to the decoder for information restoration through skip connections is as follows: A decoder is constructed to restore information. The decoder includes an upsampling layer and a decoding bottleneck layer. The upsampling layer uses transposed convolutions to gradually increase the spatial resolution of the feature map. Each decoding bottleneck layer contains two sets of convolution, batch normalization, and ReLU activation function sequences to further refine and process the fused feature map to adapt it to the current resolution. The decoder processes the multi-scale feature maps from the encoder layer by layer. Through upsampling and feature fusion, it gradually restores the detailed information of the image and finally outputs a feature map with the same resolution as the original input image.

[0029] Specifically, we also randomly dropped 50% of the neurons before outputting the feature map.

[0030] The specific steps in S4 are as follows: S41: Collect open-source abdominal multi-organ medical image datasets and preprocess the datasets; The preprocessing includes dividing the dataset into training and testing sets, and performing data augmentation on the dataset during training; the data augmentation uses random rotation and flipping operations.

[0031] S42: Design a loss function and an adaptive moment estimation (AdamW) optimizer with decoupled weight decay, and use a cosine annealing strategy to adjust the learning rate. Then, input the preprocessed data into the abdominal multi-organ image segmentation model for training. The expression for the loss function is as follows: in Indicates die loss; Represents cross-entropy loss; and These represent different weighting coefficients, which were set to 0.5 and 0.5 respectively in the experiment; The Dice loss is a metric function used to measure the similarity between two samples, and it is defined as follows: in and Let represent the predicted value and the true label value at the i-th pixel, respectively. Dice loss focuses on the overlap area between the predicted and true regions, is sensitive to pixels near the segmentation boundary, and helps improve the model's prediction accuracy for boundary locations.

[0032] The cross-entropy loss is a function used to measure the difference between the probability distribution predicted by the model and the true labels. For multi-class classification problems, it is defined as follows: Where M is the number of categories. This represents the value of the i-th class in the actual label, which can be either 0 or 1; This represents the probability that the model predicts for the i-th positive class.

[0033] The Adaptive Moment Estimator (AdamW) optimizer with decoupled weight decay is an improved version of the Adaptive Moment Estimator (Adam) optimizer. It adds support for weight decay to the original Adaptive Moment Estimator optimizer. This regularization technique can prevent the model from overfitting. The core idea behind using cosine annealing to adjust the learning rate is to use a large learning rate in the early stages of training to allow it to decrease rapidly, and then gradually reduce the learning rate as training progresses until it approaches zero. Cosine annealing helps the model converge quickly in the early stages of training, while refining the model parameters by reducing the learning rate in the later stages of training, thus avoiding overfitting.

[0034] S43: After training is complete, obtain the trained model weight file; S44: Use the model weight file to perform multi-organ segmentation on the abdominal medical image to be predicted, and obtain the abdominal multi-organ prediction mask; To test the accuracy and superiority of the segmentation method proposed in this embodiment for segmenting abdominal multi-organ medical images, the following experimental setup was used for further explanation: The experiments were conducted on a single NVIDIA A100-SXM 4090 GPU, using CUDA 11.7, PyTorch 1.21.1, and Python 3.9.19. The model was trained using an Adaptive Moment Estimation (AdamW) optimizer with decoupled weight decay. The input image size was uniformly set to 224 × 224, and the batch size was set to 32. The initial learning rate was 0.001, and cosine annealing was used to adjust the learning rate. The model was trained for a total of 250 epochs.

[0035] The segmentation performance is evaluated using the following metrics: Dice coefficient and Hd95. The Dice coefficient is a metric function that measures the similarity between two samples. It is commonly used to calculate the similarity between two samples (with a value range of [0, 1]), and its formula is as follows:

[0036] and These represent the true label and the predicted mask for segmentation, respectively. The closer the Dice coefficient is to 1, the higher the overlap between the model's prediction and the true label, meaning the better the model performance.

[0037] Hd95 is used to measure the distance between two sets. It is calculated based on the 95th percentile of the distance between boundary points in X and Y, with the aim of eliminating the influence of very small outlier subsets. The smaller the Hd95, the better the segmentation performance. First, the role of each component in the spatial detail fusion module of the segmentation method proposed in this embodiment is verified by ablation experiments. The encoders of parallel SwinTransformer and ResNet34 are used as the baseline network, and the proposed components are gradually added at the skip connections to prove their effectiveness. The ablation experiment results are shown in Table 1. In Table 1, the multi-scale feature fusion module refers to the structure remaining after removing the channel attention module and the spatial attention module from the spatial detail fusion module. Table 1. Ablation experiment results of each component of the network model.

[0038]

[0039] In this chart, ↑ indicates that the higher the value of the indicator, the better the performance; ↓ indicates that the lower the value of the indicator, the better the performance; bold text represents the optimal result.

[0040] Based on the data in Table 1, we can see that adding the channel attention module to the baseline model improved the DSC (Dice similarity coefficient) by 0.5%, but increased HD95 to 21.28. This indicates that the channel attention module alone cannot play a significant role; although it improves the model's ability to recognize features, it also leads to insufficient prediction accuracy in certain regions. To further verify the effectiveness of multi-scale feature fusion, we provide corresponding experimental data. By comparing the data in the first row (without multi-scale feature fusion) and the fourth row (with multi-scale feature fusion) of Table 1, we can clearly observe that after introducing the multi-scale feature fusion module, the number of model parameters is significantly reduced by half, while the DSC and HD95 evaluation metrics are improved by 0.4% and 1.9 mm, respectively. This improvement further confirms the potential of multi-scale feature fusion in improving model performance. In addition, based on multi-scale feature fusion, we introduced spatial and channel attention mechanisms to form the proposed spatial detail fusion module. The data in the last row of Table 1 clearly shows that this fusion method not only reduces the number of model parameters but also achieves the best segmentation effect. This result demonstrates that the spatial detail fusion module can not only acquire feature information from the global context, but also capture local spatial information through two attention mechanisms, extracting more useful details and thus significantly improving the segmentation accuracy of the network model.

[0041] In summary, our proposed spatial detail fusion module has a significant positive impact on the segmentation performance of the network model. This module enables the model to more effectively integrate feature information at different scales, improving the accuracy and efficiency of the segmentation task.

[0042] Secondly, in order to explore the contribution of different layers of convolutional neural network backbones, a series of ResNet variants were used as convolutional architectures; ablation experiments were carried out on the abdominal multi-organ medical image dataset, and the results are shown in Table 2. ResNet18, ResNet34, ResNet50, and ResNet101 were used as convolutional neural network backbones, respectively. Table 2 Ablation experiments of ResNet backbones with different layers.

[0043]

[0044] In this chart, ↑ indicates that the higher the value of the indicator, the better the performance; ↓ indicates that the lower the value of the indicator, the better the performance; bold text represents the optimal result.

[0045] As can be seen from Table 2, a larger convolutional neural network backbone does not necessarily bring performance improvements (see the fourth row in the table), which leads us to use the ResNet34 architecture as the default convolutional neural network architecture.

[0046] Furthermore, on an abdominal multi-organ medical image dataset, the proposed network model was compared with the segmentation results of existing state-of-the-art models such as TransUnet, Swin-Unet, Hiformer, and TransCeption. The results are shown in Table 3, with bold text indicating the best results for each category. It can be seen that the proposed network model outperforms other single-arm methods on the abdominal multi-organ medical image dataset, achieving the highest Dice score of 84.64% and the lowest HD95 score of 12.95mm. Specifically, the network model achieved high Dice scores of 89.25%, 73.16%, 88.22%, 85.11%, 95.26%, 71.6%, 91.78%, and 82.73% for the segmentation of different anatomical structures such as the aorta, gallbladder, left kidney, right kidney, liver, pancreas, spleen, and stomach, respectively, demonstrating its superiority in segmentation of various organs. In particular, it showed significant improvement in DSC segmentation of the right kidney and pancreas, improving by 1.27% and 1.81% respectively compared to the previous best results.

[0047] Table 3. Comparison of segmentation results of the network model with other models on abdominal multi-organ medical images.

[0048]

[0049] In this chart, ↑ indicates that the higher the value of the indicator, the better the performance; ↓ indicates that the lower the value of the indicator, the better the performance; bold text represents the optimal result.

[0050] The parallel CNN and Transformer encoder designed in this application, compared with the standard Transformer, effectively reduces computational complexity by introducing a hierarchical structure and windowing mechanism, while maintaining the ability to model long-range dependencies. Furthermore, the feature fusion strategy in this application enhances the model's ability to capture multi-scale and multi-view features, improving the richness of feature representation; and fully utilizes local and global information, combining multi-scale feature information to improve the model's ability to extract and represent image features, thereby improving the segmentation accuracy of abdominal multi-organs.

[0051] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for abdominal multi-organ image segmentation integrating multi-scale features, characterized in that: include: S1: construct a spatial detail fusion module, the spatial detail fusion module includes a spatial attention module, a channel attention module, upsampling, downsampling and feature smoothing operations, and the spatial detail fusion module is used to fuse the spatial details of the feature map; S2: designing an encoder, and connecting the encoder with a spatial detail fusion module to form an abdominal multi-organ image segmentation model; S3: The features of multiple resolutions obtained by the encoder are transmitted to the decoder through skip connections for information restoration; S4: Organize a dataset of abdominal multi-organ medical images, perform model training on the dataset, and then perform multi-organ segmentation prediction on the abdominal multi-organ medical images.

2. The abdominal multi-organ image segmentation method integrating multi-scale features as claimed in claim 1, characterized in that: The step S1 of constructing the spatial detail fusion module specifically includes: S11: inputting an abdominal medical image of size H*W*C into the spatial detail fusion module, where H is the image length, W is the image width, and C is the number of channels; S12: passing the abdominal medical image through a spatial attention module and a channel attention module in sequence to simultaneously obtain local spatial information and global channel information; The spatial attention module uses maximum pooling to capture local salient features of the image and average pooling to capture global statistical features of the image, and fuses these features through a learnable convolutional layer to finally generate an attention map; Spatial attention is expressed as: Where x is the input feature map, and its shape is (N,C,H,W), where N is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively. Indicates that the maximum pooling operation is applied on the i-th channel, Indicates that an average pooling operation is applied on the i-th channel. represents the operation of connecting different features, Represents a 2D convolution operation, kernel is the convolution kernel size, padding is the convolution padding length, and the number of output channels is half of the number of input channels. Represents the Sigmoid activation function; Channel attention is calculated by average pooling and maximum pooling to enhance the model's attention to important features and obtain global information in the spatial dimension, which can be expressed as: Among them, FC represents the fully connected layer, which is implemented by the convolutional layer in the code. AvgPool and MaxPool represent the average pooling and maximum pooling operations respectively. represents the Sigmoid activation function; FC includes two convolutional layers, the first convolutional layer compresses the number of channels from C to , after the ReLU activation function, it is restored to C through the second convolutional layer; S13: Pass the feature maps of each layer through the attention module through the convolution layer to smooth the number of channels; S14: Upsample or downsample the feature maps with the same number of channels to unify the size of the feature maps, and then fuse the feature maps through the Hadamard product to obtain more feature details.

3. The abdominal multi-organ image segmentation method integrating multi-scale features as claimed in claim 2, characterized in that: In step S2, the specific steps are as follows: S21: Construct a convolutional neural network branch as a branch of the parallel encoder; The convolutional neural network branch uses ResNet34 as the basic architecture and uses the first four residual blocks for feature extraction; each residual block contains several convolutional layers and shortcut connections; the first layer is a 7×7 convolution kernel with a step size of 2, which is used to initially extract features, followed by a 3×3 maximum pooling layer with a step size of 2. The first layer halves the height and width of the input image, while significantly reducing the number of parameters and the amount of calculation; the second layer contains three residual blocks, each of which contains two 3×3 convolutional layers with a step size of 1. The second layer further extracts features and helps the gradient flow during training through residual connections; the third layer contains four residual blocks, each of which also contains two 3×3 convolutional layers with a step size of 1. In the third layer, the network begins to capture more complex features; the fourth layer contains six residual blocks, each of which contains two 3×3 convolutional layers with a step size of 1. The fourth layer further enhances the network's ability to extract features; S22: Build the Transformer branch as another branch of the parallel encoder; The Transformer branch consists of three modules, each of which consists of a patch embedding module and two consecutive contextual interaction SwinTransformer modules; The patch embedding module extracts features from the input image through a convolutional layer, downsamples by step size control, and reduces the spatial dimension to prepare for subsequent self-attention mechanism processing; Each SwinTransformer module consists of layer normalization, multi-head self-attention module, residual connection and 2-layer multilayer perceptron with GELU nonlinearity; two consecutive Transformer blocks adopt multi-head self-attention module based on regular window and multi-head self-attention module based on moving window respectively; based on such window partitioning mechanism, SwinTransformer block can be expressed as: Among them, LN represents layer normalization, MLP represents multi-layer perceptron, W-MSA represents multi-head self-attention based on regular window, and SW-MSA represents multi-head self-attention module based on moving window. represents the output of the MLP at layer l-1, represents the output of W-MSA, represents the output of the MLP at layer l, represents the output of the SW-MSA at layer l+1, represents the output of the MLP at layer l+1, and the self-attention is calculated as follows: in Represents a query of a local window, A key representing a local window. A matrix of values ​​representing a local window; represents the number of patches in a window, represents the dimension of the query or the dimension of the key; and the values ​​in B are taken from the bias matrix ; S23: connect with the spatial detail fusion module; Specifically, it includes concatenating the output feature maps of the two branches on the same layer, and stacking the feature maps of the same size from the two branches together in the channel dimension, thereby increasing the number of channels of the feature map.

4. The abdominal multi-organ image segmentation method integrating multi-scale features as claimed in claim 3, characterized in that: In step S3, the specific process of transmitting the features of multiple different resolutions obtained by the encoder to the decoder through the skip connection for information restoration is as follows: A decoder is constructed to restore information, and the decoder includes an upsampling layer and a decoding bottleneck layer; wherein the upsampling layer uses transposed convolution to gradually increase the spatial resolution of the feature map; each decoding bottleneck layer contains two sets of convolution, batch normalization and ReLU activation function sequences, and the decoding bottleneck layer is used to further refine and process the fused feature map to adapt it to the current resolution; the decoder processes the multi-scale feature map from the encoder layer by layer, and gradually restores the detailed information of the image through upsampling and feature fusion, and finally outputs a feature map with the same resolution as the original input image.

5. The abdominal multi-organ image segmentation method integrating multi-scale features as claimed in claim 4, characterized in that: The specific steps of S4 are as follows: S41: Collect open-source abdominal multi-organ medical image datasets and preprocess the datasets; The preprocessing includes dividing the data set into a training set and a test set, and performing data enhancement on the data set during training; wherein the data enhancement adopts random rotation and flipping operations; S42: designing a loss function and an adaptive moment estimation optimizer with decoupled weight decay, using a cosine annealing strategy to adjust the learning rate, and then inputting the preprocessed data into the abdominal multi-organ image segmentation model for training; S43: After the training is completed, obtain the trained model weight file; S44: Use the model weight file to perform multi-organ segmentation on the abdominal medical image to be predicted to obtain an abdominal multi-organ prediction mask.

6. The abdominal multi-organ image segmentation method integrating multi-scale features as claimed in claim 5, characterized in that: The expression of the loss function is as follows: in represents Dice loss; represents the cross entropy loss; and They represent different weight coefficients respectively.

7. The abdominal multi-organ image segmentation method integrating multi-scale features as claimed in claim 6, characterized in that: The Dice loss is a metric function used to measure the similarity between two samples, which is defined as: in and Represent the predicted result and the true label value at the i-th pixel respectively.

8. The abdominal multi-organ image segmentation method integrating multi-scale features as claimed in claim 7, characterized in that: The cross entropy loss is a function used to measure the difference between the probability distribution predicted by the model and the true label. For multi-classification problems, it is defined as: Where M is the number of categories, Represents the value of the i-th category in the true label, which is 0 or 1; It represents the probability that the model predicts the positive class of the i-th category.

Images