Medical image segmentation method based on fusion of 3D depthwise separable convolution and EAM module

Abstract

The present invention belongs to the field of image segmentation technology, and specifically relates to a medical image segmentation method based on 3D depth-separable convolution and EAM module fusion. The method of the present invention adopts the EAM-3D module to extract feature information, and enhances the feature expression ability by calculating the attention weights in three different directions of depth, width and height. Compared with other attention mechanisms, the multi-directional attention calculation of the EAM-3D module is more comprehensive and effective. Other attention mechanisms usually only perform feature weighting in a certain dimension, while the EAM-3D module performs attention calculations in multiple directions, which makes it have significant advantages in processing high-resolution medical 3D images. The method of the present invention generates feature maps of different resolutions at each stage of the decoder, captures multi-scale information of the feature maps from coarse to fine and from global to local, and finally integrates multi-level information by fusing feature maps at different stages, thereby improving the accuracy and robustness of segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a medical image segmentation method based on 3D depth separable convolution and EAM module fusion. Background Art

[0002] In recent years, deep learning technology has made significant progress in the field of medical image processing, especially in medical image segmentation. Medical image segmentation is one of the core tasks in medical image processing. It provides key support for clinical diagnosis, treatment planning, surgical navigation and disease monitoring by segmenting different tissues, organs and lesion areas in medical images. However, traditional image segmentation methods often face the problem of balancing resolution and computational complexity when processing complex three-dimensional medical images, and cannot fully extract the details and structural information in the image. Therefore, how to efficiently process and analyze three-dimensional medical images has become the focus of researchers. With the rapid development of deep learning, deep learning models based on convolutional neural networks (CNNs) have shown excellent performance in image processing and analysis. In particular, the U-Net model and its variants, such as 3D U-Net and V-Net, have achieved remarkable results in medical image segmentation tasks. U-Net adopts an encoder-decoder structure. Through downsampling and upsampling operations, the model can capture multi-scale image features, thereby extracting detailed segmentation results while maintaining global context information. However, although these models have solved the problem of balancing resolution and computation to a certain extent, they still have shortcomings when dealing with multi-scale feature fusion and detail information extraction. 3D depth-separable convolution greatly reduces the number of parameters and computation by decomposing traditional convolution into depth-wise convolution and point-wise convolution, while maintaining good feature extraction capabilities. The efficient localized attention module enhances the model's ability to focus on important features by performing attention calculations in three dimensions: depth, width, and height, thereby improving segmentation accuracy.

[0003] With the development of deep learning, convolutional neural networks (CNNs) have gradually become the mainstream method for medical image segmentation. However, traditional 3D convolution operations require a lot of computing resources and storage space, especially when processing high-resolution 3D medical images. The computational complexity is high and the training time is long, making it difficult to deploy and apply in resource-limited environments. Many traditional deep learning models cannot fully utilize the multi-scale feature information in the image when processing medical images. This leads to poor performance of the model when processing complex images with different scale features. For example, the simultaneous segmentation of small lesions and large organs requires the model to have the ability to capture features of different scales, but traditional models often lack this. Attention mechanisms have been widely used in deep learning research in recent years and can significantly improve the model's ability to capture key information. However, the application of traditional attention mechanisms in 3D medical image segmentation is not sufficient, and it is impossible to effectively perform attention calculations in the three dimensions of depth, width, and height at the same time, thus limiting the performance improvement of the model. In the training process of deep neural networks, gradient vanishing and gradient exploding are common problems, especially in models with large depth. Traditional deep learning models are prone to gradient vanishing or exploding when processing multi-layer 3D convolution operations, resulting in an unstable training process, slow convergence, or even failure to converge. Summary of the invention

[0004] The purpose of the present invention is to solve the above problems and propose a medical image segmentation method based on the fusion of 3D deep separable convolution and EAM module. This method can more accurately extract and process multi-scale features in medical images and improve the accuracy of segmentation by introducing 3D deep separable convolution and efficient localized attention module in deep learning technology.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solution: a medical image segmentation method based on 3D depth separable convolution and EAM module fusion, comprising the following steps:

[0006] S1. First, the medical image is input into the first stage of the encoder, and the input channels are increased from 1 to 16 using depthwise separable convolution. Then, the image is normalized using BatchNorm3d to improve training stability. Then, the LeakyReLU activation function is used to provide linear transformation, and finally, the efficient localization attention mechanism EAM is introduced to weight the features to enhance important features.

[0007] S2, repeat three times the depth separable convolution, batch normalization, LeakyReLU and EAM attention processing, and output different feature maps from low to high resolution of medical images, which are used as the feature maps of stage 2, stage 3, and stage 4 respectively. The number of channels in stage 2 increases from 16 to 32, the number of channels in stage 3 increases from 32 to 64, and the number of channels in stage 4 increases from 64 to 128;

[0008] S3, feed the feature map of stage 4 in the encoder as the input image to the first stage of the decoder, whose input channel number is 128, and restore the feature map through multiple layers of depth-separable convolutional layers, BatchNorm3d layers, LeakyReLU activation layers and EfficientLocalization Attention 3D modules, whose output channel number is 256;

[0009] S4, the output feature map of stage 1 in the decoder is upsampled to expand its spatial resolution and concatenated with the feature map from stage 3 in the encoder. It is then restored by combining high-level and mid-level feature information through multiple layers of depth-separable convolutional layers, BatchNorm3d layers, LeakyReLU activation layers, and Efficient Localization Attention 3D modules to serve as the output feature map of stage 2 in the decoder;

[0010] S5, repeat S4, splice the output feature map of stage 2 in the decoder with the feature map of stage 2 in the encoder by upsampling, to obtain the output feature map of stage 3 of the decoder; splice the output feature map of stage 3 in the decoder with the feature map of stage 1 in the encoder by upsampling, to obtain the output feature map of stage 4 of the decoder;

[0011] S6, perform 1×1×1 convolution operations on the feature maps generated by the four different stages of the decoder in S3, S4, and S5, generate corresponding maps at different scales, compare the prediction results at multiple scales with the actual segmentation labels, calculate the loss function and update the model parameters;

[0012] S7. Concatenate or sum the feature maps of all scales to obtain the final multi-scale feature map, and finally output the pixel-level segmentation prediction by applying a linear projection layer.

[0013] Furthermore, as a preferred technical solution of the present invention, the encoder part in S1 includes a total of 4 stages, each of which contains multi-layer feature extraction and self-attention interaction mechanism, which captures features of different scales and enhances local feature representation, thereby improving the efficiency of feature extraction and increasing the accuracy of model segmentation.

[0014] As a further preferred technical solution of the present invention, 3D depth separable convolution greatly reduces the number of parameters and computational complexity by decomposing the traditional convolution into channel-by-channel convolution and point convolution. In the traditional convolution operation, the input feature map is operated through multiple convolution kernels to obtain the output feature map. The size of the input feature map is H×W×D×C in , the convolution kernel size is K×K×K×C in ×C out , the output feature map size is H×W×D×C out Its computational complexity is H×W×D×K×K×K×C in ×C out . 3D depth separable convolution is decomposed into two steps. The first step is to perform channel-by-channel convolution, and perform convolution operation on each input channel separately (each input channel corresponds to a convolution kernel). The number of output channels is the same as the number of input channels. The size of the input feature map and the size of the output feature map are the same as the size of the traditional convolution input feature map. The size of the convolution kernel is K×K×K×C in , its computational complexity is H×W×D×K×K×K×C in The second step is point convolution, using a 1×1×1 3D convolution kernel to linearly combine the output of the channel-by-channel convolution to obtain the final output feature map. The input feature map size is the same as the traditional convolution input feature map size, and the 3D convolution kernel size is 1×1×1×C in ×C out , the output feature map size is H×W×D×C out The computational complexity of point convolution is H×W×D×C in ×C out Combining the above two steps, we can see that the total computational complexity of 3D depthwise separable convolution is H×W×D×K×K×K×C in +H×W×D×C in ×C out Since the computational complexity of traditional convolution is H×W×D×K×K×K×C in ×C out , the computational complexity of 3D depthwise separable convolution is H×W×D×K×K×K×C in +H×W×D×C in ×C out , when the convolution kernel is large or the number of output channels is large, H×W×D×C in ×(K×K×K×C out )>H×W×D×C in ×(K×K×K+C out ), which greatly reduces the number of parameters and computational complexity.

[0015] Further as the preferred technical solution of the present invention, the EAM-3D module in S1 performs weighted processing on features of different scales by calculating attention weights in three dimensions of depth, width and height respectively, and enhances attention to important features. First, a 3D input feature map is received, and the shape is (B, C, D, H, W), where B is the batch size, C is the number of channels, and D, H, W are depth, height and width respectively. Subsequently, the attention weights are evaluated for three aspects of the depth dimension, height dimension and width dimension respectively. The first step processes the depth dimension, firstly the input feature map is averaged in the width and height dimensions, and a feature map of shape (B, C, D, 1, 1) is obtained, and then a convolution operation is performed using a depth separable convolution, and then group normalization and Sigmoid activation function are performed to obtain the attention weight of the depth dimension. According to this method, the attention weight calculation of the second step processing the height dimension and the third step processing the width dimension is performed, wherein the shape obtained by the height dimension is (B, C, D, H, 1), and the shape obtained by the width dimension is (B, C, D, 1, W). Finally, the attention weights calculated in the depth, width, and height dimensions are multiplied element-by-element with the input feature map to obtain the weighted feature map. The EAM-3D model calculates the attention weights in the three dimensions of depth, width, and height, so that the module can better capture and fuse multi-scale features and improve feature expression capabilities.

[0016] Further, as a preferred technical solution of the present invention, the encoder in S1 is divided into 4 stages, each of which includes multiple depth-separable convolutional layers, batch normalization layers, LeakyReLU activation layers and EAM-3D modules. In the initial stage of the network (i.e., stage 1), the resolution of the input feature map is high, but the number of channels of the feature map is small. At this time, the information in the feature map is relatively primitive. In order to better extract low-level features, such as edges, colors and other information, fewer repeated operations are required to extract basic features and ensure the efficiency of network calculations. If too many convolution operations are performed, the amount of calculation will increase significantly, especially when the resolution of the input feature map is high. Therefore, this model chooses to perform two convolution operations in stage 1 to balance the computational complexity. As the network goes deeper, the resolution of the feature map gradually decreases and the number of channels gradually increases. At this time, the information contained in the feature map is more abstract and rich. In order to fully extract these advanced features, more repeated operations are required to enhance the feature representation capability. Since the resolution of the feature map gradually decreases and the amount of calculation is relatively reduced, this model chooses to perform three convolution operations in the second, third and fourth stages to fully extract and process the features while controlling the computational complexity within an acceptable range.

[0017] Further, as a preferred technical solution of the present invention, the decoder in S2 is divided into 4 stages, each of which includes multiple upsampling convolution modules, depth-wise separable convolution layers and EAM modules. The upsampling operation restores the spatial resolution of the feature map layer by layer, and then the depth-wise separable convolution layer is used to further extract and optimize features, and finally the EAM module is used to enhance the expressiveness of the feature map.

[0018] Further, as a preferred technical solution of the present invention, in the initial stage of the encoder, the resolution of its feature map is low but the number of channels is high. At this time, the information in the feature map is highly abstract and rich. In order to reconstruct the resolution of the feature map, more convolution operations are required to fully extract features. Therefore, three convolution operations are performed in the first stage of the decoder to maximize the ability of feature representation and extract more useful information. In the second stage of the decoder, the resolution of the input feature map has increased after the last sample, and the number of channels has decreased. At this time, the information in the feature map is still complex and rich, and it is necessary to continue to perform three convolution operations to process and extract features. The convolution operation at this stage can further enhance the feature representation capability and fuse the feature information from the encoder, thereby improving the overall performance of the model. In the third stage of the decoder, the resolution of the input feature map is further increased and the number of channels is further reduced. The information in the feature map begins to become more specific and detailed, but it is still necessary to perform three convolution operations to fully process and extract features. The convolution operation at this stage can ensure that in the high-resolution feature map after upsampling, a high feature representation capability is maintained, and the feature information transmitted by the encoder is further integrated. In the fourth stage of the decoder, the resolution of the input feature map is the highest and the number of channels is the least. At this point, the information in the feature map is the most specific and detailed, which can reduce the number of convolution operations. Therefore, two convolution operations are performed at this stage to balance the computational complexity and feature extraction capabilities. Through these convolution operations, it is ensured that sufficient detail information is extracted in the high-resolution feature map while maintaining high computational efficiency.

[0019] Further, as a preferred technical solution of the present invention, the decoder in S6 generates feature maps of different resolutions and different abstract levels at different stages, where map1 is the feature map with the lowest resolution, containing the most abstract and high-level information; map2 is the feature map with a lower resolution, containing mid-level abstract information; map3 is the feature map with a higher resolution, containing more specific detail information; map4 is the feature map with the highest resolution, containing the most specific and detailed information. The feature maps of different resolutions generated in the four stages realize multi-scale feature prediction, which not only retains details but also extracts global information.

[0020] Further, as a preferred technical solution of the present invention, each feature map uses 3D convolution to adjust the number of channels of the feature map to 1, and then uses an upsampling function to enlarge the feature map to the original resolution, so that its final output is consistent with the size of the input image, and finally uses the Sigmoid activation function architecture to map the output to the [0,1] interval. By fusing multi-scale feature maps, the spatial resolution and details of the image can be better restored. The multi-scale output makes the model more robust to targets of different sizes and shapes, and improves the generalization ability of the segmentation task.

[0021] Further, as a preferred technical solution of the present invention, the loss function between each output feature map and the corresponding true value image is calculated using Dice loss. Dice loss is a commonly used loss function for evaluating the performance of binary classification segmentation tasks. Its formula is:

[0022] Where A is the predicted segmentation result and B is the actual segmentation annotation. In order to adapt to the training process of the neural network, the formula is usually converted to:

[0023]

[0024] where p i is the predicted value of the ith pixel, g i is the true value of the i-th pixel. Compared with cross entropy loss, Dice loss can better handle the problem of imbalance between positive and negative samples. And Dice loss directly reflects the quality of the segmentation result, which is consistent with the evaluation indicators of the segmentation task (such as Dice coefficient), and helps to improve the segmentation performance of the model.

[0025] The medical image segmentation method based on 3D depth separable convolution and EAM module fusion described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0026] (1) The present invention uses 3D depth-separable convolution to decompose the traditional convolution into two steps: first, channel-by-channel convolution and then point-by-point convolution. In the channel-by-channel convolution, each input channel is convolved separately without mixing with other channels. Then, in the 3D point-by-point convolution, the output of the channel-by-channel convolution is mixed with the channel using a 1×1×1 convolution kernel to adjust the number of output channels. The traditional 3D convolution operation requires a fully connected convolution operation between each input channel and the output channel, which results in a very large amount of calculation and parameters. After using 3D depth-separable convolution, the amount of calculation is significantly reduced. Due to the reduction in the amount of calculation and the amount of parameters, the memory required for 3D depth-separable convolution during training and inference is significantly reduced. This is particularly useful for high-resolution liver image processing, because high-resolution images themselves require a large amount of memory storage. Reducing memory usage not only makes it more feasible to process high-resolution images, but also helps to achieve efficient medical image processing on resource-limited embedded systems or mobile devices. In addition, the reduction in the number of parameters helps to reduce the complexity of the model and prevent overfitting, especially when there is less data or the data is noisy. This is particularly important for medical image processing, where medical image data are usually scarce and expensive to acquire.

[0027] (2) The present invention addresses the problems of subtle differences in grayscale and texture between the liver and surrounding organs and tissues, limited liver image data, and an imbalance in the number of samples of different categories. The method uses the EAM-3D module to extract feature information and enhances the feature expression capability by calculating attention weights in three different directions: depth, width, and height. Compared with other attention mechanisms, the multi-directional attention calculation of the EAM-3D module is more comprehensive and effective. Other attention mechanisms usually only perform feature weighting in a certain dimension, while the EAM-3D module performs attention calculations in multiple directions, giving it a significant advantage in processing high-resolution medical 3D images.

[0028] (3) The present invention aims at the problem that the liver and its surrounding tissues have a complex anatomical structure, unclear boundaries, great difficulty in segmentation, and different scale features have inconsistent focus. Small-scale features help capture details, while large-scale features help understand the overall structure. The patented method generates feature maps of different resolutions at each stage of the decoder, and captures multi-scale information of feature maps from coarse to fine and from global to local. Finally, by fusing feature maps at different stages, multi-level information can be better integrated to improve the accuracy and robustness of segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the method steps of the present invention;

[0030] Figure 2 This is a structural diagram of the depthwise separable convolution and EAM-3D module of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0032] like Figure 1-2 As shown, the medical image segmentation method based on the fusion of 3D deep separable convolution and EAM module includes the following steps:

[0033] S1: First, the medical image is input into the first stage of the encoder. In this stage, the input channels are increased from 1 to 16 using depthwise separable convolution. Next, the feature map is normalized by batch normalization (BatchNorm3d) to improve the stability of training. Subsequently, the LeakyReLU activation function is used for nonlinear transformation to enhance the expressiveness of the model. Finally, the efficient localization attention mechanism (EAM) is introduced to weight the extracted features to enhance important features.

[0034] S2: In the second stage of the encoder, depthwise separable convolution, batch normalization, LeakyReLU, and EAM attention are repeated three times. In this way, the resolution of the medical image is gradually reduced while extracting feature maps of different scales. The number of feature map channels in the second stage increases from 16 to 32, from 32 to 64 in the third stage, and from 64 to 128 in the fourth stage. Each stage enhances the feature representation capability through the above processing steps.

[0035] S3: The feature map output by the encoder stage 4 is passed as input to the decoder stage 1. In this stage, the number of input channels is 128, and the feature map is processed by multiple layers of depthwise separable convolution, BatchNorm3d layer, LeakyReLU activation layer and Efficient LocalizationAttention 3D module, and the number of output channels increases to 256.

[0036] S4: In the second stage of the decoder, the output feature map of the first stage is upsampled to expand its spatial resolution and concatenated with the feature map of the encoder stage 3. Subsequently, after multiple layers of depthwise separable convolution, BatchNorm3d layer, LeakyReLU activation layer and Efficient LocalizationAttention 3D module, the high-level and mid-level feature information are combined to restore the details of the feature map.

[0037] S5: Repeat the above steps. In the decoder stage 3, the output feature map of stage 2 is concatenated with the feature map of stage 2 of the encoder through upsampling. Similarly, in the decoder stage 4, the output feature map of stage 3 is concatenated with the feature map of stage 1 of the encoder through upsampling. Each stage fully extracts and fuses feature information through multiple layers of depth-wise separable convolution and attention mechanism.

[0038] S6: The feature maps generated at each stage of the decoder are subjected to 1x1x1 convolution operations to generate maps of corresponding scales. The prediction results of these maps at different scales are compared with the actual segmentation labels, and the parameters of the model are updated by calculating the loss function to improve the segmentation accuracy.

[0039] S7: Finally, the upsampled feature map (map4) of the decoder stage 4 is applied to the linear projection layer to output the pixel-level segmentation prediction result. In this way, accurate segmentation of medical images is achieved through the generation and fusion of multi-scale feature maps.

[0040] S8: Apply a linear projection layer to the upsampled features of the last layer and output pixel-level segmentation predictions.

[0041] Medical image segmentation is to divide an image into multiple regions or objects, and accurately locate and identify structures or lesion areas of interest. The present invention can effectively extract and process features of different scales by combining 3D depth separable convolution and EAM-3D modules, thereby improving segmentation accuracy and efficiency, and is used to efficiently and accurately segment liver tumors in liver images. The network model includes a contraction path (encoder), an expansion path (decoder) and jump connections. The encoder is divided into 4 stages, each of which contains multi-layer feature extraction and self-attention interaction mechanisms, which can capture features of different scales and enhance local feature representation. In addition, feature capture is performed at different scales of different objects through a multi-scale feature map generation module, and the feature expression capability is enhanced by combining feature maps of different scales.

[0042] Since a large amount of 3D data needs to be processed during medical image segmentation, in order to reduce computational complexity, this patent uses 3D separable convolution to replace traditional convolution, and uses the EAM-3D module to replace the traditional self-attention mechanism. Compared with traditional convolution, 3D separable convolution decomposes the convolution operation into two steps: channel-by-channel convolution and point convolution. Traditional convolution acts on all channels of the input feature map simultaneously through multiple 3D convolution kernels to generate an output feature map. Assume that the size of the input feature map is D×H×W×C in , the convolution kernel size is K×K×K×C in ×C out , then the output feature map size is D×H×W×C out , its computational complexity is O(K3 ×C in ×C out ×D×H×W). 3D separable convolution divides the convolution operation into two steps: channel-by-channel convolution and point convolution. Channel-by-channel convolution performs convolution operations on each input channel independently. The number of output channels is the same as the number of input channels, and the convolution kernel size is K×K×K×C in ; Point convolution uses a 1×1×1 convolution kernel to linearly combine the output of channel-by-channel convolution to obtain the final output feature map, and its convolution kernel size is 1×1×1×C in ×C out , the computational complexity of 3D separable convolution is O(K 3 ×C in ×D×H×W)+O(C in ×C out ×D×H×W). When K is larger or C out When the number of convolutions is large, the computational complexity of 3D separable convolution is much smaller than that of traditional 3D convolution.

[0043] The EAM-3D (Spatial Efficient Attention Mechanism 3D) module is an attention mechanism designed for three-dimensional spatial data. Unlike the SE module, which mainly focuses on channel attention, the EAM-3D module focuses more on feature extraction in the spatial dimension. Compared with the NL module, the EAM-3D module has lower computational complexity and is suitable for application scenarios with high real-time requirements. Similar to the CBAM module, the EAM-3D module can extract features at different spatial scales, but its structure is simpler and more efficient. The EAM-3D module performs weighted processing on features of different scales by calculating attention weights in three dimensions: depth, width, and height. First, the input feature map is averaged in the width and height dimensions, and then convolution is performed using a depth-wise separable convolution, followed by group normalization and Sigmoid activation function processing to obtain the attention weight in the depth dimension. The attention weights in the height and width dimensions are processed in this way, and finally these attention weights are multiplied element-by-element with the input feature map to obtain the weighted feature map. This mechanism can better capture and fuse multi-scale features and improve feature expression capabilities.

[0044] Assume that the input feature map size is X∈R B×C×D×H×W , where B is the batch size, C is the number of channels, and D, H, and W are the depth, height, and width respectively.

[0045] Perform global average pooling on the width and height respectively to obtain the depth attention weight: X d =Mean(X,dim=[4,3]), where X d ∈R B×C×D×1×1. Perform depth-wise separable convolution on the depth dimension to capture local features: X d =Conv3D(X d ,K d ), where K d ∈R 3×3×3×C×C . Use batch normalization and Sigmoid activation function to process the attention weight of the depth dimension: X d =Sigmoid(BatchNorm3D(X d )).

[0046] Similarly, the input feature map is globally averaged pooled in width and depth dimensions to obtain the height attention weight: X h =Mean(X,dim=[4,2]), where X h ∈R B×C×H×1×1 . Perform a depth-wise separable convolution operation on the height dimension to capture local features: X h =Conv3D(X h ,K h ), where K h ∈R 3×3×3×C×C . Use batch normalization and Sigmoid activation function to process the attention weight of the depth dimension: X h =Sigmoid(BatchNorm3D(X h )).

[0047] Perform global average pooling on the input feature map in depth and height dimensions to obtain the width attention weight: X w =Mean(X,dim=[3,2]), where X w ∈R B×C×W×1×1 . Perform a depth-wise separable convolution on the width dimension to capture local features: X w =Conv3D(X w ,K w ), where K w ∈R 3×3×3×C×C . Use batch normalization and sigmoid activation function to process the attention weights of the width dimension: X w =Sigmoid(BatchNorm3D(X w )).

[0048] Finally, the attention weights of the depth, height, and width dimensions are multiplied element-wise with the input feature map to obtain the final attention weight: output =X×X d ×X h ×X w .

[0049] Since the input image has the highest resolution and the least number of channels in the first stage of the encoder, in order to balance the computational complexity and feature extraction capability, two convolution operations are performed in the first stage. As the network goes deeper, the resolution of the feature map gradually decreases and the number of channels gradually increases. In order to fully extract high-level features, three convolution operations are performed in the second, third and fourth stages to enhance the feature representation capability while controlling the computational complexity within an acceptable range. The decoder adopts a symmetrical mechanism with the encoder. Each decoder is also divided into four stages, each of which contains multiple upsampling convolution modules, depthwise separable convolution layers and EAM modules. The upsampling operation restores the spatial resolution of the feature map layer by layer, and then uses the depthwise separable convolution layer to further extract features. Finally, the EAM module is used to enhance the expression capability of the feature map. Each stage of the decoder realizes the gradual recovery and optimization of features by splicing the feature map with the encoder and combining the high-level and mid-level feature information.

[0050] Since deep neural networks are prone to gradient vanishing or gradient exploding during training, in order to ensure that the original feature information is not lost during the deep information fusion process, the encoder and decoder use jump connections at different stages to ensure that the model can be trained normally while improving the model's feature expression ability. When implementing jump connections, there are usually two ways: residual connection and cascade connection. Assuming X is the input feature map, F(X) is the output feature map of the current layer, the output H(X) of the residual connection can be expressed as: H(X) = F(X) + X. This form ensures the retention of input information and increases the linear expression ability of the network. The output of the cascade connection G(X) = concat(X, F(X)), this form not only increases the dimension of the feature map, but also contains all the feature information of the input and output.

[0051] Since liver images may have different degrees of lesions in different human bodies, the segmented objects will appear in different scales. In order to achieve multi-scale feature mapping and combine feature maps of different scales, this patent generates feature maps of different resolutions and abstract levels at different stages of the decoder. Map1 is the lowest resolution feature map, which contains the most abstract information; map2 is a lower resolution feature map, which contains mid-level abstract information; map3 is a higher resolution feature map, which contains more specific detail information; map4 is the highest resolution feature map, which contains the most specific and detailed information. Through the generation and fusion of multi-scale feature maps, multi-scale prediction of medical images is achieved, details are retained and global information is extracted, and the segmentation performance of the model is improved.

[0052] Assume that the input feature map is X, X∈R H×W×C, where H, W, and C represent the height, width, and number of channels of the input feature map, respectively. First, perform an initial convolution operation on the input feature map X to obtain the initial feature map F0: F0 = Conv(X). This patent performs different downsampling operations: F s =DownSample s (F0), where DownSample s represents the downsampling operation of scale s. Then, the feature map of each scale is convolved to obtain the convolution feature map of each scale: G s =Conv(F s ). Then the feature maps of each scale are upsampled by UpSample s Restore to original size: H s =UpSample s (G s ). Finally, all scale feature maps are concatenated or summed to obtain the final fused multi-scale feature map:

[0053] Since the target area occupies a small part of the image in the liver segmentation task, resulting in a serious imbalance between positive and negative samples, this patent uses the Dice loss function to calculate the loss value between each output feature map and the corresponding true label image. Dice loss is a commonly used loss function for evaluating the performance of binary classification segmentation tasks. It is based on the Dice similarity coefficient (Dice Coefficient) and is used to measure the similarity between two samples. The Dice similarity coefficient (Dice Coefficient) is defined as:

[0054] Where A and B represent the predicted segmentation and true segmentation areas, respectively, |A| and |B| represent their areas, respectively, and the formulas are: and A∩B| represents the area of ​​the overlapping region, and the formula is:

[0055] Its loss function form is:

[0056] where p i and g i They represent the predicted value and true value of the i-th pixel respectively, and N is the total number of pixels.

[0057] Dice loss can better handle the problem of imbalance between positive and negative samples, directly reflects the quality of the segmentation results, is consistent with the evaluation indicators of the segmentation task (such as the Dice coefficient), and helps to improve the segmentation performance of the model.

[0058] The present invention proposes a three-dimensional convolutional neural network method based on deep separable convolution and attention mechanism for efficient feature extraction and positioning. By combining 3D deep separable convolution and EAM-3D modules, features of different scales can be effectively extracted and processed, thereby improving segmentation accuracy and efficiency, and being used to efficiently and accurately segment liver tumors in liver images. The network model includes a contraction path (encoder), an expansion path (decoder) and a jump connection. The encoder and the decoder are symmetrical, and the encoder is divided into 4 stages, each of which contains multi-layer feature extraction and self-attention interaction mechanisms, which can capture features of different scales and enhance local feature representation. The decoder is also divided into 4 stages, each of which contains multiple upsampling convolution modules, deep separable convolution layers and EAM modules, which are spliced ​​with the feature map of the encoder and combined with high-level and mid-level feature information to achieve gradual recovery and optimization of features. The upsampling operation restores the spatial resolution of the feature map layer by layer, and then uses the deep separable convolution layer to further extract features, and finally uses the EAM module to enhance the expression ability of the feature map. In order to avoid information loss when upsampling to restore image details, the encoder and decoder are combined through jump connections. In addition, a multi-scale feature map generation module is used to capture features at different scales of different objects, and the feature expression capability is enhanced by combining feature maps of different scales.

[0059] The medical image segmentation method based on 3D depth separable convolution and EAM module fusion includes the following steps:

[0060] S1: First, the medical image is input into the first stage of the encoder, and the input channels are increased from 1 to 16 using depthwise separable convolution. Then, batch normalization (BatchNorm3d) is used to normalize the image to improve training stability. Then, the LeakyReLU activation function is used to provide linear transformation, and finally, the efficient localization attention mechanism (EAM) is introduced to weight the features to enhance important features.

[0061] S2: Repeat depth-wise separable convolution, batch normalization, LeakyReLU and EAM attention processing three times, and output different feature maps of the medical image from low to high resolution, which are used as the feature maps of stage 2, stage 3, and stage 4 respectively. The number of channels in stage 2 increases from 16 to 32, the number of channels in stage 3 increases from 32 to 64, and the number of channels in stage 4 increases from 64 to 128.

[0062] S3: The feature map of stage 4 in the encoder is fed as the input image to the first stage of the decoder, whose input channel number is 128. The feature map is restored through multiple layers of depth-wise separable convolutional layers, BatchNorm3d layers, LeakyReLU activation layers and EfficientLocalizationAttention 3D modules, whose output channel number is 256.

[0063] S4: The output feature map of stage 1 in the decoder is upsampled to expand its spatial resolution and concatenated with the feature map from stage 3 in the encoder. It is then restored by combining high-level and mid-level feature information through multiple layers of depthwise separable convolutional layers, BatchNorm3d layers, LeakyReLU activation layers, and Efficient LocalizationAttention 3D modules as the output feature map of stage 2 in the decoder.

[0064] S5: Repeat S4, and concatenate the output feature map of stage 2 in the decoder with the feature map of stage 2 in the encoder by upsampling to obtain the output feature map of stage 3 in the decoder. Finally, concatenate the output feature map of stage 3 in the decoder with the feature map of stage 1 in the encoder by upsampling to obtain the output feature map of stage 4 in the decoder.

[0065] S6: Perform 1×1×1 convolution operations on the feature maps generated by the four different stages of the decoder in S3, S4, and S5, respectively, and generate corresponding maps at different scales. Finally, compare the prediction results at multiple scales with the actual segmentation labels, calculate the loss function, and update the model parameters.

[0066] S7: Concatenate or sum the feature maps of all scales to obtain the final multi-scale feature map, and finally output the pixel-level segmentation prediction by applying a linear projection layer.

[0067] Assuming that the input image is a three-dimensional liver image data with a size of 512×512×75, it first enters the channel-by-channel convolution in the first stage of the encoder, and performs K1×K2×K3 convolution on each input channel separately. The formula is:

[0068]

[0069] Secondly, after point convolution, the features of each channel are integrated together using 1×1×1 convolution. The formula is:

[0070]

[0071] Then pooling is performed in three dimensions: depth, height, and width. The formula is:

[0072]

[0073]

[0074]

[0075] Secondly, convolution and activation operations are performed on each pooling result to extract the attention weight. The formula is:

[0076] W d (x,y)=σ(Conv(A d (x,y)))

[0077] W h (x,z)=σ(Conv(A h (x,z)))

[0078] W w (y,z)=σ(Conv(A w (y,z)))

[0079] Finally, the attention weights are multiplied element-by-element with the original features to enhance feature extraction. The formula is:

[0080] V'(x,y,z)=V(x,y,z)·W d (x,y)·W h (x,z)·W w (y,z)

[0081] The second, third, and fourth stages also use depthwise separable convolution and attention mechanisms, but the number of output channels of each layer is increased to 32, 64, and 128, respectively.

[0082] Next, after the first stage of the decoder, upsampling is performed using trilinear interpolation, with the formula:

[0083]

[0084] Then it is fused with the feature map of the corresponding layer in the encoder, and the formula is:

[0085] V fusion (x,y,z)=V up (x,y,z)+V enc (x,y,z)

[0086] Repeat the above upsampling operation to restore the spatial resolution of the feature maps in stage 2, stage 3, and stage 4 of the decoder layer by layer. Then perform a 1×1×1 convolution operation on the feature map of each scale to obtain the convolution feature map of each scale: G s =Conv(F s). Then the feature maps of each scale are upsampled by UpSample s Restore to original size: H s =UpSample s (G s ). Finally, all scale feature maps are concatenated or summed to obtain the final fused multi-scale feature map: The processed three-dimensional feature map is output with a size of 512×512×75.

[0087] Finally, for each layer of feature maps generated, the Dice loss between them and the true label is calculated. The formula is:

[0088]

[0089] Where P is the result of the predicted segmentation and T is the true label. At each stage of the decoder, the Dice loss between the generated feature map and the true label is calculated as follows:

[0090]

[0091] Where L is the number of decoder stages, P l is the prediction result of the lth stage, and T is the true label.

[0092] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A medical image segmentation method based on 3D depth separable convolution and EAM module fusion, characterized in that: The following steps are involved: S1. First, the medical image is input into the first stage of the encoder, and the input channels are increased from 1 to 16 using depthwise separable convolution. Then, the image is normalized using BatchNorm3d to improve training stability. Then, the LeakyReLU activation function is used to provide linear transformation, and finally, the efficient localization attention mechanism EAM is introduced to weight the features to enhance important features. S2, repeating three times the depth-wise separable convolution, batch normalization, LeakyReLU and EAM processing, outputting different feature maps from low to high resolution of the medical image, which are used as the feature maps of stage 2, stage 3, and stage 4 respectively. The number of channels in stage 2 increases from 16 to 32, the number of channels in stage 3 increases from 32 to 64, and the number of channels in stage 4 increases from 64 to 128; S3, feed the feature map of stage 4 in the encoder as the input image to the first stage of the decoder, whose input channel number is 128, and restore the feature map through multiple layers of depth-separable convolutional layers, BatchNorm3d layers, LeakyReLU activation layers and EfficientLocalization Attention 3D modules, whose output channel number is 256; S4, the output feature map of stage 1 in the decoder is upsampled to expand its spatial resolution and concatenated with the feature map from stage 3 in the encoder. It is then restored by combining high-level and mid-level feature information through multiple layers of depth-separable convolutional layers, BatchNorm3d layers, LeakyReLU activation layers, and Efficient Localization Attention 3D modules to serve as the output feature map of stage 2 in the decoder; S5, repeat S4, splice the output feature map of stage 2 in the decoder with the feature map of stage 2 in the encoder by upsampling, to obtain the output feature map of stage 3 of the decoder; splice the output feature map of stage 3 in the decoder with the feature map of stage 1 in the encoder by upsampling, to obtain the output feature map of stage 4 of the decoder; S6, perform 1×1×1 convolution operations on the feature maps generated by the four different stages of the decoder in S3, S4, and S5, generate corresponding maps at different scales, compare the prediction results at multiple scales with the actual segmentation labels, calculate the loss function and update the model parameters; S7. Concatenate or sum the feature maps of all scales to obtain the final multi-scale feature map, and finally output the pixel-level segmentation prediction by applying a linear projection layer.

2. The medical image segmentation method based on 3D depth-separable convolution and EAM module fusion according to claim 1, characterized in that: The encoder part in S1 consists of 4 stages in total. Each stage contains multi-layer feature extraction and self-attention interaction mechanism, which captures features of different scales and enhances local feature representation, which not only improves the efficiency of feature extraction but also increases the accuracy of model segmentation.

3. The medical image segmentation method based on 3D depth-separable convolution and EAM module fusion according to claim 2, characterized in that: The 3D depth separable convolution is decomposed into two steps. The first step is to perform channel-by-channel convolution, and perform convolution operation on each input channel separately, that is, each input channel corresponds to a convolution kernel, and the number of output channels is the same as the number of input channels; the size of the input feature map and the size of the output feature map are the same as the size of the traditional convolution input feature map, and the size of the convolution kernel is K×K×K×C in , its computational complexity is H×W×D×K×K×K×C in ; The second step is point convolution, using a 1×1×1 3D convolution kernel to linearly combine the output of channel-by-channel convolution to obtain the final output feature map; The size of its input feature map is the same as that of the traditional convolution input feature map, and the size of the 3D convolution kernel is 1×1×1×C in ×C out , the output feature map size is H×W×D×C out ; The computational complexity of point convolution is H×W×D×C in ×C out ; The total computational complexity of 3D depthwise separable convolution is H×W×D×K×K×K×C in +H×W×D×C in ×C out .

4. The medical image segmentation method based on 3D depth-separable convolution and EAM module fusion according to claim 3, characterized in that: The EAM-3D module in S1 calculates the attention weights in the three dimensions of depth, width and height respectively, performs weighted processing on features of different scales, and enhances the attention to important features. First, it receives a 3D input feature map with a shape of (B, C, D, H, W), where B is the batch size, C is the number of channels, and D, H, and W are depth, height, and width respectively. Then, the attention weights are evaluated for the depth dimension, height dimension, and width dimension respectively. The first step processes the depth dimension. First, the input feature map is averaged in the width and height dimensions to obtain a shape of (B,C,D,1,1) feature map, and then use depth-wise separable convolution for convolution operation, and then perform group normalization and Sigmoid activation function to obtain the attention weight of the depth dimension; according to this method, the second step processes the height dimension and the third step processes the width dimension. The shape of the height dimension is (B,C,D,H,1), and the shape of the width dimension is (B,C,D,1,W); finally, the attention weights calculated in the depth, width and height dimensions are multiplied element by element with the input feature map to obtain the weighted feature map.

5. The medical image segmentation method based on 3D depth-separable convolution and EAM module fusion according to claim 4, characterized in that: The encoder in S1 is divided into 4 stages, each of which includes multiple depth-wise separable convolutional layers, batch normalization layers, LeakyReLU activation layers and EAM-3D modules; in the initial stage of the network, i.e., stage 1, the resolution of the input feature map is high, but the number of channels of the feature map is small; at this time, the information in the feature map is relatively primitive. In order to better extract low-level features, such as edges and color information, fewer repeated operations are required to extract basic features and ensure the efficiency of network calculations; two convolution operations are selected in stage 1 to balance the computational complexity; as the network goes deeper, the resolution of the feature map gradually decreases and the number of channels gradually increases. At this time, the information contained in the feature map is more abstract and rich; in order to fully extract these high-level features, more repeated operations are required to enhance the feature representation capability; since the resolution of the feature map gradually decreases and the amount of calculation is relatively reduced, three convolution operations are selected in stages 2, 3 and 4 to fully extract and process features while controlling the computational complexity within an acceptable range.

6. The medical image segmentation method based on 3D depth-separable convolution and EAM module fusion according to claim 5, characterized in that: The decoder in S2 is divided into 4 stages, each of which contains multiple upsampling convolution modules, depthwise separable convolution layers, and EAM modules; The upsampling operation restores the spatial resolution of the feature map layer by layer, and then uses the depthwise separable convolutional layer to further extract and optimize features. Finally, the EAM module is used to enhance the expressiveness of the feature map.

7. The medical image segmentation method based on 3D depth-separable convolution and EAM module fusion according to claim 6, characterized in that: In the initial stage of the encoder, the feature map has a low resolution but a high number of channels. At this time, the information in the feature map is highly abstract and rich. In order to reconstruct the resolution of the feature map, more convolution operations are required to fully extract the features. Therefore, three convolution operations are performed in the first stage of the decoder to maximize the ability of feature representation and extract more useful information; In the second stage of the decoder, the resolution of the input feature map is increased after the last sampling, and the number of channels is reduced; at this time, the information in the feature map is still complex and rich, and three convolution operations are required to process and extract features; the convolution operation at this stage further enhances the feature representation capability and integrates the feature information from the encoder, thereby improving the overall performance of the model; in the third stage of the decoder, the resolution of the input feature map is further increased, and the number of channels is further reduced; the information in the feature map begins to become more specific and detailed, but three convolution operations are still required to fully process and extract features; the convolution operation at this stage can ensure that in the high-resolution feature map after upsampling, a high feature representation capability is maintained, and the feature information transmitted by the encoder is further integrated; in the fourth stage of the decoder, the resolution of the input feature map is the highest and the number of channels is the least; at this time, the information in the feature map is the most specific and detailed, reducing the number of convolution operations; therefore, two convolution operations are performed at this stage to balance the computational complexity and feature extraction capability; through these convolution operations, it is ensured that sufficient detail information is extracted in the high-resolution feature map while maintaining a high computational efficiency.

8. The medical image segmentation method based on 3D depth-separable convolution and EAM module fusion according to claim 7, characterized in that: In S6, the decoder generates feature maps of different resolutions and different abstraction levels at different stages, where map1 is the feature map with the lowest resolution, containing the most abstract and high-level information; map2 is the feature map with lower resolution, containing mid-level abstract information; map3 is the feature map with higher resolution, containing more specific detail information; map4 is the feature map with the highest resolution, containing the most specific and detailed information; the feature maps of different resolutions generated in the four stages realize multi-scale feature prediction, which not only retains details but also extracts global information.

9. The medical image segmentation method based on 3D depth-separable convolution and EAM module fusion according to claim 8, characterized in that: Each feature map uses 3D convolution to adjust the number of channels of the feature map to 1, and then uses the upsampling function to enlarge the feature map to the original resolution so that its final output is consistent with the size of the input image. Finally, the Sigmoid activation function architecture output is mapped to the [0,1] interval; through the fusion of multi-scale feature maps, the spatial resolution and details of the image are better restored; the multi-scale output makes the model more robust to targets of different sizes and shapes, improving the generalization ability of segmentation tasks.

10. The medical image segmentation method based on 3D depth-separable convolution and EAM module fusion according to claim 9, characterized in that: The loss function between each output feature map and the corresponding true value image is calculated using Dice loss; Dice loss is a commonly used loss function for evaluating the performance of binary classification segmentation tasks; The formula is: Where A is the predicted segmentation result, and B is the actual segmentation annotation; In order to adapt to the training process of the neural network, its formula is usually converted to: where p i is the predicted value of the ith pixel, g i is the true value of the i-th pixel.

Images