Unet medical image segmentation method based on multi-attention mechanism improvement

Abstract

The invention discloses an improved Unet medical image segmentation method based on multiple attention mechanisms. Firstly, a medical image to be segmented is input into a CNN encoder containing attention modules, a Transformer is added to the bottommost layer of the encoder and used for obtaining local feature information, then encoding information of the attention module at the bottommost layer is input into a global and local interaction module, the feature information and the attention modules are summed element by element, and the feature information and the attention modules are summed element by element. The method comprises the following steps of: acquiring the input information of a decoder, using a three-level attention module in a decoder stage, using dense skipping connection of dense connection for the three-level attention module, using depth supervision for the side output of the decoder in a training process, and finally acquiring a final medical image segmentation result by utilizing complementary information among different scales, intermediates and predictive maps. Through experiments, the thought is feasible in the segmentation problem, the method has good performance when being applied to the segmentation field of each medical image data set, and the accuracy of medical image segmentation can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation and is a medical image segmentation method based on Unet improved by multiple attention mechanisms. Background Art

[0002] In today's medical image processing field, medical image segmentation has attracted much attention as an important task. It aims to segment different structures and tissues in medical images into regions with specific meanings to assist doctors in accurate diagnosis, effective treatment and in-depth research. However, medical images usually have complex structures and high noise, which are difficult to distinguish by the naked eye of doctors. Medical image segmentation has become an important task in the field of medical image processing. It aims to segment different structures and tissues in medical images into regions with specific meanings to assist doctors in diagnosis, treatment and research.

[0003] Since deep learning shows strong task-oriented learning capabilities, deep neural networks have been widely used in a variety of image analysis tasks in the field of medical image segmentation. However, many medical images are challenging, such as X-rays, CT scans, MRI, etc. These images are of great significance for tumor detection, organ segmentation, lesion recognition, etc., but due to their complexity, they need to be automatically analyzed and processed with the help of computer vision and deep learning technologies. Medical image segmentation plays an important role in clinical diagnosis, disease research and medical image analysis. For example, in tumor diagnosis, medical image segmentation can help doctors accurately identify the location and size of tumors; in organ segmentation, it can help doctors perform quantitative analysis of different organs. The evolution of segmentation methods is also an important part of medical image segmentation research. From traditional methods based on thresholds, edge detection and region growing to the rise of deep learning-based methods in recent years, medical image segmentation methods have undergone significant development. Deep learning methods such as convolutional neural networks (CNNs) and semantic segmentation networks have made remarkable progress and can more accurately identify structures and tissues in images.

[0004] This method inputs the medical image to be segmented into an encoder containing multiple display attenuation attention modules to obtain the encoded features output by each display attenuation attention module, which are then input into the cross-slice attention module to obtain the cross-slice features corresponding to the display attenuation attention module. Finally, the global local features in the decoder and the cross-slice features are combined to obtain the decoding features output by the top-level decoder module, and finally a medical image segmentation result with higher accuracy is obtained. Summary of the invention

[0005] The present invention aims to solve at least one of the problems existing in the prior art. To this end, the present invention proposes a medical image segmentation method based on multiple attentions, which can improve the accuracy of medical image segmentation.

[0006] A medical image segmentation method based on Unet improved by multiple attention mechanisms, characterized in that the specific steps are as follows:

[0007] S1: Obtain a medical image to be segmented, and input the medical image to be segmented into a CNN encoder including an attention module to obtain encoding features;

[0008] S2: In the encoding stage, a skip connection outside the compression and excitation encoder is designed, and a Transformer is added at the output of the encoder as a bridge between the encoder and decoder to obtain global and local information;

[0009] S3: Input the encoded features output by the attention module at the bottom layer of the encoder into the global and local interaction module to obtain global local features, and perform element-by-element summation of the global local features and the attention module to obtain the input features of the decoder module;

[0010] S4: A three-level attention module is used in the decoder stage. The three-level attention module uses dense skip connections with dense connections, and the spatial normalization output of the Transformer is used to enrich the feature map derived from the skip connection. The side output of the decoder uses deep supervision during the training process. Finally, a complete medical image segmentation picture is obtained.

[0011] The specific steps of inputting the medical image into the attention module to obtain the encoding features in step S1 are as follows:

[0012] S1.1: Input the medical image to be segmented into the encoder of each attention module, and use the local perception block to perform position encoding on the medical image to be segmented to obtain the encoded position information;

[0013] S1.2: The encoded position information is layer-normalized and then input into the preservation self-attention. Through the preservation self-attention, the attention is expanded to the bidirectional and two-dimensional form of image structure alignment and the spatial distance prior knowledge is introduced into the encoded position information to obtain the output result of the preservation self-attention.

[0014] S1.3: Add the output of the retention self-attention to the encoded position information element by element to obtain the addition result, and perform layer normalization on the result;

[0015] S1.4: Input the layer normalization result into the multi-scale token aggregation feed-forward network to perform multi-scale token aggregation, and finally add the output result and the element-by-element addition result element-by-element to obtain the encoding features output by each attention module.

[0016] The specific steps of compressing and exciting the skip connection outside the encoder and adding the Transformer as a bridge in step S2 are as follows:

[0017] S2.1: Input the output feature map of the encoder into the compression and excitation module, compress the feature map through global average pooling to obtain the global feature description of each channel, use two fully connected layers and ReLU activation function to reduce the number of channels, and then restore it to the original number of channels. Finally, generate the weight of each channel through the Sigmoid activation function, and multiply the generated weight with the original feature map to enhance important features and suppress unimportant features;

[0018] S2.2: Flatten the encoder output feature map and convert it into a format suitable for Transformer input through a linear layer;

[0019] S2.3: In the Transformer, a self-attention mechanism is used to calculate the relationship between features and generate a global context representation. The output features of the Transformer are then converted back to a format suitable for the decoder input through a linear layer.

[0020] The step S3 inputs the encoded features output by the attention module into the global and local interaction module, and the specific steps of obtaining the global and local features are as follows:

[0021] S3.1: Input the encoded features output by the bottom-level attention module in the encoder into the global and local interaction modules, and use the convolution layer to obtain smooth features;

[0022] S3.2: Input the smooth features into the global interaction block and the local interaction block, use the global interaction block to capture the long-range dependencies of the smooth features, and use the local interaction block to aggregate the local area features to obtain the global features and local features;

[0023] S3.3: sum the global local features and the attention module element by element to obtain the input features of the decoder module;

[0024] The step S4 uses a three-level attention module in the decoder stage as follows:

[0025] S4.1: In the three-level attention module, the feature maps from different layers of the decoder are first fused. These feature maps contain semantic information of different sizes and can provide contextual information. For each feature map, a self-attention mechanism is applied to calculate the importance of each position in the feature map. By calculating the similarity matrix of the feature map, attention weights are generated, which are used to weight different parts of the input feature map;

[0026] S4.2: The feature map processed by the attention mechanism is spatially normalized to ensure the consistency of the feature map at different spatial positions, and then sent to the subsequent decoder layer to form the final output feature map;

[0027] S4.3: Design multiple side outputs in each main stage of the decoder, connect them with the feature maps of the intermediate layers, define an independent loss function for each side output, and use the complementary information between different scales and intermediate features through multiple side outputs to improve the model performance and obtain the final medical segmentation image. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is an overall flow chart of the medical image segmentation method based on Unet improved by multiple attention mechanisms described in the present invention.

[0029] Figure 2 This is a flow chart of inputting a medical image to be segmented into an encoder to obtain encoding features according to the present invention.

[0030] Figure 3 This is a flowchart of compressing and exciting the skip connection outside the encoder and adding Transformer as a bridge as described in the present invention.

[0031] Figure 4 This is a flow chart of obtaining input features of a decoder module through a global local feature map as described in the present invention.

[0032] Figure 5 The architecture diagram of decoding the global local features of the present invention

[0033] Figure 6 Flowchart of using a three-level attention module for the decoder stage of the present invention and obtaining the final segmented image. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings:

[0035] like Figure 1 As shown, a medical image segmentation method based on Unet improved by multiple attention mechanisms from step 101 to step 104:

[0036] Step 101: obtaining a medical image to be segmented, and inputting the medical image to be segmented into a CNN encoder including an attention module to obtain encoding features;

[0037] Step 102: In the encoding stage, a skip connection outside the compression and excitation encoder is designed, and a Transformer is added at the output of the encoder as a bridge between the encoder and the decoder to obtain global and local information;

[0038] Step 103: Input the encoded features output by the attention module at the bottom layer of the encoder into the global and local interaction module to obtain global local features, and perform element-by-element summation of the global local features and the attention module to obtain the input features of the decoder module;

[0039] Step 104: A three-level attention module is used in the decoder stage. The three-level attention module uses dense skip connections with dense connections, and uses the spatially normalized output of the Transformer to enrich the feature map derived from the skip connection. The side output of the decoder uses deep supervision during the training process. Finally, a complete medical image segmentation picture is obtained.

[0040] like Figure 2 As shown: the process step 101 of inputting the medical image to be segmented into the encoder to obtain the encoding feature from step 201 to step 203

[0041] Step 201: inputting the medical image to be segmented into the encoder of each attention module, and using the local perception block to perform position encoding on the medical image to be segmented to obtain encoding position information;

[0042] Step 202: normalize the encoded position information by layer and input it into the preservation self-attention, expand the attention to the bidirectional and two-dimensional forms of image structure alignment through the preservation self-attention and introduce the spatial distance prior knowledge into the encoded position information, and obtain the output result of the preservation self-attention;

[0043] Step 203: Add the output result of the retention self-attention to the encoding position information element by element to obtain the addition result, and perform layer normalization on the result, input the result into the multi-scale token aggregation feedforward network to perform multi-scale token aggregation, and finally add the output result to the element-by-element addition result element by element to obtain the encoding feature output by each attention module;

[0044] like Figure 3 As shown: Step 102 of compressing and exciting the skip connection outside the encoder and adding Transformer as a bridge from step 301 to step 304:

[0045] Step 301: Input the output feature map of the encoder into the compression and excitation module, compress the feature map through global average pooling to obtain the global feature description of each channel, use two fully connected layers and ReLU activation function to reduce the number of channels, and then restore it to the original number of channels;

[0046] Step 302: Generate the weight of each channel through the Sigmoid activation function, and multiply the generated weight with the original feature map to enhance important features and suppress unimportant features;

[0047] Step 303: Flatten the encoder output feature map and convert it into a format suitable for Transformer input through a linear layer;

[0048] Step 304: In the Transformer, a self-attention mechanism is used to calculate the relationship between features, generate a global context representation, and convert the output features of the Transformer back to a format suitable for decoder input through a linear layer.

[0049] like Figure 4-5 As shown: the process step 103 of obtaining the decoder module through the global local feature map from step 401 to step 403:

[0050] Step 401 inputs the encoded features output by the bottom-level attention module in the encoder into the global and local interaction module, and uses the convolution layer to obtain smooth features;

[0051] Step 402: inputting the smooth features into the global interaction block and the local interaction block, using the global interaction block to capture the long-range dependency of the smooth features, and using the local interaction block to aggregate the local area features to obtain the global features and the local features;

[0052] Step 403: sum the global local features and the attention module element by element to obtain the input features of the decoder module;

[0053] like Figure 6 As shown: The decoder stage uses a three-level attention module and obtains the final segmented image in step 104 from step 501 to step 504:

[0054] Step 501: In the three-level attention module, the feature maps from different layers of the decoder are first fused. These feature maps contain semantic information of different sizes and can provide contextual information. For each feature map, a self-attention mechanism is applied to calculate the importance of each position in the feature map. By calculating the similarity matrix of the feature map, attention weights are generated, which are used to weight different parts of the input feature map;

[0055] Step 502: The feature map processed by the attention mechanism is spatially normalized to ensure the consistency of the feature map at different spatial positions, and then sent to the subsequent decoder layer to form the final output feature map;

[0056] Step 503: Design multiple side outputs in each main stage of the decoder, connect them with the feature map of the intermediate layer, define an independent loss function for each side output, and use the complementary information between different scales and intermediate features through multiple side outputs to improve model performance and obtain the final medical segmentation image;

[0057] The network is designed to achieve higher segmentation progress by capturing local positioning details and global information. To achieve this goal, several complementary improvements are proposed. The most important is the introduction of a three-level attention module in the decoder, which uses the attention map from the Transformer to focus on the most relevant details. The three-level attention module consists of two consecutive convolutional layers and ReLU layers, and finally a batch normalization layer, just like the standard decoder block in the UNet model. The last stage of the three-level attention is the element-wise multiplication between the coefficients of the Transformer and the feature map. First, in order to suppress irrelevant areas in the image, the jump connection features are and from the lower encoder block and Transformer The feature map of is fed into the attention gate:

[0058]

[0059] In the formula, , , represents linear transformation; S is the S-type activation function; represents element-wise multiplication. The result is then concatenated with the feature map and used as the input for channel attention, as shown in the following formula:

[0060]

[0061] The outputs of the last two convolutions are Indicates that the spatial normalization mapping produced by Transformer is represented as follows:

[0062]

[0063] in, The product of .

Claims

1. A medical image segmentation method based on Unet improved by multiple attention mechanisms, characterized in that: The specific steps are as follows: S1: Obtain a medical image to be segmented, and input the medical image to be segmented into a CNN encoder including an attention module to obtain encoding features; S2: In the encoding stage, a skip connection outside the compression and excitation encoder is designed, and a Transformer is added at the output of the encoder as a bridge between the encoder and decoder to obtain global and local information; S3: Input the encoded features output by the attention module at the bottom layer of the encoder into the global and local interaction module to obtain the global local features, and perform element-by-element summation of the global local features and the attention module to obtain the input features of the decoder module; S4: A three-level attention module is used in the decoder stage. The three-level attention module uses dense skip connections with dense connections. The spatially normalized output of the Transformer is used to enrich the feature maps derived from the skip connections. The side output of the decoder uses deep supervision during training; finally, a complete medical image segmentation picture is obtained.

2. According to claim 1, a medical image segmentation method based on Unet improved by multiple attention mechanism is characterized in that: The specific steps of inputting the medical image into the attention module to obtain the encoding features in step S1 are as follows: S1.1: Input the medical image to be segmented into the encoder of each attention module, and use the local perception block to perform position encoding on the medical image to be segmented to obtain the encoded position information; S1.2: The encoded position information is layer-normalized and then input into the preservation self-attention. Through the preservation self-attention, the attention is expanded to the bidirectional and two-dimensional form of image structure alignment and the spatial distance prior knowledge is introduced into the encoded position information to obtain the output result of the preservation self-attention. S1.3: Add the output of the retention self-attention to the encoded position information element by element to obtain the addition result, and perform layer normalization on the result; S1.4: Input the layer normalization result into the multi-scale token aggregation feed-forward network to perform multi-scale token aggregation, and finally add the output result and the element-by-element addition result element-by-element to obtain the encoding features output by each attention module.

3. According to claim 1, a medical image segmentation method based on Unet improved by multiple attention mechanism is characterized in that: The specific steps of compressing and exciting the skip connection outside the encoder and adding the Transformer as a bridge in step S2 are as follows: S2.1: Input the output feature map of the encoder into the compression and excitation module, compress the feature map through global average pooling to obtain the global feature description of each channel, use two fully connected layers and ReLU activation function to reduce the number of channels, and then restore it to the original number of channels. Finally, generate the weight of each channel through the Sigmoid activation function, and multiply the generated weight with the original feature map to enhance important features and suppress unimportant features; S2.2: Flatten the encoder output feature map and convert it into a format suitable for Transformer input through a linear layer; S2.3: In the Transformer, a self-attention mechanism is used to calculate the relationship between features and generate a global context representation. The output features of the Transformer are then converted back to a format suitable for the decoder input through a linear layer.

4. According to claim 1, a medical image segmentation method based on Unet improved by multiple attention mechanisms is characterized in that: The step S3 inputs the encoded features output by the attention module into the global and local interaction module, and the specific steps of obtaining the global and local features are as follows: S3.1: Input the encoded features output by the bottom-level attention module in the encoder into the global and local interaction modules, and use the convolution layer to obtain smooth features; S3.2: Input the smooth features into the global interaction block and the local interaction block, use the global interaction block to capture the long-range dependencies of the smooth features, and use the local interaction block to aggregate the local area features to obtain the global features and local features; S3.3: Element-wise summation of the global local features with the attention module is performed to obtain the input features of the decoder module.

5. According to claim 1, a medical image segmentation method based on Unet improved by multiple attention mechanism is characterized in that: The step S4 designs a three-level attention module in the decoder stage, and the specific steps of using deep supervision are as follows: S4.1: In the three-level attention module, the feature maps from different layers of the decoder are first fused; these feature maps contain semantic information of different sizes and can provide contextual information. For each feature map, a self-attention mechanism is applied to calculate the importance of each position in the feature map; by calculating the similarity matrix of the feature maps, attention weights are generated, which are used to weight different parts of the input feature map; S4.2: The feature map processed by the attention mechanism is spatially normalized to ensure the consistency of the feature map at different spatial positions, and then sent to the subsequent decoder layer to form the final output feature map; S4.3: Design multiple side outputs in each main stage of the decoder, connect them with the feature maps of the intermediate layers, define an independent loss function for each side output, and use the complementary information between different scales and intermediate features through multiple side outputs to improve the model performance and obtain the final medical segmentation image.

Images