A method for image segmentation using a semantic segmentation network

Abstract

The application discloses a kind of high-efficiency semantic segmentation networks, suitable for medical image analysis, automatic driving etc. The network adopts encoder-decoder architecture, combines CNN with Transformer (converter model), balances global modeling and computational efficiency. Encoder extracts multi-scale features through lightweight convolution, and introduces spatial selection module: its gate convolution splits channel into gate signal and reserved information, and key spatial features are activated and strengthened by Sigmoid activation function; Grouped pooling module extracts details using multi-scale pooling, and restores channels after upsampling and splicing. The decoder fuses multi-scale features and restores resolution through upsampling, enhances efficient channel attention, fuses global max pooling and average pooling, generates channel weight using one-dimensional convolution, and optimizes feature dependence. The design improves small target segmentation accuracy through gate mechanism and multi-scale pooling, is lightweight and easy to expand, and has high precision and practicality.

Description

Technical Field

[0001] This invention relates to the field of semantic segmentation technology, and in particular to a semantic segmentation network. Background Technology

[0002] Image segmentation, a common task in image processing, is widely used in medical image processing, remote sensing image processing, and road scene processing. The main task of image segmentation is to separate different categories of objects in an image, focusing on a few categories of interest. Computer vision includes several different tasks: image classification, object detection, semantic segmentation, instance segmentation, and panoptic segmentation. These tasks aim to divide objects in an image into different parts and assign unique labels to each object. The difference lies in the accuracy requirements of the labels. Object recognition and classification do not require high precision in distinguishing the actual contour boundaries of objects; the goal is to differentiate between the main object and the background. Semantic, instance, and panoptic segmentation, on the other hand, primarily focus on pixel-level differences between objects. Correctly distinguishing adjacent and easily confused objects during dense object prediction is a significant challenge. In recent years, with the development of deep learning and artificial intelligence, the accuracy of image segmentation has improved along with advancements in network models. How to use deep learning models to better perform image segmentation tasks and how to achieve an effective balance between accuracy and efficiency in network construction are important research directions.

[0003] The excellent performance of attention mechanisms has led to their widespread application; however, existing attention fusion methods are insufficient, thus a new approach is urgently needed to overcome these shortcomings. Summary of the Invention

[0004] In view of the problems existing in the prior art, the present invention provides a semantic segmentation network, characterized in that: the network includes an encoder, a decoder and a spatial selection module;

[0005] The encoder uses a convolutional neural network to extract image features, obtaining feature maps x at different resolutions. i Where i = 1, 2, 3, 4; the smaller i is, the corresponding x i The higher the resolution; x i The spatial selection module is used to downsample and fuse the data to x. i+1 middle;

[0006] The spatial selection module includes: a gated convolution module and a grouped pooling module;

[0007] The high-resolution feature map x obtained by the gated convolution module i-1The number of channels is doubled. The expanded channels are then split, with one portion used for gating and the other portion's information retained and multiplied element-wise with the gated feature map to obtain the gated feature map x. i ;

[0008] The group pooling module divides the input feature map channel C into 4 parts to obtain a C / 4 feature map. It then processes the separated feature maps using pooling at different scales to obtain important information from the features. Next, it uses upsampling to restore the feature map resolution to the input size. Finally, it concatenates the 4 C / 4 channel feature maps to restore the channel dimension of the feature map.

[0009] Preferably, the calculation formula for the gated convolution module is:

[0010]

[0011] x i =Gate conv(x) i-1 )+x i-1 ;

[0012] Split() is used to split information and generate gating information, Split′() is used to split information and retain information, and conv 1×1 It is a one-dimensional convolution; This represents element-wise multiplication, and σ() is the sigmoid activation function.

[0013] Preferably, the network further includes a channel attention module, which serves as an improved convolutional backbone for feature extraction. The input is x, and the output y of the enhanced channel attention residual block is as follows:

[0014] y = rule(E 2 CAM(f(x))+x)

[0015] Among them, E 2 CAM() is an enhanced efficient channel attention module, f(x) is the convolution result through the residual network; rule() is a nonlinear transformation function in this field used to enhance the expressive power of the residual structure.

[0016] Preferably, the specific E 2 CAM is calculated as follows:

[0017]

[0018] Here, the input is x, and expand_x() is a PyTorch function that expands the number of channels back to the original number of channels. 1×1For one-dimensional convolution, GM pooling() and GA pooling() represent global max pooling and global average pooling, respectively, and σ() is the sigmoid activation function. The resulting E 2 The channel attention is then multiplied by the input matrix to obtain the final output y.

[0019] Compared with existing technical solutions, the present invention has at least the following beneficial effects:

[0020] 1) The spatial information selection module provided by this invention extracts important information from the feature map through gating and retains it for training in the next layer. This improves the network's ability to understand spatial information in the deep network and strengthens the fusion of spatial features and semantic features. Because of its simple structure and ability to retain information well, it can be embedded in any backbone.

[0021] 2) The enhanced efficient channel attention module (E) provided by this invention 2 CAM) adds a global max pooling branch, which is then fused with the obtained global average pooling channel attention to obtain enhanced fused attention. Attached Figure Description

[0022] Figure 1 This is the overall structure of the semantic segmentation network of this invention;

[0023] Figure 2 This is a structural diagram of the spatial information selection module provided by the present invention;

[0024] Figure 3 This is a structural diagram of a high-efficiency channel attention module provided by the present invention.

[0025] The present invention will now be described in further detail. However, the examples described below are merely simplified examples of the present invention and do not represent or limit the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims. Specific Implementation

[0026] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0027] The semantic segmentation network provided by this invention uses ResNet18 with enhanced channel attention. A Transformer structure is added to the connection between the encoder and decoder to process global information. The overall network is a hybrid structure of CNN and Transformer.

[0028] like Figure 1The diagram shows the main structure of the semantic segmentation network of this invention, where CAA represents hybrid cross-axial attention, Sea-Att represents enhanced compressed axial attention, ssm represents spatial selection module, and α and λ are weight parameters. Preferably, λ is set to 0.6.

[0029] The semantic network consists of two parts: an encoder and a decoder. The encoder uses a lightweight convolutional neural network to extract image features. At each different resolution, compressed axial attention is used to extract the axial information of the overall space, and the resulting feature maps are input into the decoder. After obtaining feature maps at different resolutions, the decoder upsamples the low-resolution feature maps by a factor of 2 and fuses them element-wise with the processed high-resolution images, enhancing the decoder's ability to acquire contextual information from multi-scale feature maps. Enhanced compressed axial attention adds a contextual information branch to the compressed axial attention mechanism. It uses a basic MobileNetV2 module to process the Q, K, and V feature maps, and multiplies the resulting feature maps element-wise with the output of compressed axial attention, thereby enhancing the fusion of contextual information in the axial attention mechanism.

[0030] like Figure 2 The diagram shown is a structural diagram of the spatial information selection module provided by this invention. The spatial information selection module consists of two parts: the first part is gated convolution, and the second part is to split the input features and perform multi-scale pooling to obtain the output of fused features. The gated convolution generates a high-resolution feature map x. i-1 Channel expansion is performed, doubling the original number of channels. The expanded channels are then split, with a portion used for gating and the remaining portion, retaining information, multiplied element-wise with the gated feature map to obtain the gated feature map x'. A residual connection is then used to connect the new feature map x' to the input feature map x. i-1 By adding each element, we obtain the output x of the next layer. i The second part, group pooling, divides the input feature map's channel C into four parts, resulting in a C / 4 feature map. Pooling at different scales [1, 2, 3, 5] is used to process the separated feature maps, extracting important information from the features. Then, upsampling is used to restore the feature map resolution to the input size. Finally, the four C / 4 channel feature maps are concatenated to restore the channel dimension of the feature map.

[0031] Existing input feature map After gated convolution, spatial information is obtained from the feature map. In the model, each downsampling step fuses the high-resolution gated convolution features with the next layer, enhancing the attention to spatial information and improving the segmentation effect for small objects. The specific calculation formula is as follows:

[0032]

[0033] xi =Gate conv(x) i-1 )+x i-1 ;

[0034] Split() is used to split information and generate gating information, while Split′() is used to split information and retain information. σ() represents element-wise multiplication, and σ() is the sigmoid activation function. Gated convolution does not change the resolution and number of channels of the input feature map, but when selecting modules through spatial information, it is equivalent to performing a spatial attention operation. Gating extracts important information from the feature map and retains it for the training of the next layer. In the deep network, it improves the network's ability to understand spatial information and strengthens the fusion of spatial and semantic features. Because of its simple structure and ability to retain information well, it can be embedded in any backbone.

[0035] like Figure 3 As shown, this invention provides an efficient channel attention module. Based on the SE module, it replaces the two-layer fully connected structure with a one-dimensional convolution, similar to the method of removing fully connected layers in a fully convolutional network. This one-dimensional convolution reduces the number of channels, decreasing computation, and then activates the resulting channel sequence to restore the number of input channels. Since the feature map pooling only uses global average pooling, an enhanced efficient channel attention module is proposed, adding a global max pooling branch and fusing it with the obtained global average pooling channel attention to obtain an enhanced fused attention.

[0036] The channel attention module provided in this invention is a plug-and-play module. In this chapter's model, it is inserted into the ResNet18 backbone as an improved convolutional backbone for feature extraction. Thus, the output Y of the input x through the enhanced channel attention residual block is as follows:

[0037] Y = rule(E) 2 CAM(f(x))+x);

[0038] Among them, E 2 CAM stands for Enhanced Efficient Channel Attention Module, where f(x) is the convolution result through the residual network. `rule()` is a non-linear transformation function used to enhance the expressive power of the residual structure. Specifically, E... 2 CAM is calculated as follows:

[0039] E 2 CAM(x) = expand_x(σ(conv) 1×1 (GMpooling(x)))+σ(conv 1×1 (GApooling(x))));

[0040] Here, the input is x, and expand_x is a PyTorch function that expands the number of channels back to the original number of channels. 1×1 This is a one-dimensional convolution with a kernel size of 5. `GMPooling()` and `GApooling()` represent global max pooling and global average pooling, respectively, and `σ()` is the sigmoid activation function. The resulting E... 2 The channel attention is then multiplied by the input matrix to obtain the final output y. PyTorch is an open-source machine learning library, specifically used for building and training neural networks. Think of it as a powerful toolbox that provides many convenient functions and modules, making it easier to design, implement, and experiment with various deep learning models.

[0041] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0042] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable way without contradiction. In order to avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

[0043] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed by the present invention.

Claims

1. A method for image segmentation using a semantic segmentation network, the method comprising: The network comprises an encoder, a decoder and a space selection module; The encoder extracts image features using a convolutional neural network to obtain feature maps x of different resolutions i , where i = 1, 2, 3, 4; the smaller i is, the higher the corresponding x i resolution is; x i is fused into x i+1 through the spatial selection module and down-sampling. The space selection module comprises a gating convolution module and a group pooling module; The high-resolution feature map x obtained by the gating convolution module i-1 Channel expansion is performed, the number of channels becomes twice the original number, and the expanded channels are further split, wherein a part is used for gating, and the other part is multiplied element by element with the feature map of the gate to obtain the gated feature map x i ; The gating convolution module has a specific calculation formula as follows: x i = Gateconv(x i-1 ) + x i-1 ; where Split() is split for generating gating information, Split'() is split for preserving information; conv 1×1 is a one-dimensional convolution; represents element multiplication, and σ() is a sigmoid activation function; The group pooling module divides the channel C of an input feature map into four parts to obtain a feature map with C / 4 channels, processes the separated feature maps using different scale pooling to obtain important information in the features, restores the resolution of the feature map to the input size using up-sampling, and finally splices the four C / 4 channel feature maps to restore the channel dimension of the feature map.

2. The method of image segmentation according to claim 1, characterized in that: The network further comprises a channel attention module, which is used as an improved convolution backbone to extract features; the input is x, and the output y of the enhanced channel attention residual block is as follows: y = rule(E 2 CAM(f(x))+x) wherein E 2 CAM() is an enhanced efficient channel attention module, f(x) is a convolution result through a residual network; rule() is a nonlinear transformation function for enhancing the expression ability of the residual structure.

3. The method of image segmentation according to claim 2, characterized in that: Specific E 2 The CAM is calculated as follows: E 2 CAM(x) = expand_x(σ(conv 1×1 (GMpooling(x)))+σ(conv 1×1 (GApooling(x)))) Where, the input is x, expand_x is a function in pytorch that can expand the channel number to the original channel number, conv 1×1 is one-dimensional convolution, GM pooling() and GA pooling() represent global maximum pooling and global average pooling, σ() is the activation function sigmoid, and the obtained E 2 The channel attention is multiplied with the input matrix to obtain the final output y.

Images