An attention-guided deformable self-attention semantic segmentation method

Abstract

This invention discloses an attention-guided deformable self-attention semantic segmentation method. It uses a weighted, channel-based attention module to separate attention feature channels with different weights. A deformable self-attention Transformer is used to extract features from high-weight feature maps. A deformable self-attention window module transforms the segmented image blocks into irregular shapes adapted to the target shape, then fills them into square image blocks before feeding them into the Transformer. This allows self-attention to focus more on the more critical regions in the feature maps and acquire more feature information. A dual-weighted dual-attention module is used to extract features from low-weight feature maps. By acquiring and aggregating spatial and channel attention at different scales of the feature maps, spatial and channel attention can better capture the spatial and channel information of the feature maps. This invention reduces the memory and computational costs of semantic segmentation while improving segmentation accuracy.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and more specifically to an attention-guided deformable self-attention semantic segmentation method. Background Technology

[0002] Semantic segmentation divides each pixel in an image into different semantic regions and categorizes these regions according to specified classes. As an important branch of computer vision and a key task, semantic segmentation has increasingly wide applications, such as geographic information systems, autonomous driving, medical image analysis, and robotics. Its importance is becoming increasingly apparent with the rapid development of artificial intelligence.

[0003] Currently, semantic segmentation algorithms are generally based on convolutional neural networks (CNNs) or Transformers. While CNN-based methods have achieved good results in semantic segmentation, the localized operations of convolution make it difficult to learn global and long-range semantic interaction information of an image. Transformers are now widely used in Natural Language Processing (NLP). This model relies on self-attention mechanisms and captures dependencies between words in a sentence. The success of Transformers in NLP has inspired research methods in computer vision. Researchers have introduced Transformers into the vision field, and subsequent models such as ViT, DETR, and SETR have shown great potential in image recognition, object detection, and semantic segmentation, respectively. However, the large receptive field of Transformers leads to large memory usage and high computational costs, and they are also affected by irrelevant parts of the image. Summary of the Invention

[0004] This invention provides an attention-guided deformable self-attention semantic segmentation method that can perform semantic segmentation on images, reducing the memory and computational costs of semantic segmentation and improving segmentation accuracy.

[0005] To solve the above problems, the present invention is achieved through the following technical solution:

[0006] An attention-guided, deformable self-attention semantic segmentation method includes the following steps:

[0007] Step 1: Preprocess the original image by scaling its height and width to the same size.

[0008] Step 2: Perform feature extraction operations twice on the image obtained in Step 1 to obtain the feature map after one feature extraction and the feature map after two feature extractions, respectively.

[0009] Step 3: Generate attention weights for the feature maps obtained from the two feature extractions in Step 2, and enhance the important features of the feature maps to obtain weighted feature maps with important features;

[0010] Step 4: Set the pixel values ​​of the feature channels with low attention weights in the feature map with important feature weights obtained in Step 3 to 0, thereby obtaining a high-weight feature map; at the same time, set the pixel values ​​of the feature channels with high attention weights in the feature map with important feature weights obtained in Step 3 to 0, thereby obtaining a low-weight feature map.

[0011] Step 5: Pass the high-weight feature map obtained in Step 4 through a deformable self-attention Transformer operation to obtain a feature map with rich global context information.

[0012] Step 6: Perform a dual-weight dual-attention operation on the low-weight feature map obtained in Step 4 to obtain a dual-attention feature map;

[0013] Step 7: Add the feature map with rich global context information obtained in Step 5 and the dual attention feature map obtained in Step 6 to obtain the attention feature map;

[0014] Step 8: Perform three attention map upsampling operations on the attention feature map obtained in Step 7 to obtain the upsampled attention feature map;

[0015] Step 9: Perform a convolution operation on the upsampled attention feature map obtained in Step 8 to obtain the feature map after channel reduction;

[0016] Step 10: Perform channel concatenation on the feature map obtained in Step 9 and the feature map obtained in Step 2 after two feature extractions to obtain the first channel concatenated feature map.

[0017] Step 11: Perform an upsampling operation on the first channel stitching feature map obtained in Step 10 to obtain the first upsampled channel stitching feature map;

[0018] Step 12: Perform channel stitching operation on the first upsampled channel stitched feature map obtained in Step 11 and the feature map after the first feature extraction obtained in Step 2 to obtain the second channel stitched feature map.

[0019] Step 13: Perform an upsampling operation on the second channel stitching feature map obtained in Step 12 to obtain the second upsampled channel stitching feature map;

[0020] Step 14: Perform a convolution operation on the second upsampled channel spliced ​​feature map obtained in Step 13 to obtain the segmentation result image.

[0021] In step 2 above, each feature extraction operation requires a 3×3 convolution operation, a 1×1 convolution operation, a ReLU activation operation, and a max pooling operation in sequence.

[0022] The specific process of step 3 above is as follows:

[0023] Step 3.1: Perform global average pooling on the feature maps after the two feature extractions to obtain a feature map with c channels;

[0024] Step 3.2: Perform two fully connected operations on the feature map with c channels, and train the weights using the correlation between channels to obtain the attention weights; the first fully connected operation compresses c channels into cr channels to reduce the computational cost, and the first fully connected operation is followed by a ReLU activation operation; the second fully connected layer restores the number of channels to c, and the second fully connected operation is followed by a Sigmoid activation operation; where r is the compression ratio.

[0025] Step 3.3: Perform a full multiplication operation on the feature maps and attention weights after the two feature extractions to obtain a weighted feature map with important features.

[0026] In step 4 above, the highest and lowest attention weights of each feature channel of the feature map with important feature weights are added together and then divided by 2 to obtain the average attention weight. If the attention weight of a feature channel of the feature map with important feature weights is less than or equal to the average attention weight, it is a feature channel with low attention weight. If the attention weight of a feature channel of the feature map with important feature weights is greater than the average attention weight, it is a feature channel with high attention weight.

[0027] The specific process of step 5 above is as follows:

[0028] Step 5.1: First, divide the high-weight feature map into uniform original image blocks;

[0029] Step 5.2: Perform a deformable self-attention window operation on each original image patch obtained in Step 5.1 to obtain deformable image patches; that is:

[0030] First, perform a depthwise convolution operation on each original image block to capture local features, and then use GELU activation and convolution operations to obtain the offset of each pixel in each image block;

[0031] Then, the coordinates of each contour reference point of each original image block are added with its offset, so that the position of each contour reference point is shifted, and the shifted contour reference point is used as the new contour of the original image block.

[0032] Then, bilinear interpolation is used to fill the pixels in the new contour of each original image block with pixel values ​​to obtain an irregular image block. Then, pixels with a pixel value of 0 are used to fill the irregular image block to the same size as the original image block to obtain a deformed image block.

[0033] Step 5.3: Perform linear mapping on each deformed image block obtained in Step 5.2, so that the dimension of each deformed image block is mapped to the specified dimension;

[0034] Step 5.4: Add a learnable positional encoding vector to each deformed image obtained after linear mapping in step 5.3 to obtain an image patch with positional information;

[0035] Step 5.5: Input each image block with positional information obtained in Step 5.4 into the Transformer encoder to obtain the self-attention perceptual feature vector of each image block with positional information;

[0036] Step 5.6: Input all the self-attention-aware feature vectors obtained in Step 5.5 into the Transformer decoder to obtain a feature map with rich global context information.

[0037] In step 5.2 above, each image block includes 9 contour reference points, namely the top left pixel, the top right pixel, the bottom left pixel, the bottom right pixel, the center pixel, the center pixel of the top row, the center pixel of the bottom row, the center pixel of the leftmost column, and the center pixel of the rightmost column.

[0038] The specific process of step 6 above is as follows:

[0039] Step 6.1: Use a residual network with dilated convolution strategy to extract features from the low-weight feature map;

[0040] Step 6.2: Simultaneously perform dual-weighted spatial attention extraction and dual-weighted channel attention extraction on the feature maps extracted by the residual network to obtain spatial attention feature maps and channel attention feature maps, respectively.

[0041] ① When performing spatial attention extraction with two weights:

[0042] First, the feature maps extracted by the residual network are subjected to five parallel convolution operations, including two 3×3 convolution operations and three 1×1 convolution operations. Then, the convolutional feature maps obtained from these five parallel convolution operations undergo a morphological transformation to obtain a two-dimensional feature matrix. This transformation converts the number of channels in the feature map into the number of rows in the matrix, and the product of the feature map dimensions into the number of columns in the matrix. Each row in the matrix represents a feature map. Figure 1The value of each channel; the morphological transformation from matrix to feature map is to transform each row of the matrix into a channel of the feature map;

[0043] Then, the feature matrix obtained after one 3×3 convolution operation and shape transformation is transposed and multiplied with the feature matrix obtained after one 3×3 convolution operation and shape transformation. The softmax function is then used to obtain the first spatial attention matrix. At the same time, the feature matrix obtained after one 1×1 convolution operation and shape transformation is transposed and multiplied with the feature matrix obtained after one 1×1 convolution operation and shape transformation. The softmax function is then used to obtain the second spatial attention matrix. Finally, the first spatial attention matrix and the second spatial attention matrix are added together and divided by 2 to obtain the spatial attention matrix.

[0044] Next, the feature matrix after a 1×1 convolution operation and morphological transformation is multiplied with the spatial attention matrix, and then the spatial feature map is obtained through morphological transformation.

[0045] Finally, the spatial feature map is added to the feature map extracted by the residual network, and a 1×1 convolution operation is performed to obtain the spatial attention feature map.

[0046] ② When performing dual-weighted channel attention extraction:

[0047] First, the feature matrix obtained by morphological transformation of the feature map extracted by the residual network is multiplied with the feature matrix obtained by morphological transformation and transpose of the feature map extracted by the residual network. Then, the softmax function is used to obtain the first channel attention matrix. Simultaneously, the feature map extracted by the residual network is max-pooled, and the feature matrix obtained by morphological transformation of the max-pooled feature map extracted by the residual network is multiplied with the feature matrix obtained by morphological transformation and transpose of the max-pooled feature map extracted by the residual network. Then, the softmax function is used to obtain the second channel attention matrix. Finally, the first channel attention matrix and the second channel attention matrix are added together and divided by 2 to obtain the channel attention matrix.

[0048] Then, the feature matrix obtained by morphological transformation of the feature map extracted by the residual network is multiplied with the channel attention matrix, and the channel feature map is obtained by morphological transformation.

[0049] Finally, the channel feature map is added to the feature map extracted by the residual network, and a 1×1 convolution operation is performed to obtain the channel attention feature map.

[0050] Step 6.3: Add the features of the spatial attention feature map and the channel attention feature map to obtain the dual attention feature map.

[0051] In step 8 above, each upsampling operation of the attention map needs to be implemented by sequentially performing a 3×3 convolution operation, a ReLU activation operation, and a 2x upsampling operation.

[0052] In steps 11 and 13 above, each upsampling operation of the channel stitching image needs to be implemented by sequentially performing a 1×1 convolution operation, a 3×3 convolution operation, a batch normalization operation, a ReLU activation operation, and a 2x upsampling operation.

[0053] Compared to existing technologies, this invention uses a weighted channel attention module to separate attention feature channels with different weights. Subsequently, two different feature extraction modules are used to extract object feature information from high-weight and low-weight feature maps respectively. A deformable self-attention Transformer is used to extract features from the high-weight feature map. A deformable self-attention window module transforms the segmented image blocks into irregular shapes adapted to the target shape, then fills them into square image blocks before feeding them into the Transformer. This allows the self-attention to focus more on the more critical regions in the feature map and acquire more feature information. A dual-weight dual-attention module is used to extract features from the low-weight feature map. By acquiring and aggregating spatial and channel attention at different scales of the feature map, spatial and channel attention can better capture the spatial and channel information of the feature map. This results in lower memory and computational costs for semantic segmentation and improved segmentation accuracy. Attached Figure Description

[0054] Figure 1 This is a schematic diagram illustrating the principle of an attention-guided, deformable self-attention semantic segmentation method.

[0055] Figure 2 This is a schematic diagram of the channel attention module with weight separation.

[0056] Figure 3 This is a schematic diagram of the principle of a deformable self-attention window module.

[0057] Figure 4 This is a schematic diagram of the Transformer layer in a Transformer encoder.

[0058] Figure 5 This is a schematic diagram illustrating the principle of the Transformer decoder.

[0059] Figure 6 This is a schematic diagram of the principle of a dual-weighted spatial attention module.

[0060] Figure 7 This is a schematic diagram illustrating the principle of a dual-weighted channel attention module. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific examples.

[0062] An attention-guided, deformable, self-attention-based semantic segmentation method, such as... Figure 1 As shown, the specific steps include the following:

[0063] (1) Feature extraction.

[0064] Step 1: Preprocess the original image by scaling its height and width to the same size. In this example, the input image is uniformly scaled to 1024(h) × 1024(w).

[0065] Step 2: Perform feature extraction twice on the image obtained in Step 1 to obtain the feature map after one feature extraction (first feature map) and the feature map after two feature extractions (second feature map).

[0066] In the first feature extraction, the preprocessed image first undergoes a convolution operation with a kernel of 3×3, padding of 1, stride of 1, and 3 output channels; then it undergoes a convolution operation with a kernel of 1×1, padding of 0, stride of 1, and 64 output channels; next, it undergoes a ReLU activation operation; and finally, it undergoes a 2×2 max pooling operation with a stride of 2 to obtain the first feature map.

[0067] In the second feature extraction, the first feature map obtained from the first feature extraction is first subjected to a convolution operation with a kernel of 3×3, padding of 1, stride of 1, and output channels of 64; then it is subjected to a convolution operation with a kernel of 1×1, padding of 0, stride of 1, and output channels of 128; then it is subjected to a ReLU activation operation; and finally it is subjected to a 2×2 max pooling operation with a stride of 2 to obtain the second feature map.

[0068] In this embodiment, after feature extraction, the original image has a first feature map of 512(h)×512(w)×64(c) and a second feature map of 256(h)×256(w)×128(c). h represents the height of the image, w represents the width of the image, and c represents the number of channels of the image.

[0069] (2) Compression and excitation of weight separation, see [link to relevant documentation] Figure 2 .

[0070] Step 3: By generating attention weights for the feature maps obtained from the two feature extractions in Step 2, the important features of the feature maps are enhanced to obtain weighted feature maps with important features.

[0071] See Figure 2 The attention weight generation process is as follows:

[0072] First, the 256(h)×256(w)×128(c) feature map (second feature map) x input to the channel attention module is subjected to global average pooling. The pooling size is the same as the input feature map size, which is 256(h)×256(w). After pooling, a 1×1×c (c=128) feature map is obtained.

[0073] Next, to train the weights using the correlation between channels, two fully connected operations are performed. The first fully connected operation compresses c channels into cr (r is a hyperparameter representing the compression ratio) channels to reduce computation. The second fully connected operation restores the number of channels to c. ReLU and Sigmoid activation operations are then performed after these two fully connected operations, resulting in a 1×1×c (c=128) attention weight s.

[0074] Finally, the feature map x (second feature map) input to the channel attention module and the generated attention weights s are multiplied by a full multiplication operation, that is, each value in the channel is multiplied by the weight, to obtain the weighted feature map x′ with important features, as shown in formula (1):

[0075] x′=F scale (x c ,s c )=x c ·s c (1)

[0076] Where, x c Let s be the c-th two-dimensional matrix in feature map x. c This represents the c-th learned weight. The index c represents the number of channels.

[0077] Step 4: Set the pixel values ​​of the feature channels with low attention weights in the feature map with important feature weights obtained in Step 3 to 0, thereby obtaining the high-weight feature map x1′. At the same time, set the pixel values ​​of the feature channels with high attention weights in the feature map with important feature weights obtained in Step 3 to 0, thereby obtaining the low-weight feature map x2′.

[0078] The average attention weight is obtained by adding the highest and lowest attention weights of each feature channel of the feature map with important feature weights and then dividing by 2. If the attention weight of a feature channel of the feature map with important feature weights is less than or equal to the average attention weight, it is a feature channel with low attention weight. If the attention weight of a feature channel of the feature map with important feature weights is greater than the average attention weight, it is a feature channel with high attention weight.

[0079] (3) Deformable Self-Attention Transformer

[0080] Step 5: Pass the high-weight feature map x1′ obtained in Step 4 through a deformable self-attention Transformer to obtain a feature map with rich global context information.

[0081] Step 5.1: Divide the high-weighted graph x1′ obtained in Step 4 into 256 segments of size 1. A square image block.

[0082] Step 5.2: Perform a deformable self-attention window operation on each original image patch obtained in Step 5.1 to obtain deformable image patches. See [link / reference] Figure 3 .

[0083] First, each original image patch is subjected to a depthwise convolution with a 3×3 kernel, 1 padding, a stride of 1, and 128 output channels to capture local features. Then, a GELU activation operation and a 1×1 convolution with 0 padding, a stride of 1, and 2 output channels are used to obtain the offset of each pixel in each image patch, where the offset of each pixel includes the offset in the x-direction and the offset in the y-direction.

[0084] Next, the contour reference points of each original image block are offset.

[0085] Each original image block has nine contour reference points: the top left pixel, the top right pixel, the bottom left pixel, the bottom right pixel, the center pixel, the center pixel of the top row, the center pixel of the bottom row, the center pixel of the leftmost column, and the center pixel of the rightmost column.

[0086] The position offset of each contour reference point in each original image block is achieved by adding the coordinates of the contour reference point to the offset of the contour reference point.

[0087] Then, for each image block, the new contour of the original image block is obtained by using the nine offset contour reference points, and bilinear interpolation is used to fill the pixel values ​​of the pixels in the new contour.

[0088] Finally, each irregular image patch is filled with pixels of size 0 using pixels with a pixel value of 0. Square image blocks.

[0089] Step 5.3: In order to use the image patch as input to the Transformer, perform a linear mapping operation on each deformed image patch obtained in step 5.2, mapping the dimension of each image patch to a specified 1×1024 dimension.

[0090] Step 5.4: In order for the model to know the location information of the image blocks in the sequence data, a learnable location encoding vector is added to each deformed image after linear mapping obtained in step 5.3 to obtain image blocks with location information.

[0091] Step 5.5: Input each image block with positional information obtained in step 5.4 into the Transformer encoder to obtain the self-attention perceptual feature vector of each image block with positional information.

[0092] A Transformer encoder consists of N standard Transformer layers. Each standard Transformer layer comprises layer normalization, a multi-head self-attention layer, a multilayer perceptron, and residual connections. Figure 4 As shown. In this invention, the Transformer layer has 6 layers. Image blocks with location information are obtained through a Transformer encoder. Each 1×1024 self-attention-aware feature vector.

[0093] Step 5.6: Input all the self-attention-aware feature vectors obtained in Step 5.5 into the Transformer decoder to obtain a feature map with rich global context information.

[0094] The structure of the Transformer decoder is as follows: Figure 5 As shown. In the Transformer layer: First, the output of the Transformer encoder in step 5.5 is... The 1×1024 self-attention-aware feature vectors are merged and their shapes are changed to... The feature map is then processed. Next, a convolution with a kernel of 1×1, padding of 0, stride of 1, input channels of 1024, and output channels of 256, and a 2x upsampling operation is performed to obtain a feature map with rich global contextual information.

[0095] (4) Dual-weighted dual attention

[0096] Step 6: Perform a dual-weight dual-attention operation on the low-weight feature map x2′ obtained in Step 4 to obtain a dual-attention feature map.

[0097] Step 6.1: Input the low-weight feature map x2′ into a residual network (ResNet-50) using a dilated convolution strategy for feature extraction, thereby reducing the size of the feature map to one-eighth of the input image. Let A be the feature map x2′ that is 32(h)×32(w)×2048(c) obtained by the residual network using the dilated convolution strategy.

[0098] Step 6.2: Simultaneously perform dual-weighted spatial attention extraction and dual-weighted channel attention extraction on feature map A to obtain spatial attention feature map and channel attention feature map, respectively.

[0099] See Figure 6 The spatial attention extraction process with dual weights is as follows:

[0100] First, the feature map A obtained from the residual network using the dilated convolution strategy is subjected to 5 parallel convolution operations: feature map A undergoes 3 parallel convolution operations with a 1×1 kernel, 0 padding, a stride of 1, 2048 input channels, and 512 output channels to obtain 3 feature maps A1, A2, and A3; feature map A undergoes 2 parallel convolution operations with a 3×3 kernel, 1 padding, a stride of 1, 2048 input channels, and 512 output channels to obtain 2 feature maps A4 and A5. Then, the convolutional feature maps A1, A2, A3, A4, and A5 are morphologically transformed into a two-dimensional feature matrix (c×n), where n = h×w. The feature map to matrix morphological transformation converts the number of channels c of the feature map into the number of rows c of the matrix, and the product of the feature map dimensions h×w into the number of columns n of the matrix. Each row of the matrix contains the feature map data. Figure 1 The value of each channel. The morphological transformation from matrix to feature map transforms each row of the matrix into a channel of the feature map.

[0101] Then, multiply the transpose (n×c) of the feature matrix after transformation A1 with the feature matrix after transformation A2 (c×n), and then apply the softmax function to obtain the spatial attention matrix S1 (n×n). Simultaneously, multiply the transpose (n×c) of the feature matrix after transformation A4 with the feature matrix after transformation A5 (c×n), and then apply the softmax function to obtain the spatial attention matrix S2 (n×n). Finally, add the spatial attention matrices S1 and S2 and divide by 2 to obtain the spatial attention matrix S.

[0102] Next, the feature matrix (c×n) after A3 transformation and the transpose (n×n) of the spatial attention matrix S are multiplied by matrix, and then morphological transformation is performed to obtain feature map B;

[0103] Finally, feature map B is added to feature map A, and a convolution with a kernel of 1×1, padding of 0, stride of 1, and output channels of 256 is used to perform channel dimensionality reduction to obtain the spatial attention feature map.

[0104] See Figure 7 The dual-weighted channel attention extraction process is as follows:

[0105] First, the feature map A obtained through the residual network using the dilated convolution strategy undergoes a morphological transformation to obtain a feature matrix (c×n). Simultaneously, feature map A is transformed and transposed to obtain a feature matrix (n×c). The two feature matrices are then multiplied together, and the softmax function is used to obtain the channel attention matrix X1 (c×c). Next, feature map A is subjected to max pooling with a pooling size of 3×3, padding of 1, stride of 2, and dilation of 1, halving the feature matrix size. The feature matrix obtained by morphological transformation of the max-pooled feature map is then described. The feature matrix obtained by morphological transformation and transpose of the feature map after max pooling. After multiplying the two matrices, the softmax function is used to obtain the channel attention matrix X2 (c×c). Then, X1 and X2 are added together and divided by 2 to obtain the channel attention matrix X.

[0106] Then, the feature matrix (c×n) after the feature map A is transformed and the transpose (c×c) of the channel attention matrix X are multiplied by matrix multiplication and morphological transformation to obtain feature map C.

[0107] Finally, feature map C is added to feature map A, and channel dimensionality reduction is performed using a convolution with a kernel of 1×1, padding of 0, stride of 1, and output channels of 256 to obtain the channel attention feature map.

[0108] Step 6.3: Add the output feature maps of the spatial attention module and the spatial attention module to obtain an attention feature map with 256 channels and a size one-eighth of the feature map x1′. Summarizing the output attention feature maps after passing through two attention modules yields better pixel-level predictive feature representations.

[0109] (5) Decoding

[0110] Step 7: Add the feature map with rich global context information obtained in Step 5 and the dual attention feature map obtained in Step 6 to obtain the attention feature map.

[0111] Step 8: Perform three attention map upsampling operations on the attention feature map obtained in Step 7 to obtain the upsampled attention feature map.

[0112] This invention requires the attention feature map to be upsampled three times sequentially. Each upsampling of the attention map requires first undergoing a convolution operation with a kernel size of 3×3, padding of 1, and stride of 1; then undergoing a ReLU activation operation; and finally undergoing a 2x upsampling operation.

[0113] Step 9: Perform a convolution operation on the upsampled attention feature map obtained in Step 8 to obtain the feature map after channel reduction.

[0114] The convolution operation has a 1×1 kernel, 0 padding, a stride of 1, 256 input channels, and 128 output channels.

[0115] Step 10: Perform channel splicing operation on the feature map obtained in step 9 and the second feature map obtained in step 2 to obtain the first channel spliced ​​feature map.

[0116] Step 11: Perform an upsampling operation on the first channel stitching feature map obtained in Step 10 to obtain the first upsampled channel stitching feature map.

[0117] When upsampling the channel mosaic, it first undergoes a convolution operation with a kernel of 1×1, padding of 0, stride of 1, input channels of 256, and output channels of 64; then it undergoes a convolution operation with a kernel of 3×3, padding of 1, stride of 1, and output channels of 64; next, it undergoes batch normalization and ReLU activation operations; finally, it undergoes a 2x upsampling.

[0118] Step 12: Perform channel stitching operation between the first upsampled channel stitched feature map obtained in Step 11 and the first feature map obtained in Step 2 to obtain the second channel stitched feature map.

[0119] Step 13: Perform an upsampling operation on the second channel stitching feature map obtained in Step 12 to obtain the second upsampled channel stitching feature map.

[0120] When upsampling the channel mosaic, it first undergoes a convolution operation with a kernel of 1×1, padding of 0, stride of 1, input channels of 128, and output channels of 32; then it undergoes a convolution operation with a kernel of 3×3, padding of 1, stride of 1, and output channels of 32; next, it undergoes batch normalization and ReLU activation operations; finally, it performs a 2x upsampling.

[0121] Step 14: Perform a convolution operation on the second upsampled channel spliced ​​feature map obtained in Step 13 to obtain the segmentation result image.

[0122] The convolution operation has a 1×1 kernel, 0 padding, a stride of 1, 32 input channels, and an output channel equal to the number of classes in the dataset.

[0123] It should be noted that although the embodiments described above are illustrative, they are not intended to limit the invention. Therefore, the invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.

Claims

1. An attention-guided, deformable self-attention semantic segmentation method, characterized in that, The steps include the following: Step 1: Preprocess the original image by scaling its height and width to the same size. Step 2: Perform feature extraction operations twice on the image obtained in Step 1 to obtain the feature map after one feature extraction and the feature map after two feature extractions, respectively. Step 3: Generate attention weights for the feature maps obtained from the two feature extractions in Step 2, and enhance the important features of the feature maps to obtain weighted feature maps with important features; Step 4: Set the pixel values ​​of the feature channels with low attention weights in the feature map with important feature weights obtained in Step 3 to 0, thereby obtaining a high-weight feature map; at the same time, set the pixel values ​​of the feature channels with high attention weights in the feature map with important feature weights obtained in Step 3 to 0, thereby obtaining a low-weight feature map. Step 5: Pass the high-weight feature map obtained in Step 4 through a deformable self-attention Transformer operation to obtain a feature map with rich global context information. Step 6: Perform a dual-weight dual-attention operation on the low-weight feature map obtained in Step 4 to obtain a dual-attention feature map; Step 7: Add the feature map with rich global context information obtained in Step 5 and the dual attention feature map obtained in Step 6 to obtain the attention feature map; Step 8: Perform three attention map upsampling operations on the attention feature map obtained in Step 7 to obtain the upsampled attention feature map; Step 9: Perform a convolution operation on the upsampled attention feature map obtained in Step 8 to obtain the feature map after channel reduction; Step 10: Perform channel concatenation on the feature map obtained in Step 9 and the feature map obtained in Step 2 after two feature extractions to obtain the first channel concatenated feature map. Step 11: Perform an upsampling operation on the first channel stitching feature map obtained in Step 10 to obtain the first upsampled channel stitching feature map; Step 12: Perform channel stitching operation on the first upsampled channel stitched feature map obtained in Step 11 and the feature map after the first feature extraction obtained in Step 2 to obtain the second channel stitched feature map. Step 13: Perform an upsampling operation on the second channel stitching feature map obtained in Step 12 to obtain the second upsampled channel stitching feature map; Step 14: Perform a convolution operation on the second upsampled channel spliced ​​feature map obtained in Step 13 to obtain the segmentation result image.

2. The attention-guided deformable self-attention semantic segmentation method according to claim 1, characterized in that, In step 2, each feature extraction operation requires a 3×3 convolution operation, a 1×1 convolution operation, a ReLU activation operation, and a max pooling operation in sequence.

3. The attention-guided deformable self-attention semantic segmentation method according to claim 1, characterized in that, The specific process of step 3 is as follows: Step 3.1: Perform global average pooling on the feature maps after the two feature extractions to obtain a feature map with c channels; Step 3.2: Perform two fully connected operations on the feature map with c channels, and train the weights using the correlation between channels to obtain the attention weights; The first fully connected layer compresses c channels into c / r channels to reduce computation, and is followed by a ReLU activation operation. The second fully connected layer restores the number of channels to c, and is followed by a Sigmoid activation operation. Here, r is the compression ratio. Step 3.3: Perform a full multiplication operation on the feature maps and attention weights after the two feature extractions to obtain a weighted feature map with important features.

4. The attention-guided deformable self-attention semantic segmentation method according to claim 1, characterized in that, In step 4, the highest and lowest attention weights of each feature channel of the feature map with important feature weights are added together and then divided by 2 to obtain the average attention weight. If the attention weight of a feature channel in a feature map with important feature weights is less than or equal to the average attention weight, then it is a feature channel with low attention weight. If the attention weight of a feature channel in a feature map with important feature weights is greater than the average attention weight, then it is a feature channel with high attention weights.

5. The attention-guided deformable self-attention semantic segmentation method according to claim 1, characterized in that, The specific process of step 5 is as follows: Step 5.1: First, divide the high-weight feature map into uniform original image blocks; Step 5.2: Perform a deformable self-attention window operation on each original image patch obtained in Step 5.1 to obtain deformable image patches; that is: First, perform a depthwise convolution operation on each original image block to capture local features, and then use GELU activation and convolution operations to obtain the offset of each pixel in each image block; Then, the coordinates of each contour reference point of each original image block are added with its offset, so that the position of each contour reference point is shifted, and the shifted contour reference point is used as the new contour of the original image block. Then, bilinear interpolation is used to fill the pixels in the new contour of each original image block with pixel values ​​to obtain an irregular image block. Then, pixels with a pixel value of 0 are used to fill the irregular image block to the same size as the original image block to obtain a deformed image block. Step 5.3: Perform linear mapping on each deformed image block obtained in Step 5.2, so that the dimension of each deformed image block is mapped to the specified dimension; Step 5.4: Add a learnable positional encoding vector to each deformed image obtained after linear mapping in step 5.3 to obtain an image patch with positional information; Step 5.5: Input each image block with positional information obtained in Step 5.4 into the Transformer encoder to obtain the self-attention perceptual feature vector of each image block with positional information; Step 5.6: Input all the self-attention-aware feature vectors obtained in Step 5.5 into the Transformer decoder to obtain a feature map with rich global context information.

6. The attention-guided deformable self-attention semantic segmentation method according to claim 5, characterized in that, In step 5.2, each image block includes 9 contour reference points, namely the top left pixel, the top right pixel, the bottom left pixel, the bottom right pixel, the center pixel, the center pixel of the top row, the center pixel of the bottom row, the center pixel of the leftmost column, and the center pixel of the rightmost column.

7. The attention-guided deformable self-attention semantic segmentation method according to claim 1, characterized in that, The specific process of step 6 is as follows: Step 6.1: Use a residual network with dilated convolution strategy to extract features from the low-weight feature map; Step 6.2: Simultaneously perform dual-weighted spatial attention extraction and dual-weighted channel attention extraction on the feature maps extracted by the residual network to obtain spatial attention feature maps and channel attention feature maps, respectively. ① When performing spatial attention extraction with two weights: First, the feature map extracted by the residual network is subjected to five parallel convolution operations, including two 3×3 convolution operations and three 1×1 convolution operations; then, the convolution feature map obtained from the five parallel convolution operations is subjected to morphological transformation to obtain a two-dimensional feature matrix. Then, the feature matrix obtained after one 3×3 convolution operation and shape transformation is transposed and multiplied with the feature matrix obtained after one 3×3 convolution operation and shape transformation. The softmax function is then used to obtain the first spatial attention matrix. At the same time, the feature matrix obtained after one 1×1 convolution operation and shape transformation is transposed and multiplied with the feature matrix obtained after one 1×1 convolution operation and shape transformation. The softmax function is then used to obtain the second spatial attention matrix. Finally, the first spatial attention matrix and the second spatial attention matrix are added together and divided by 2 to obtain the spatial attention matrix. Next, the feature matrix after a 1×1 convolution operation and morphological transformation is multiplied with the spatial attention matrix, and then the spatial feature map is obtained through morphological transformation. Finally, the spatial feature map is added to the feature map extracted by the residual network, and a 1×1 convolution operation is performed to obtain the spatial attention feature map. ② When performing dual-weighted channel attention extraction: First, the feature matrix obtained by morphological transformation of the feature map extracted by the residual network is multiplied with the feature matrix obtained by morphological transformation and transpose of the feature map extracted by the residual network. Then, the softmax function is used to obtain the first channel attention matrix. Simultaneously, the feature map extracted by the residual network is max-pooled, and the feature matrix obtained by morphological transformation of the max-pooled feature map extracted by the residual network is multiplied with the feature matrix obtained by morphological transformation and transpose of the max-pooled feature map extracted by the residual network. Then, the softmax function is used to obtain the second channel attention matrix. Finally, the first channel attention matrix and the second channel attention matrix are added together and divided by 2 to obtain the channel attention matrix. Then, the feature matrix obtained by morphological transformation of the feature map extracted by the residual network is multiplied with the channel attention matrix, and the channel feature map is obtained by morphological transformation. Finally, the channel feature map is added to the feature map extracted by the residual network, and a 1×1 convolution operation is performed to obtain the channel attention feature map. Step 6.3: Add the features of the spatial attention feature map and the channel attention feature map to obtain the dual attention feature map.

8. The attention-guided deformable self-attention semantic segmentation method according to claim 1, characterized in that, In step 8, each upsampling operation of the attention map needs to be implemented by sequentially performing a 3×3 convolution operation, a ReLU activation operation, and a 2x upsampling operation.

9. The attention-guided deformable self-attention semantic segmentation method according to claim 1, characterized in that, In steps 11 and 13, each upsampling operation of the channel stitching image needs to be implemented by sequentially performing a 1×1 convolution operation, a 3×3 convolution operation, a batch normalization operation, a ReLU activation operation, and a 2x upsampling operation.

Images