Image semantic segmentation method based on multi-scale linear attention and U-shaped structure

Abstract

An image semantic segmentation method based on multi-scale linear attention and a U-shaped structure belongs to the technical field of image segmentation and deep learning, and comprises the following steps: inputting an image into an encoder, and carrying out local information extraction by adopting convolution stem; an encoder is constructed, and a pyramid structure is adopted to use Transform Block to calculate multi-scale features across four stages; constructing a decoder which comprises four stages, and combining features of each stage of the encoder and the decoder by using jump connection; and performing up-sampling operation on the output of the fourth stage of the decoder, recovering the resolution to the input resolution, and converting the feature map into a segmentation mask by using a linear layer. According to the invention, a multi-scale linear attention mechanism technology is integrated into a U-row encoder decoder architecture, so that the calculation cost is reduced, and the receptive field interaction and segmentation precision is improved. In the decoder, a CARAFE layer is adopted for up-sampling, so that complex details can be reserved, and the accuracy of organ edge segmentation can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation and deep learning technology, and in particular relates to an image semantic segmentation algorithm based on a Transformer network framework. Background Art

[0002] Image segmentation is a technique for classifying visual content pixel by pixel. Its core task is to assign each pixel in an image to its corresponding target category. Within this conceptual framework, semantic segmentation, as a subtask, focuses on achieving pixel-level semantic understanding.

[0003] Image segmentation, a core technology branch in the field of computer vision, is crucial for endowing machines with human-like visual parsing capabilities. By building pixel-level semantic understanding models, this technology enables computers to accurately identify the semantic attribution of each pixel in a visual scene, thereby reconstructing a visual parsing picture that matches human cognition. In dynamic scenarios such as autonomous driving, semantic segmentation systems, through real-time pixel classification, can not only identify the presence of vehicles, pedestrians, traffic signs, and other objects on the road, but also accurately outline their spatial distribution characteristics, providing fine-grained environmental modeling for decision-making systems.

[0004] Throughout the evolution of image segmentation technology, traditional convolutional neural networks (CNNs) have demonstrated excellent performance in capturing local texture features and the correlation between adjacent regions of an image, leveraging their local receptive field properties and hierarchical feature extraction mechanisms. However, when faced with high-resolution and complex scenes, traditional architectures are unable to adequately model global semantic relationships, directly limiting their ability to understand cross-region semantic consistency.

[0005] To overcome the limitations of CNN, the full-image modeling capability of the self-attention mechanism of the visual Transformer architecture effectively solves the structural defects of traditional methods in capturing long-range semantic associations by establishing a global dependency map between pixels. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an image semantic segmentation method based on multi-scale linear attention and U-shaped structure. In view of the problem that traditional convolutional neural networks have insufficient global semantic modeling capabilities when processing high-resolution images, a multi-scale linear attention mechanism is introduced to enhance the global context modeling capabilities while maintaining hardware efficiency.

[0007] The image semantic segmentation method based on multi-scale linear attention and U-shaped structure includes the following steps, which are performed in sequence:

[0008] Step 1: Input the image into the encoder and use convolutional stem to extract local information;

[0009] Step 2: Build an encoder using a pyramid structure across four stages using Transformer Block to calculate multi-scale features;

[0010] Step 3: Construct a decoder, which consists of four stages and uses skip connections to merge the features of each stage of the encoder and decoder;

[0011] Step 4: Upsample the output of the fourth stage of the decoder to restore the resolution to the input resolution, and use a linear layer to convert the feature map into a segmentation mask.

[0012] The method for extracting local information using the convolutional stem in step 1 is:

[0013] The convolution kernel size is 7×7 and the step size is 4. The efficient spatial dimension reduction and wide-area feature extraction of tokens and channels C provide a foundation of high semantic information for subsequent deep networks, while balancing computational cost and model performance.

[0014] The encoder constructed in step 2 consists of four stages, each of which generates a dimension of feature map, where i∈{1.2,3.4}; H, W and D i are the height of the image, the width of the image, and the number of channels after each stage i.

[0015] The pyramid structure described in step 2 is obtained by downsampling the resolution of the feature map by performing a 3*3 convolution with a stride of 2 at each stage, reducing the resolution to half the input size and doubling the number of channels.

[0016] The TransformerBlock described in step 2 consists of a multi-scale linear attention module and an FFN with deep convolution. The multi-scale linear attention is used to capture contextual information, and the FFN with deep convolution is used to capture local information. After obtaining the Q, K, and V tokens through the linear projection layer, the multi-scale tokens are generated by aggregating nearby tokens through lightweight small-kernel convolution. ReLU linear attention is applied to the multi-scale tokens, and the outputs are connected and fed to the final linear projection layer for feature fusion.

[0017] Each stage of the decoder in step 3 consists of two parts, one is obtained by upsampling, and the other is obtained by copying the decoder features and performing center cropping.

[0018] The upsampling adopts the CARAFE layer, which includes a kernel prediction module and a content-aware reconstruction module; the kernel prediction module starts from a convolutional layer and predicts the reconstructed kernel based on the encoding features; the kernel prediction module includes three submodules: a channel compressor, a context encoder, and a kernel normalizer.

[0019] The channel compressor is used to reduce the dimension of the channel space in the input feature map X∈R H×W×C , where R represents the real space and C represents the number of channels, which reduces the computational complexity and focuses on the basic feature information; the context encoder is used to process the simplified feature map, encode the context information and generate the recombinant kernel; the kernel normalizer is used to normalize the recombinant kernel through the softmax function in the kernel normalizer to ensure that the output distribution of the weight is probabilistic and the sum is 1; through the upsampling ratio σ, the CARAFE layer generates an extended feature map X'∈R σH×σW×C , where each pixel l'=(i',j') in X' corresponds to a specific pixel l=(i,j) in X, given by and Get; the kernel prediction module ψ is based on N(X l ,k) predict a unique reconstruction kernel W' for each pixel l' l , a k×k region centered on pixel l on X; extract local features from the k×k neighborhood of pixel l=(i,j) in the input feature map X, and predict the kernel local features to effectively reorganize and upsample the feature map; expressed as:

[0020] W′ l =ψ(N(X l ,K encoder ))

[0021] Where: K encoder Represents the receptive field of the content encoder.

[0022] The content-aware reassembly module uses convolutional layers to reassemble input features, and the content-aware reassembly module φ uses the reassembly kernel W' l Reassemble N(X l ,k):

[0023] X′ l =φ(N(X l ,k up ),W′ l )

[0024] Where: k up is the size of the reorganization kernel, for each reorganization kernel W' l , the content-aware reassembly module reassembles the features in the local release area; the module φ performs weighted summation;

[0025] The pixel position l and its central neighborhood N(X l ,k up ), the reorganization process is:

[0026]

[0027] N(X l ,k up ) contributes differently to the upsampled pixel l', and the reassembled feature map is used to enhance attention to relevant information in the local area and provide semantic information that is more robust than the original feature map.

[0028] The above design scheme achieves the following beneficial effects: Based on a multi-scale linear attention and U-shaped structure image semantic segmentation method, the multi-scale linear attention mechanism is integrated into the U-row encoder-decoder architecture. This integration not only reduces computational costs but also improves receptive field interaction and segmentation accuracy. In the decoder, the use of the CARAFE layer for upsampling helps preserve complex details and improves the accuracy of organ edge segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0030] Figure 1 This is a framework diagram of the image semantic segmentation algorithm based on multi-scale linear attention and U-shaped structure of the present invention;

[0031] Figure 2 Schematic diagram of the Transformer Block module structure in the present invention;

[0032] Figure 3 Schematic diagram of the multi-scale linear attention structure in this invention. DETAILED DESCRIPTION

[0033] To make the objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the present invention are described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the present invention is not limited to the following embodiments, and specific implementation methods can be determined based on the technical solutions of the present invention and actual conditions. To avoid obscuring the essence of the present invention, well-known methods, processes, and procedures are not described in detail.

[0034] This paper addresses the problem of traditional convolutional neural networks' insufficient global semantic modeling capabilities when processing high-resolution images. By introducing a multi-scale linear attention mechanism, this method enhances global context modeling capabilities while maintaining hardware efficiency. We propose an image semantic segmentation algorithm based on multi-scale linear attention and a U-shaped architecture. Figure 1The framework diagram of the image semantic segmentation algorithm based on multi-scale linear attention and U-shaped structure of the present invention is shown. The specific implementation process of this method includes the following steps:

[0035] A. When the image is input into the encoder, the multi-level small-kernel convolutional architecture achieves progressive aggregation of contextual information by superimposing local receptive fields, maintaining fine spatial structural features during spatial downsampling operations; its unique channel dimension expansion strategy effectively improves the expressive power of the low-dimensional feature space, while the hierarchical feature mapping mechanism provides semantically rich input features for the subsequent Transformer module, significantly reducing the computational burden of the self-attention mechanism in processing long-distance dependencies.

[0036] Specifically, the convolution kernel size is 7×7 and the step size is 4 to obtain The efficient spatial dimension reduction and wide-area feature extraction of tokens and channels C provide a foundation of high semantic information for subsequent deep networks, while balancing computational cost and model performance.

[0037] B. The encoder uses a pyramid structure to calculate multi-scale features using Transformer Block across four stages, and each stage generates a dimension of feature map, where i∈{1.2,3.4}. H, W and D i are the height of the image, the width of the image, and the number of channels after each stage i. The resolution of the feature map is downsampled by performing a 3*3 convolution with a stride of 2 at each stage, reducing the resolution to half the input size while doubling the number of channels. To obtain a pyramid structure, the resolution of the feature map is downsampled by performing a 3*3 convolution with a stride of 2 at each stage, reducing the resolution to half the input size while doubling the number of channels.

[0038] C. The decoder consists of four stages, each of which consists of two parts, one of which is obtained by upsampling and the other by copying the decoder features and performing center cropping. Like UNet, skip connections are used to merge the features of each stage of the encoder and decoder to better preserve contextual information.

[0039] D. Upsample the output of the fourth stage in step C) to restore the resolution to the input resolution and use a linear layer to convert the feature map into a segmentation mask.

[0040] E. In step B, the Transformer Block consists of a multi-scale linear attention module and an FFN with depthwise convolution. The multi-scale linear attention is responsible for capturing contextual information, and the FFN with depthwise convolution is responsible for capturing local information. After obtaining the Q, K, and V tokens through the linear projection layer, multi-scale tokens are generated by aggregating nearby tokens through lightweight small-kernel convolution. ReLU linear attention is applied to the multi-scale tokens, and then the outputs are connected and fed into the final linear projection layer for feature fusion. Specifically: In the ReLU linear attention, the similarity function is defined as Sim(Q,K)=ReLU(Q)ReLU(K) T , Enhance ReLU linear attention through convolution, that is, use deep convolution in the FFN layer to solve the limitation of not being able to generate focused attention. Aggregate the information of nearby Q, K, V tokens to obtain multi-scale tokens to enhance the multi-scale learning ability of ReLU linear attention. This information aggregation process is performed independently for each Q, K, and V in each head. Only small kernel depthwise separable convolution is used for information aggregation. Specifically, all DWConvs are fused into a single DWConv, and all 1x1 Convs are merged into a single 1x1 group convolution, where the number of groups is 3×heads and the number of channels in each group is d. After obtaining multi-scale tokens, ReLU linear attention is performed on them to extract multi-scale global features. Finally, we concatenate the features along the head dimension and feed them into the final linear projection layer to fuse the features.

[0041] F. In step C), the upsampling adopts the CARAFE layer, which mainly consists of two main components: kernel prediction module and content-aware reconstruction module. The kernel prediction module starts from a convolutional layer and its task is to predict the reconstruction kernel based on the encoded features. It consists of three submodules: channel compressor, context encoder and kernel normalizer. The channel compressor reduces the dimensionality of the channel space in the input feature map (X∈R H×W×C ), thereby reducing computational complexity and focusing on essential feature information. After channel compression, the context encoder processes the simplified feature map to encode contextual information, which helps generate the reconstructed kernel. Each predicted reconstructed kernel is normalized by the softmax function in the kernel normalizer to ensure that the output distribution of weights is probabilistic and sums to 1, thereby enhancing the stability and performance of the upsampling process. With the upsampling ratio σ, CARAFE aims to generate an expanded feature map X'∈R σH×σW×C For each pixel l'=(i',j') in X', it corresponds to a specific pixel l=(i,j) in X, given by and The kernel prediction module ψ is based on N(X l ,k) predict a unique reconstruction kernel W' for each pixel l' l, which is a k×k region centered on pixel l on X. This neighborhood extracts local features, and the prediction kernel uses these features to effectively reorganize and upsample the feature map. Specifically, it can be expressed as:

[0042] W′ l =ψ(N(X l ,k encoder ))

[0043] where k encoder Represents the receptive field of the content encoder.

[0044] This is followed by content-aware reassembly, where the input features are reassembled using convolutional layers. The content-aware reassembly module φ uses the reassembly kernel W' l Reassemble N(X l ,k):

[0045] X′ l =φ(N(X l ,k up ),W′ l )

[0046] where k up is the size of the reorganization kernel. For each reorganization kernel W' l , the content-aware reassembly module reassembles the features within the local release region. Module φ performs weighted summation. For a pixel location l and its center neighborhood N(X l ,k up ), the reorganization process is as follows:

[0047]

[0048] N(X l ,k up ) contributes differently to the upsampled pixel l'. The reassembled feature map can enhance attention to relevant information within the local region, providing more robust semantic information than the original feature map. Furthermore, skip connections are used to merge the feature maps output from the encoder and decoder, providing richer and more accurate spatial information and helping to recover image details. Subsequently, a 1×1 convolution kernel is used to reduce the number of channels after cascading to ensure consistency with the number of feature channels during the upsampling process.

Claims

1. An image semantic segmentation method based on multi-scale linear attention and U-shaped structure, characterized by: The method comprises the following steps, and the following steps are performed in sequence: Step 1: Input the image into the encoder and use convolutional stem to extract local information; Step 2: Build an encoder using a pyramid structure across four stages using Transformer Block to calculate multi-scale features; Step 3: Construct a decoder, which consists of four stages and uses skip connections to merge the features of each stage of the encoder and decoder; Step 4: Upsample the output of the fourth stage of the decoder to restore the resolution to the input resolution, and use a linear layer to convert the feature map into a segmentation mask.

2. The image semantic segmentation method based on multi-scale linear attention and U-shaped structure according to claim 1, characterized in that the method of extracting local information by the convolutional stem in step 1 is: The convolution kernel size is 7×7 and the step size is 4. The efficient spatial dimension reduction and wide-area feature extraction of tokens and channels C provide a foundation of high semantic information for subsequent deep networks, while balancing computational cost and model performance.

3. The image semantic segmentation method based on multi-scale linear attention and U-shaped structure according to claim 1 is characterized by: The encoder constructed in step 2 consists of four stages, each of which generates a dimension of feature map, where i∈{1.2,3.4}; H, W and D i are the height of the image, the width of the image, and the number of channels after each stage i.

4. The image semantic segmentation method based on multi-scale linear attention and U-shaped structure according to claim 1 is characterized by: The pyramid structure described in step 2 is obtained by downsampling the resolution of the feature map by performing a 3*3 convolution with a stride of 2 at each stage, reducing the resolution to half the input size and doubling the number of channels.

5. The image semantic segmentation method based on multi-scale linear attention and U-shaped structure according to claim 1, characterized in that: The Transformer Block described in step 2 consists of a multi-scale linear attention module and an FFN with deep convolution. The multi-scale linear attention is used to capture contextual information, and the FFN with deep convolution is used to capture local information. After obtaining the Q, K, and V tokens through the linear projection layer, the multi-scale tokens are generated by aggregating nearby tokens through lightweight small-kernel convolution. ReLU linear attention is applied to the multi-scale tokens, and the outputs are connected and fed to the final linear projection layer for feature fusion.

6. The image semantic segmentation method based on multi-scale linear attention and U-shaped structure according to claim 1, characterized in that: Each stage of the decoder in step 3 consists of two parts, one is obtained by upsampling, and the other is obtained by copying the decoder features and performing center cropping.

7. The image semantic segmentation method based on multi-scale linear attention and U-shaped structure according to claim 6, characterized in that: The upsampling adopts the CARAFE layer, which includes a kernel prediction module and a content-aware reconstruction module; the kernel prediction module starts from a convolutional layer and predicts the reconstructed kernel based on the encoding features; the kernel prediction module includes three submodules: a channel compressor, a context encoder, and a kernel normalizer.

8. The image semantic segmentation method based on multi-scale linear attention and U-shaped structure according to claim 7, characterized in that: The channel compressor is used to reduce the dimension of the channel space in the input feature map X∈R H×W×C , where R represents the real number space and C represents the number of channels, which reduces the computational complexity and focuses on basic feature information; The context encoder is used to process the simplified feature map and encode the context information to generate a reconstructed kernel; the kernel normalizer is used to normalize the reconstructed kernel through the softmax function in the kernel normalizer to ensure that the output distribution of the weight is probabilistic and the sum is 1; through the upsampling ratio σ, the CARAFE layer generates an extended feature map X'∈R σH×σW×C , where each pixel l'=(i',j') in X' corresponds to a specific pixel l=(i,j) in X, given by and Get; the kernel prediction module ψ is based on N(X l ,k) predict a unique reconstruction kernel W' for each pixel l' l , a k×k region centered on pixel l on X; extract local features from the k×k neighborhood of pixel l=(i,j) in the input feature map X, and predict the kernel local features to effectively reorganize and upsample the feature map; expressed as: W′ l ψ(N(X l ,k encoder )) Where: k encoder Represents the receptive field of the content encoder.

9. The image semantic segmentation method based on multi-scale linear attention and U-shaped structure according to claim 6, characterized in that: The content-aware reassembly module uses convolutional layers to reassemble input features, and the content-aware reassembly module φ uses the reassembly kernel W' l Reassemble N(X l ,k): X′ l =φ(M(X l ,k up ),W l ) Where: k up is the size of the reorganization kernel, for each reorganization kernel W' l ,The content-aware reassembly module reassembles the features within the local release region; Module φ performs weighted summation; The pixel position l and its central neighborhood N(X l ,k up ), the reorganization process is: N(X l ,k up ) contributes differently to the upsampled pixel l', and the reassembled feature map is used to enhance attention to relevant information in the local area and provide semantic information that is more robust than the original feature map.

Images