Image semantic segmentation method based on gated attention and multi-scale enhancement

Abstract

The invention discloses an image semantic segmentation method based on gating attention and multi-scale enhancement, and belongs to the field of image semantic segmentation, and the method comprises the steps: building a channel space attention gating sequence building module; constructing a multi-scale feature enhancement block; constructing a gating fusion block; an encoder-decoder network is improved based on a channel space attention gating sequence building module, a multi-scale feature enhancement block and a gating fusion block, and a semantic segmentation neural network is obtained; obtaining an image training set, and training the semantic segmentation neural network by using image training to obtain a trained semantic segmentation neural network; and obtaining a to-be-classified image, and classifying the to-be-classified image by using the trained semantic segmentation neural network. According to the method, the problems that the context information capturing capability is limited, the global modeling capability is relatively weak, the multi-scale fusion lacks a dynamic adaptation mechanism and the jump connection information is redundant in the existing method are solved.

Description

Technical Field

[0001] This invention belongs to the field of image semantic segmentation, and particularly relates to an image semantic segmentation method based on gated attention and multi-scale enhancement. Background Technology

[0002] With the development of deep learning, convolutional neural networks have been widely used in image segmentation tasks. While traditional UNet and its improved structures have achieved some success in image segmentation and other fields, they still have the following problems: Limited ability to capture contextual information: Convolution operations themselves are highly localized, making it difficult to effectively extract global dependencies.

[0003] Limitations of attention mechanisms: Traditional attention mechanisms are weak in global modeling and have difficulty taking into account both semantics and spatial distribution.

[0004] The multi-scale fusion method is coarse: multi-scale features are often directly spliced ​​or superimposed, lacking a dynamic adaptation mechanism, which affects the accuracy of segmentation boundaries.

[0005] Redundancy in skip connection information: Directly superimposing skip connections on the decoder may introduce low-level noise, requiring a more refined fusion mechanism.

[0006] To address the aforementioned issues, there is an urgent need for an efficient segmentation network structure that can combine global modeling, multi-scale enhancement, and dynamic fusion capabilities. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, this invention provides an image semantic segmentation method based on gated attention and multi-scale enhancement, which solves the problems of limited contextual information capture capability, weak global modeling capability, lack of dynamic adaptation mechanism in multi-scale fusion, and redundant skip connection information in existing methods.

[0008] To achieve the aforementioned objectives, the present invention employs the following technical solution: an image semantic segmentation method based on gated attention and multi-scale enhancement, comprising: Construct a module for channel-space attention gating sequences to fuse spatial and channel attention; Construct multi-scale feature enhancement blocks that utilize convolutional kernels of different scales to extract multi-scale contextual information; Construct a gated fusion block for dynamically controlling the transmission of low-level information; The encoder-decoder network is improved based on channel spatial attention gated sequence construction module, multi-scale feature enhancement block and gated fusion block to obtain semantic segmentation neural network; Obtain an image training set, use the image training to train the semantic segmentation neural network, and obtain the trained semantic segmentation neural network; The image to be classified is obtained, and the trained semantic segmentation neural network is used to classify the image.

[0009] The beneficial effects of this invention are as follows: This invention significantly improves the model's ability to capture key target regions and the accuracy of boundary reconstruction. This invention combines a channel-space gated attention mechanism, multi-scale feature enhancement blocks, and a gated fusion mechanism to form a unified image semantic segmentation network structure with gated attention and multi-scale enhancement. It utilizes a high-order state-space sequence model with low computational complexity to fuse spatial and channel attention, introducing sequence modeling capabilities to enhance target region capture; it introduces multi-scale feature enhancement blocks to strengthen local and multi-scale semantic feature expression; and it employs gated fusion blocks to dynamically control low-level information transmission, effectively enhancing boundary detail recovery. The model structure has strong versatility and is adaptable to various input types, including RGB, RGB-D, medical images, and remote sensing images.

[0010] Furthermore, the channel spatial attention gating sequence construction module includes a first data input terminal, an adaptive average pooling layer connected to the first data input terminal, a first convolutional layer connected to the adaptive average pooling layer, a first activation layer connected to the first convolutional layer, a second convolutional layer connected to the first activation layer, a second activation layer connected to the second convolutional layer, a first multiplication gate connected to the data input terminal and the second activation layer, a fourth max pooling layer and a first average pooling layer connected to the multiplication gate, a first addition gate connected to the fourth max pooling layer and the first average pooling layer, a second multiplication gate connected to the first multiplication gate and the first addition gate, a first normalization layer connected to the second multiplication gate, a first linear layer connected to the first normalization layer, a third activation layer connected to the first linear layer, a second linear layer connected to the third activation layer, and a first data output terminal connected to the second linear layer.

[0011] The beneficial effects of the above-mentioned further scheme are as follows: the channel-space attention gating sequence construction module combines the channel attention mechanism (upper path) and the spatial attention mechanism (lower path), achieving dual enhancement through the gating mechanism. It also possesses lightweight design, integrates deep attention and long-range dependency modeling capabilities, significantly improves the model's feature selectivity and expressiveness, and is a cost-effective general-purpose structural module suitable for multi-task scenarios such as segmentation, classification, and detection.

[0012] Furthermore, the output expression of the channel spatial attention gating sequence construction module is:

[0013] in, The output of the module for building channel-space attention-gated sequences; The inverse of the dimension reshaping operation; These are the features of the modeled sequence; This is the weight matrix for the second-level linear transformation; For activation functions; This is the weight matrix for the first-level linear transformation; These are the normalized sequence features; For layer normalization; Features of the flattened sequence; For dimensional reshaping operations; The feature map is after spatial attention enhancement; This is an intermediate feature map; Spatial attention map; Use the Sigmoid activation function; This is a 7×7 convolution operation; This is the feature map after concatenating Favg and Fmax along the channel dimension; For splicing operations; This is the average pooled feature map along the channel dimension; This is a max-pooling feature map along the channel dimension; Total number of channels; For the first The characteristic components of the channel; For channel indexing; Input for building the module for channel-space attention-gated sequences; Channel attention weights; Weights for fully connected layers used for dimensionality enhancement; For activation functions; Weights of the fully connected layer used for dimensionality reduction; This is the output of global average pooling; For global average pooling; for Height; for The width; For the first The tallest, the first Each feature component is wide; For high index; It is a wide index.

[0014] The beneficial effects of the above-mentioned further scheme are as follows: the channel-space attention gating sequence construction module combines the channel attention mechanism (upper path) and the spatial attention mechanism (lower path), achieving dual enhancement through the gating mechanism. It also possesses lightweight design, integrates deep attention and long-range dependency modeling capabilities, significantly improves the model's feature selectivity and expressiveness, and is a cost-effective general-purpose structural module suitable for multi-task scenarios such as segmentation, classification, and detection.

[0015] Furthermore, the multi-scale feature enhancement block includes a second data input terminal, a 1×1 convolution, a 3×3 convolution, and a 5×5 convolution connected to the second data input terminal, a second addition gate connected to the 1×1 convolution, the 3×3 convolution, and the 5×5 convolution, a convolutional layer connected to the second addition gate, and a second data output terminal connected to the convolutional layer.

[0016] The beneficial effects of the above-mentioned further scheme are as follows: the multi-scale feature enhancement block is used for feature enhancement operations in image processing. By setting convolutional kernels of different sizes (1×1, 3×3, and 5×5) in parallel, multi-scale feature extraction is performed on the input features, and channel compression convolution is used to complete the fusion output, realizing the simultaneous extraction of fine-grained and large-scale semantic features in a single module. This module has a lightweight structure, does not introduce branch complexity, has good pluggability and multi-task adaptability, and has significantly improved in terms of computational overhead and adaptability.

[0017] Furthermore, the output expression of the multi-scale feature enhancement block is:

[0018] in, The output of the multi-scale feature enhancement block; To fuse convolutions; The spliced ​​multi-scale features; For splicing operations; The output features are those of a 1×1 convolution; The output features are those of a 3×3 convolution; The output features are those of a 5×5 convolution; This is a 1×1 convolution operation; This serves as the input for the multi-scale feature enhancement block; This is a 1×1 convolution operation; This is a 5×5 convolution operation.

[0019] The beneficial effects of the above-mentioned further scheme are as follows: the multi-scale feature enhancement block is used for feature enhancement operations in image processing. By setting convolutional kernels of different sizes (1×1, 3×3, and 5×5) in parallel, multi-scale feature extraction is performed on the input features, and channel compression convolution is used to complete the fusion output, realizing the simultaneous extraction of fine-grained and large-scale semantic features in a single module. This module has a lightweight structure, does not introduce branch complexity, has good pluggability and multi-task adaptability, and has significantly improved in terms of computational overhead and adaptability.

[0020] Furthermore, the gated fusion block includes a third data input terminal, a third addition gate and a 1×1 convolutional layer respectively connected to the third data input terminal, a fourth activation layer connected to the 1×1 convolutional layer, a gated layer connected to the fourth activation layer, a gated layer respectively connected to the third addition gate, and a third data output terminal.

[0021] The beneficial effects of the above-mentioned further scheme are as follows: the gated fusion block learns dynamic weights (gating factors) to adaptively fuse decoder features and skip connection features, suppressing noise information and enhancing useful features, thus achieving "selective enhancement." By introducing lightweight 1×1 convolutions and sigmoid activation functions, the importance of skip connection features is learned, thereby achieving dynamic fusion with decoder features. This module has a simple structure, low computational cost, and strong adaptive and feature selection capabilities. Compared with existing fixed fusion or redundant concatenation methods, it effectively improves model performance and robustness while maintaining model lightweightness, making it particularly suitable for semantic segmentation tasks with high requirements for real-time performance, accuracy, and deployability.

[0022] Furthermore, the output expression of the gated fusion block is:

[0023] in, This is the output of the gated fusion block; Features are output by the decoder; The gating factor; This is element-wise multiplication; This is a skip connection feature; Use the Sigmoid activation function; This is a 1×1 convolution operation.

[0024] The beneficial effects of the above-mentioned further scheme are as follows: the gated fusion block learns dynamic weights (gating factors) to adaptively fuse decoder features and skip connection features, suppressing noise information and enhancing useful features, thus achieving "selective enhancement." By introducing lightweight 1×1 convolutions and sigmoid activation functions, the importance of skip connection features is learned, thereby achieving dynamic fusion with decoder features. This module has a simple structure, low computational cost, and strong adaptive and feature selection capabilities. Compared with existing fixed fusion or redundant concatenation methods, it effectively improves model performance and robustness while maintaining model lightweightness, making it particularly suitable for semantic segmentation tasks with high requirements for real-time performance, accuracy, and deployability.

[0025] Further, the semantic segmentation neural network includes an input layer, a first convolutional activation normalization layer connected to the input layer, a first max pooling layer connected to the first convolutional activation normalization layer, a second convolutional activation normalization layer and a first multi-scale feature enhancement block connected to the first max pooling layer, a second max pooling layer connected to the second convolutional activation normalization layer, a third convolutional activation normalization layer and a second multi-scale feature enhancement block connected to the second max pooling layer, a third max pooling layer connected to the third convolutional activation normalization layer, a first bridging layer and a third multi-scale feature enhancement block connected to the third max pooling layer, a channel spatial attention gating sequence construction module connected to the first bridging layer, and a channel spatial attention gating sequence construction module. The system comprises: a first deconvolutional layer connected to the first deconvolutional layer; a fourth convolutional activation normalization layer connected to the first deconvolutional layer; a first gated fusion block connected to the third multi-scale feature enhancement block and the fourth convolutional activation normalization layer, respectively; a second deconvolutional layer connected to the first gated fusion block; a fifth convolutional activation normalization layer connected to the second deconvolutional layer; a second gated fusion block connected to the second multi-scale feature enhancement block and the fifth convolutional activation normalization layer, respectively; a third deconvolutional layer connected to the second gated fusion block; a sixth convolutional activation normalization layer connected to the third deconvolutional layer; a third gated fusion block connected to the first multi-scale feature enhancement block and the sixth convolutional activation normalization layer, respectively; a seventh convolutional layer connected to the third gated fusion block; and an output layer connected to the seventh convolutional layer.

[0026] The beneficial effects of the above-mentioned further scheme are as follows: it combines channel-space gated attention mechanism, multi-scale feature enhancement block, and gated fusion mechanism to form a unified image semantic segmentation network structure with gated attention and multi-scale enhancement. It utilizes a high-order state-space sequence model with low computational complexity to fuse spatial and channel attention, introducing sequence modeling capabilities to improve target region capture; it introduces multi-scale feature enhancement blocks to strengthen local and multi-scale semantic feature expression; it employs gated fusion blocks to dynamically control low-level information transmission, effectively enhancing boundary detail recovery; and its model structure has strong versatility, adapting to various input types such as RGB, RGB-D, medical images, and remote sensing images.

[0027] Furthermore, the structures of the first multi-scale feature enhancement block, the second multi-scale feature enhancement block, and the third multi-scale feature enhancement block are all the same as those of the multi-scale feature enhancement block; the structures of the first gated fusion block, the second gated fusion block, and the third gated fusion block are all the same as those of the gated fusion block.

[0028] The beneficial effects of the above-mentioned further scheme are as follows: it combines channel-space gated attention mechanism, multi-scale feature enhancement block, and gated fusion mechanism to form a unified image semantic segmentation network structure with gated attention and multi-scale enhancement. It utilizes a high-order state-space sequence model with low computational complexity to fuse spatial and channel attention, introducing sequence modeling capabilities to improve target region capture; it introduces multi-scale feature enhancement blocks to strengthen local and multi-scale semantic feature representation; and it employs gated fusion blocks to dynamically control low-level information transmission, effectively enhancing boundary detail recovery.

[0029] Furthermore, the classification of the image to be classified using the trained semantic segmentation neural network specifically involves: The image to be classified is input into the trained semantic segmentation neural network, and then transmitted through the input layer to the first convolutional activation normalization layer for shallow feature extraction to obtain the first shallow feature map. The first shallow feature map is input into the first max pooling layer for downsampling to obtain the second shallow feature map; The second shallow feature map is input into the first multi-scale feature enhancement block, and multi-scale context information is extracted using convolutional kernels of different scales to obtain the first multi-scale feature map. The second shallow feature map is input into the second convolutional activation normalization layer for deep feature extraction, resulting in the first deep feature map. The first deep feature map is input into the second max pooling layer for downsampling to obtain the second deep feature map; The second deep feature map is input into the second multi-scale feature enhancement block, and multi-scale contextual information is extracted using convolutional kernels of different scales to obtain the second multi-scale feature map. The second deep feature map is input into the third convolutional activation normalization layer for deep feature extraction, resulting in the third deep feature map. The third deep feature map is input into the third max pooling layer for downsampling to obtain the fourth deep feature map; The fourth deep feature map is input into the third multi-scale feature enhancement block, and multi-scale context information is extracted using convolutional kernels of different scales to obtain the third multi-scale feature map. The fourth deep feature map is input into the first bridging layer to extract deep semantic features, resulting in a semantic feature map; The semantic feature map is input into the channel spatial attention gating sequence construction module to fuse spatial and channel attention, resulting in the first enhanced semantic feature map; The first enhanced semantic feature map is input into the first deconvolution layer for deconvolution operation to obtain the first deconvolution feature map; The first deconvolutional feature map is input into the fourth convolutional activation normalization layer for refinement and semantic enhancement, resulting in the second deconvolutional feature map. The second deconvolution feature map and the third multi-scale feature map are input into the first gated fusion block for fusion to obtain the first fused feature map; The first fused feature map is input into the second deconvolution layer for deconvolution operation to obtain the third deconvolution feature map; The third deconvolutional feature map is input into the fifth convolutional activation normalization layer for refinement and semantic enhancement, resulting in the fourth deconvolutional feature map. The fourth deconvolution feature map and the second multi-scale feature map are input into the second gated fusion block for fusion to obtain the second fused feature map; The second fused feature map is input into the third deconvolution layer for deconvolution operation to obtain the fifth deconvolution feature map; The fifth deconvolutional feature map is input into the sixth convolutional activation normalization layer for refinement and semantic enhancement, resulting in the sixth deconvolutional feature map. The sixth deconvolutional feature map and the first multi-scale feature map are input into the third gated fusion block for fusion to obtain the third fused feature map. The third fused feature map is input into the seventh convolutional layer, and a 1×1 convolution is used to map the channels to the number of output channels to obtain the classification result.

[0030] The beneficial effects of the above-mentioned further scheme are as follows: it combines channel-space gated attention mechanism, multi-scale feature enhancement block, and gated fusion mechanism to form a unified image semantic segmentation network structure with gated attention and multi-scale enhancement. It utilizes a high-order state-space sequence model with low computational complexity to fuse spatial and channel attention, introducing sequence modeling capabilities to improve target region capture; it introduces multi-scale feature enhancement blocks to strengthen local and multi-scale semantic feature representation; and it employs gated fusion blocks to dynamically control low-level information transmission, effectively enhancing boundary detail recovery. Attached Figure Description

[0031] Figure 1 This is a network structure diagram of the present invention.

[0032] Figure 2 This is a structural diagram of the channel spatial attention gating sequence building module in an embodiment of the present invention.

[0033] Figure 3 This is a diagram of the multi-scale feature enhancement block structure in an embodiment of the present invention.

[0034] Figure 4 This is a structural diagram of the gated fusion block in an embodiment of the present invention. Detailed Implementation

[0035] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0036] like Figure 1 As shown, in one embodiment of the present invention, an image semantic segmentation method based on gated attention and multi-scale enhancement includes: Construct a module for channel-space attention gating sequences to fuse spatial and channel attention; Construct multi-scale feature enhancement blocks that utilize convolutional kernels of different scales to extract multi-scale contextual information; Construct a gated fusion block for dynamically controlling the transmission of low-level information; The encoder-decoder network is improved based on channel spatial attention gated sequence construction module, multi-scale feature enhancement block and gated fusion block to obtain semantic segmentation neural network; Obtain an image training set, use the image training to train the semantic segmentation neural network, and obtain the trained semantic segmentation neural network; The image to be classified is obtained, and the trained semantic segmentation neural network is used to classify the image.

[0037] In this embodiment, the network structure of the image semantic segmentation method of the present invention includes: Overall structure: Encoder-decoder network structure; Encoder path: The input image is processed through multiple levels of convolutional blocks to extract features. Each level is followed by pooling downsampling to extract features from low to high levels layer by layer. Bridging stage: A standard convolutional block and a high-order state-space sequence model block with low computational complexity are added after the last stage of the encoder to achieve joint modeling of channel attention and spatial attention, and global dependencies are extracted by combining a lightweight sequence modeling mechanism. Multi-scale feature enhancement block: The encoder output is fused by three-scale convolutions (1×1, 3×3, 5×5) in each skip connection branch to improve the multi-scale receptive field and detail preservation; Gated fusion block: During the decoding stage, upsampled features and skip connection features are weighted and fused through a gating mechanism to introduce selective information control and reduce redundant interference; Decoder path: It uses upsampling + convolutional blocks to gradually restore spatial resolution, while introducing gating fusion to enhance the utilization of information from skip connections.

[0038] In this embodiment, the gated attention and multi-scale enhanced image semantic segmentation network structure of the present invention is as follows: Figure 1 As shown, the overall architecture is a typical encoder-decoder structure, which includes an attention mechanism and a multi-scale enhancement module.

[0039] The channel spatial attention gating sequence construction module includes a first data input terminal, an adaptive average pooling layer connected to the first data input terminal, a first convolutional layer connected to the adaptive average pooling layer, a first activation layer connected to the first convolutional layer, a second convolutional layer connected to the first activation layer, a second activation layer connected to the second convolutional layer, a first multiplication gate connected to the data input terminal and the second activation layer, a fourth max pooling layer and a first average pooling layer connected to the multiplication gate, a first addition gate connected to the fourth max pooling layer and the first average pooling layer, a second multiplication gate connected to the first multiplication gate and the first addition gate, a first normalization layer connected to the second multiplication gate, a first linear layer connected to the first normalization layer, a third activation layer connected to the first linear layer, a second linear layer connected to the third activation layer, and a first data output terminal connected to the second linear layer.

[0040] In this embodiment, the channel spatial attention gating sequence construction module is as follows: Figure 2 As shown, this structure combines channel attention (upper path) and spatial attention (lower path) mechanisms, achieving dual enhancement through gating. It is lightweight, integrates deep attention and long-range dependency modeling capabilities, significantly improving the model's feature selectivity and expressiveness. It is a cost-effective general-purpose structural module suitable for multi-task scenarios such as segmentation, classification, and detection.

[0041] (a) The first path Figure 2 (Main path above) 1. Input Input feature maps with shapes B, C, H, and W, representing batch size, number of channels, height, and width, respectively.

[0042] 2. Adaptive average pooling layer Global spatial compression is applied to the input to extract channel-level global information. This serves to compress the spatial dimension while preserving channel information.

[0043] 3. Convolutional layer + activation layer The compressed channel information is subjected to nonlinear transformation to enhance the expressive power between channels.

[0044] The operation is a 1x1 convolution, combined with a rectified linear unit activation function.

[0045] 4. Convolutional layer + activation layer Enhance the nonlinear representation and further model the relationships between channels.

[0046] 5. Dot product (multiplication gating) The output is multiplied channel by channel with the original input feature map to form a weighted feature map.

[0047] (ii) Second path (lower enhanced path) This path models attention in the spatial dimension.

[0048] 1. Max pooling layer and average pooling layer (along the channel dimension) Max pooling and average pooling are performed along the channel direction to preserve the spatial structure and extract different spatial features.

[0049] 2. Addition operation The results of max pooling and average pooling are added together to integrate multiple spatial information.

[0050] 3. Dot product operation The fusion result is multiplied element-wise with the main branch output to form a feature map with spatial attention.

[0051] 4. Normalization layer Normalize the product result to stabilize training and enhance generalization ability.

[0052] 5. Linear layer + activation layer + linear layer A linear transformation is applied to further model the normalized features. This enhances the non-linear expressive power of the features and is used for post-processing in the construction of the attention module.

[0053] 6. Output The final output is an enhanced feature map, which is then fed into downstream tasks (such as segmentation heads, classification heads, etc.).

[0054] The output expression of the channel spatial attention gating sequence construction module is:

[0055] in, The output of the module for building channel-space attention-gated sequences; The inverse of the dimension reshaping operation; These are the features of the modeled sequence; This is the weight matrix for the second-level linear transformation; For activation functions; This is the weight matrix for the first-level linear transformation; These are the normalized sequence features; For layer normalization; Features of the flattened sequence; For dimensional reshaping operations; The feature map is after spatial attention enhancement; This is an intermediate feature map; Spatial attention map; Use the Sigmoid activation function; This is a 7×7 convolution operation; To be and Feature map after concatenation along the channel dimension; For splicing operations; This is the average pooled feature map along the channel dimension; This is a max-pooling feature map along the channel dimension; Total number of channels; For the first The characteristic components of the channel; For channel indexing; Input for building the module for channel-space attention-gated sequences; Channel attention weights; Weights for fully connected layers used for dimensionality enhancement; For activation functions; Weights of the fully connected layer used for dimensionality reduction; This is the output of global average pooling; For global average pooling; for Height; for The width; For the first The tallest, the first Each feature component is wide; For high index; It is a wide index.

[0056] In this embodiment, the input feature map has a shape of B×C×H×W, i.e., batch × channel × height × width; global average pooling outputs are averaged across each channel in the spatial dimension to obtain a channel descriptor (channel compression); the weights of the fully connected layer map the channel relationships. Used for dimensionality reduction Used for dimensionality enhancement; activation function Increase non-linear expressive power; the Sigmoid activation function compresses channel weights to [0,1], which are then used as attention weights; channel attention weights With the same number of feature channels as the original, its function is to suppress / enhance channel responses; the output is enhanced features. Each channel is multiplied by a corresponding weight to achieve channel enhancement. The intermediate feature map after processing through the first path has a shape of B×C×H×W, i.e., batch × channel × height × width; the average pooling feature map along the channel dimension. For each spatial location (H×W), the average value of all channels is taken, and the output dimension is B×1×H×W; the max-pooling feature map is then generated along the channel dimension. For each spatial location (H×W), take the maximum value of all channels and output the dimension B×1×H×W; and The feature map, after concatenation along the channel dimension, takes the maximum value for all channels at each spatial location (H×W), and outputs a dimension of B×1×H×W; and The output dimension of the feature map after concatenation along the channel dimension is B×2×H×W; the concatenated feature map... A 7×7 convolution operation is performed to fuse spatial information and reduce the number of channels from 2 to 1, resulting in an output dimension of B×1×H×W. The sigmoid activation function normalizes the convolution output to the [0,1] interval, generating a spatial attention weight map. The spatial attention map has dimensions B×1×H×W, where each element represents the importance weight of the corresponding spatial location. The feature map after spatial attention enhancement still has dimensions B×C×H×W, but features at important spatial locations are amplified while features at less important locations are suppressed. Dimension reshaping flattens the spatial dimension H×W to a sequence length L=H×W. The flattened sequence feature dimension is B×L×C (L=H×W: sequence length, corresponding to the number of spatial locations in the image), converting the 2D spatial structure of the image into a 1D sequence. Layer normalization normalizes the sequence features S along the channel dimension (the last dimension C). The normalized sequence feature dimension remains B×L×C. The first-layer linear transformation weight matrix has a shape of C×D (D is the intermediate hidden dimension, set to D=4C), used to expand the feature dimension and increase model capacity. Activation function... A nonlinear transformation is introduced to enhance feature representation; the second-layer linear transformation weight matrix has a shape of D×C, mapping the hidden dimension D back to the original number of channels C, ensuring that the output dimension is consistent with the input; the modeled sequence features maintain a dimension of B×L×C, fusing long-distance dependencies within the sequence; the inverse dimension reshaping operation transforms the sequence features... The length dimension L=H×W is restored to the original spatial dimension H×W; the final output feature map dimension is the same as... Consistent (B×C×H×W), used for subsequent image tasks.

[0057] The multi-scale feature enhancement block includes a second data input terminal, a 1×1 convolution, a 3×3 convolution, and a 5×5 convolution connected to the second data input terminal, a second addition gate connected to the 1×1 convolution, the 3×3 convolution, and the 5×5 convolution, a convolutional layer connected to the second addition gate, and a second data output terminal connected to the convolutional layer.

[0058] In this embodiment, the multi-scale feature enhancement block is as follows: Figure 3As shown, this module is used for feature enhancement operations in image processing. By setting convolutional kernels of different sizes (1×1, 3×3, and 5×5) in parallel, it performs multi-scale feature extraction on the input features and uses channel compression convolution to complete the fusion output, achieving the simultaneous extraction of fine-grained and large-scale semantic features in a single module. This module has a lightweight structure, introduces no branching complexity, has good pluggability and multi-task adaptability, and significantly improves in terms of computational cost and adaptability.

[0059] 1. Input Input a feature map for multi-scale feature extraction.

[0060] 2. Branch-parallel convolution operation The input features are copied to multiple branches with different kernel sizes, and each branch uses a different receptive field for feature extraction: (1) Branch 1: 1×1 convolution The "experience the wild" approach focuses on capturing local details and information interaction within the channels.

[0061] (2) Branch 2: 3×3 convolution Standard convolution operations balance local and contextual perception. It can extract a moderate range of spatial contextual information.

[0062] (3) Branch 3: 5×5 convolution It has a larger receptive field, making it suitable for extracting global or large-scale contextual features. This helps improve the ability to perceive the shape of a target.

[0063] 3. Feature fusion (addition) The outputs of 1×1, 3×3, and 5×5 convolutions are fused channel-by-channel by addition. This integrates multi-scale information and enhances the network's ability to perceive targets of different sizes. The output dimensions are typically kept consistent with each convolutional branch or fused after unifying the number of channels.

[0064] 4. Post-processing convolutional layer An additional convolution operation is used to fuse the processed features; a 1×1 convolution is selected.

[0065] By integrating multi-scale information, adjusting the number of channels, and introducing nonlinear transformation.

[0066] 5. Output Output a fused feature map, which contains contextual information at multiple scales, which helps improve the model's ability to perceive complex structures.

[0067] The output expression of the multi-scale feature enhancement block is:

[0068] in, The output of the multi-scale feature enhancement block; To fuse convolutions; The spliced ​​multi-scale features; For splicing operations; The output features are those of a 1×1 convolution; The output features are those of a 3×3 convolution; The output features are those of a 5×5 convolution; This is a 1×1 convolution operation; This serves as the input for the multi-scale feature enhancement block; This is a 1×1 convolution operation; This is a 5×5 convolution operation.

[0069] The gated fusion block includes a third data input terminal, a third addition gate and a 1×1 convolutional layer connected to the third data input terminal, a fourth activation layer connected to the 1×1 convolutional layer, a gated layer connected to the fourth activation layer, a gated layer connected to the third addition gate, and a third data output terminal.

[0070] In this embodiment, the gating fusion block is as follows: Figure 4 As shown, by learning dynamic weights (gating factors), decoder features and skip connection features are adaptively fused to suppress noise information and enhance useful features, achieving "selective enhancement." By introducing lightweight 1×1 convolutions and a sigmoid activation function, the importance of skip connection features is learned, thus achieving dynamic fusion with decoder features. This module has a simple structure, low computational cost, and strong adaptive and feature selection capabilities. Compared to existing fixed fusion or redundant concatenation methods, it effectively improves model performance and robustness while maintaining model lightweightness, making it particularly suitable for semantic segmentation tasks with high requirements for real-time performance, accuracy, and deployability.

[0071] 1. Input The feature map from the previous layer is passed in two paths: one path skips the processing (forming a residual connection); the other path is used to generate the gating factor.

[0072] 2. 1×1 convolutional layer Applying a 1×1 convolution operation to the input features does not change the spatial dimension, but adjusts the number of channels. This compresses / adjusts the channel dimension. Channel information is then aggregated to extract important features.

[0073] 3. Activation layer The rectified linear unit activation function is used to provide the activated feature representation for subsequent gating.

[0074] 4. Gating layer The activated features are further processed to compress the values ​​to the [0, 1] interval, which serves as the gating factor. The gating factor is obtained by performing a convolution and sigmoid activation on the skip feature skip_feat.

[0075] 5. Adding residuals The gated features calculated above are added element-wise to the original input. The residual connection mechanism ensures the preservation of the original features, while the gating provides modulation.

[0076] 6. Output The output is a fused feature map with a certain degree of adaptive adjustment capability. It can autonomously "selectively enhance" important features based on the input content, that is, give "higher weight" to "important information" in the input feature map and "give low weight or suppress redundant information" in an adaptive feature processing.

[0077] The output expression of the gated fusion block is:

[0078] in, This is the output of the gated fusion block; Features are output by the decoder; The gating factor; This is element-wise multiplication; This is a skip connection feature; Use the Sigmoid activation function; This is a 1×1 convolution operation.

[0079] In this embodiment, the decoder output features are the high-level semantic features to be fused; the skip connection features are the low-level detail features output by the encoder; the sigmoid activation function normalizes the convolution output to [0,1] and generates gating weights; the gating factor is a dynamic weight matrix, and the higher the value, the more important the skip_feat at the corresponding position, with dimensions of B×C×H×W. Let be the residual sum of the decoder features and the weighted jump features, with dimensions B×C×H×W.

[0080] The semantic segmentation neural network includes an input layer, a first convolutional activation normalization layer connected to the input layer, a first max pooling layer connected to the first convolutional activation normalization layer, a second convolutional activation normalization layer and a first multi-scale feature enhancement block connected to the first max pooling layer, a second max pooling layer connected to the second convolutional activation normalization layer, a third convolutional activation normalization layer and a second multi-scale feature enhancement block connected to the second max pooling layer, a third max pooling layer connected to the third convolutional activation normalization layer, a first bridging layer and a third multi-scale feature enhancement block connected to the third max pooling layer, a channel spatial attention gating sequence construction module connected to the first bridging layer, and a module connected to the channel spatial attention gating sequence construction module. A first deconvolutional layer, a fourth convolutional activation normalization layer connected to the first deconvolutional layer, a first gated fusion block connected to the third multi-scale feature enhancement block and the fourth convolutional activation normalization layer respectively, a second deconvolutional layer connected to the first gated fusion block, a fifth convolutional activation normalization layer connected to the second deconvolutional layer, a second gated fusion block connected to the second multi-scale feature enhancement block and the fifth convolutional activation normalization layer respectively, a third deconvolutional layer connected to the second gated fusion block, a sixth convolutional activation normalization layer connected to the third deconvolutional layer, a third gated fusion block connected to the first multi-scale feature enhancement block and the sixth convolutional activation normalization layer respectively, a seventh convolutional layer connected to the third gated fusion block, and an output layer connected to the seventh convolutional layer.

[0081] The structures of the first multi-scale feature enhancement block, the second multi-scale feature enhancement block, and the third multi-scale feature enhancement block are all the same as those of the multi-scale feature enhancement block; the structures of the first gated fusion block, the second gated fusion block, and the third gated fusion block are all the same as those of the gated fusion block.

[0082] The process of classifying the image to be classified using the trained semantic segmentation neural network is as follows: The image to be classified is input into the trained semantic segmentation neural network, and then transmitted through the input layer to the first convolutional activation normalization layer for shallow feature extraction to obtain the first shallow feature map. The first shallow feature map is input into the first max pooling layer for downsampling to obtain the second shallow feature map; The second shallow feature map is input into the first multi-scale feature enhancement block, and multi-scale context information is extracted using convolutional kernels of different scales to obtain the first multi-scale feature map. The second shallow feature map is input into the second convolutional activation normalization layer for deep feature extraction, resulting in the first deep feature map. The first deep feature map is input into the second max pooling layer for downsampling to obtain the second deep feature map; The second deep feature map is input into the second multi-scale feature enhancement block, and multi-scale contextual information is extracted using convolutional kernels of different scales to obtain the second multi-scale feature map. The second deep feature map is input into the third convolutional activation normalization layer for deep feature extraction, resulting in the third deep feature map. The third deep feature map is input into the third max pooling layer for downsampling to obtain the fourth deep feature map; The fourth deep feature map is input into the third multi-scale feature enhancement block, and multi-scale context information is extracted using convolutional kernels of different scales to obtain the third multi-scale feature map. The fourth deep feature map is input into the first bridging layer to extract deep semantic features, resulting in a semantic feature map; The semantic feature map is input into the channel spatial attention gating sequence construction module to fuse spatial and channel attention, resulting in the first enhanced semantic feature map; The first enhanced semantic feature map is input into the first deconvolution layer for deconvolution operation to obtain the first deconvolution feature map; The first deconvolutional feature map is input into the fourth convolutional activation normalization layer for refinement and semantic enhancement, resulting in the second deconvolutional feature map. The second deconvolution feature map and the third multi-scale feature map are input into the first gated fusion block for fusion to obtain the first fused feature map; The first fused feature map is input into the second deconvolution layer for deconvolution operation to obtain the third deconvolution feature map; The third deconvolutional feature map is input into the fifth convolutional activation normalization layer for refinement and semantic enhancement, resulting in the fourth deconvolutional feature map. The fourth deconvolution feature map and the second multi-scale feature map are input into the second gated fusion block for fusion to obtain the second fused feature map; The second fused feature map is input into the third deconvolution layer for deconvolution operation to obtain the fifth deconvolution feature map; The fifth deconvolutional feature map is input into the sixth convolutional activation normalization layer for refinement and semantic enhancement, resulting in the sixth deconvolutional feature map. The sixth deconvolutional feature map and the first multi-scale feature map are input into the third gated fusion block for fusion to obtain the third fused feature map. The third fused feature map is input into the seventh convolutional layer, and a 1×1 convolution is used to map the channels to the number of output channels to obtain the classification result.

[0083] In this embodiment, the functions and roles of each module are explained in detail step by step: (a) Input stage Input: The input is the original image with dimensions H×W×3.

[0084] (II) Encoder stage (feature extraction + downsampling) 1. First convolutional activation normalization layer It consists of two convolutional layers, a batch normalization layer, and a rectified linear unit activation function. The number of output channels is increased while the spatial size remains unchanged. It initially extracts shallow features such as edges and textures.

[0085] 2. First max pooling layer Downsampling is achieved using max pooling. The spatial size is reduced to H / 2 × W / 2.

[0086] 3. First multi-scale feature enhancement block Multi-scale contextual information is extracted using convolutional kernels of different scales (1×1, 3×3, 5×5). Spatial dimensions are preserved while semantic expression is enriched.

[0087] 4. Second convolutional activation normalization layer and second max pooling layer By extracting deeper features, the number of output channels continues to increase, and the space shrinks to H / 4 × W / 4.

[0088] 5. Second Multi-Scale Feature Enhancement Block Multi-scale contextual information is extracted using convolutional kernels of different scales (1×1, 3×3, 5×5) to enhance the output features of the second layer of the encoder.

[0089] 6. The third convolutional activation normalization layer and the third max pooling layer The deepest encoder layer has a spatial resolution of H / 8 and W / 8. It extracts high-level semantic features but has limited spatial information.

[0090] 7. Third Multi-Scale Feature Enhancement Block Multi-scale contextual information is extracted using convolutional kernels of different scales (1×1, 3×3, 5×5) to enhance the third-layer output of the decoder.

[0091] 8. First bridging layer Convolution extracts the deepest semantic features, reaching the maximum number of channels, namely 512.

[0092] 9. Channel-space attention-gated high-order state-space sequence model block A high-order state-space sequence model is used for modeling, combining channel attention and spatial attention to enhance semantic representation and global perception capabilities.

[0093] (III) Decoder stage (upsampling + skip connections) 1. First deconvolution layer By using deconvolution, the spatial size is changed to H / 4 × W / 4, and the number of channels is reduced to 256, which facilitates fusion with the third layer features in the decoder stage.

[0094] 2. Fourth convolutional activation normalization layer We refine and semantically enhance the segmentation map using convolution and activation operations. During upsampling, we recapture local edge structures, helping to reconstruct more accurate boundaries from the segmentation map.

[0095] 3. First gated fusion block A gating mechanism is used to fuse the upsampled features with the features output by the third multi-scale feature enhancement block in the decoder stage, retaining key features and filtering redundant information.

[0096] 4. Second deconvolution layer By using deconvolution, the spatial size is changed to H / 2 × W / 2, and the number of channels is reduced to 128, which facilitates the fusion with the second layer features in the decoder stage.

[0097] 5. Fifth convolutional activation normalization layer We further refine and semantically enhance the segmentation using convolution and activation operations. During upsampling, we recapture local edge structures, helping to reconstruct more accurate boundaries from the segmentation map.

[0098] 6. Second gated fusion block A gating mechanism is used to fuse the upsampled features with the features output by the second multi-scale feature enhancement block in the decoder stage, retaining key features and filtering redundant information.

[0099] 7. Third deconvolution layer By using deconvolution, the spatial size is changed to H × W, and the number of channels is reduced to 64, which facilitates fusion with the first layer features in the decoder stage.

[0100] 8. Sixth convolutional activation normalization layer We further refine and semantically enhance the segmentation using convolution and activation operations. During upsampling, we recapture local edge structures, helping to reconstruct more accurate boundaries from the segmentation map.

[0101] 9. Third gated fusion block A gating mechanism is used to fuse the upsampled features with the features output by the first multi-scale feature enhancement block in the decoder stage, retaining key features and filtering out redundant information.

[0102] (iv) Output stage The seventh convolutional layer: uses 1×1 convolutions to map channels to the number of output channels (e.g., 1 for binary classification output).

Claims

1. An image semantic segmentation method based on gated attention and multi-scale enhancement, characterized in that, include: Construct a module for channel-space attention gating sequences to fuse spatial and channel attention; Construct multi-scale feature enhancement blocks that utilize convolutional kernels of different scales to extract multi-scale contextual information; Construct a gated fusion block for dynamically controlling the transmission of low-level information; The encoder-decoder network is improved based on channel spatial attention gated sequence construction module, multi-scale feature enhancement block and gated fusion block to obtain semantic segmentation neural network; Obtain an image training set, use the image training to train the semantic segmentation neural network, and obtain the trained semantic segmentation neural network; The image to be classified is obtained, and the trained semantic segmentation neural network is used to classify the image.

2. The image semantic segmentation method based on gated attention and multi-scale enhancement according to claim 1, characterized in that, The channel spatial attention gating sequence construction module includes a first data input terminal, an adaptive average pooling layer connected to the first data input terminal, a first convolutional layer connected to the adaptive average pooling layer, a first activation layer connected to the first convolutional layer, a second convolutional layer connected to the first activation layer, a second activation layer connected to the second convolutional layer, a first multiplication gate connected to the data input terminal and the second activation layer, a fourth max pooling layer and a first average pooling layer connected to the multiplication gate, a first addition gate connected to the fourth max pooling layer and the first average pooling layer, a second multiplication gate connected to the first multiplication gate and the first addition gate, a first normalization layer connected to the second multiplication gate, a first linear layer connected to the first normalization layer, a third activation layer connected to the first linear layer, a second linear layer connected to the third activation layer, and a first data output terminal connected to the second linear layer.

3. The image semantic segmentation method based on gated attention and multi-scale enhancement according to claim 1, characterized in that, The output expression of the channel spatial attention gating sequence construction module is: in, The output of the module for building channel-space attention-gated sequences; The inverse of the dimension reshaping operation; These are the features of the modeled sequence; This is the weight matrix for the second-level linear transformation; For activation functions; This is the weight matrix for the first-level linear transformation; These are the normalized sequence features; For layer normalization; Features of the flattened sequence; For dimensional reshaping operations; The feature map is after spatial attention enhancement; This is an intermediate feature map; Spatial attention map; Use the Sigmoid activation function; This is a 7×7 convolution operation; This is the feature map after concatenating Favg and Fmax along the channel dimension; For splicing operations; This is the average pooled feature map along the channel dimension; This is a max-pooling feature map along the channel dimension; Total number of channels; For the first The characteristic components of the channel; For channel indexing; Input for building the module for channel-space attention-gated sequences; Channel attention weights; Weights for fully connected layers used for dimensionality enhancement; For activation functions; Weights of the fully connected layer used for dimensionality reduction; This is the output of global average pooling; For global average pooling; for Height; for The width; For the first The tallest, the first Each feature component is wide; For high index; It is a wide index.

4. The image semantic segmentation method based on gated attention and multi-scale enhancement according to claim 1, characterized in that, The multi-scale feature enhancement block includes a second data input terminal, a 1×1 convolution, a 3×3 convolution, and a 5×5 convolution connected to the second data input terminal, a second addition gate connected to the 1×1 convolution, the 3×3 convolution, and the 5×5 convolution, a convolutional layer connected to the second addition gate, and a second data output terminal connected to the convolutional layer.

5. The image semantic segmentation method based on gated attention and multi-scale enhancement according to claim 1, characterized in that, The output expression of the multi-scale feature enhancement block is: in, The output of the multi-scale feature enhancement block; To fuse convolutions; The spliced ​​multi-scale features; For splicing operations; The output features are those of a 1×1 convolution; The output features are those of a 3×3 convolution; The output features are those of a 5×5 convolution; This is a 1×1 convolution operation; This serves as the input for the multi-scale feature enhancement block; This is a 1×1 convolution operation; This is a 5×5 convolution operation.

6. The image semantic segmentation method based on gated attention and multi-scale enhancement according to claim 1, characterized in that, The gated fusion block includes a third data input terminal, a third addition gate and a 1×1 convolutional layer connected to the third data input terminal, a fourth activation layer connected to the 1×1 convolutional layer, a gated layer connected to the fourth activation layer, a gated layer connected to the third addition gate, and a third data output terminal.

7. The image semantic segmentation method based on gated attention and multi-scale enhancement according to claim 1, characterized in that, The output expression of the gated fusion block is: in, This is the output of the gated fusion block; Features are output by the decoder; The gating factor; This is element-wise multiplication; This is a skip connection feature; Use the Sigmoid activation function; This is a 1×1 convolution operation.

8. The image semantic segmentation method based on gated attention and multi-scale enhancement according to claim 1, characterized in that, The semantic segmentation neural network includes an input layer, a first convolutional activation normalization layer connected to the input layer, a first max pooling layer connected to the first convolutional activation normalization layer, a second convolutional activation normalization layer and a first multi-scale feature enhancement block connected to the first max pooling layer, a second max pooling layer connected to the second convolutional activation normalization layer, a third convolutional activation normalization layer and a second multi-scale feature enhancement block connected to the second max pooling layer, a third max pooling layer connected to the third convolutional activation normalization layer, a first bridging layer and a third multi-scale feature enhancement block connected to the third max pooling layer, a channel spatial attention gating sequence construction module connected to the first bridging layer, and a module connected to the channel spatial attention gating sequence construction module. A first deconvolutional layer, a fourth convolutional activation normalization layer connected to the first deconvolutional layer, a first gated fusion block connected to the third multi-scale feature enhancement block and the fourth convolutional activation normalization layer respectively, a second deconvolutional layer connected to the first gated fusion block, a fifth convolutional activation normalization layer connected to the second deconvolutional layer, a second gated fusion block connected to the second multi-scale feature enhancement block and the fifth convolutional activation normalization layer respectively, a third deconvolutional layer connected to the second gated fusion block, a sixth convolutional activation normalization layer connected to the third deconvolutional layer, a third gated fusion block connected to the first multi-scale feature enhancement block and the sixth convolutional activation normalization layer respectively, a seventh convolutional layer connected to the third gated fusion block, and an output layer connected to the seventh convolutional layer.

9. The image semantic segmentation method based on gated attention and multi-scale enhancement according to claim 8, characterized in that, The structures of the first multi-scale feature enhancement block, the second multi-scale feature enhancement block, and the third multi-scale feature enhancement block are all the same as those of the multi-scale feature enhancement block; the structures of the first gated fusion block, the second gated fusion block, and the third gated fusion block are all the same as those of the gated fusion block.

10. The image semantic segmentation method based on gated attention and multi-scale enhancement according to claim 1, characterized in that, The process of classifying the image to be classified using the trained semantic segmentation neural network is as follows: The image to be classified is input into the trained semantic segmentation neural network, and then transmitted through the input layer to the first convolutional activation normalization layer for shallow feature extraction to obtain the first shallow feature map. The first shallow feature map is input into the first max pooling layer for downsampling to obtain the second shallow feature map; The second shallow feature map is input into the first multi-scale feature enhancement block, and multi-scale context information is extracted using convolutional kernels of different scales to obtain the first multi-scale feature map. The second shallow feature map is input into the second convolutional activation normalization layer for deep feature extraction, resulting in the first deep feature map. The first deep feature map is input into the second max pooling layer for downsampling to obtain the second deep feature map; The second deep feature map is input into the second multi-scale feature enhancement block, and multi-scale contextual information is extracted using convolutional kernels of different scales to obtain the second multi-scale feature map. The second deep feature map is input into the third convolutional activation normalization layer for deep feature extraction, resulting in the third deep feature map. The third deep feature map is input into the third max pooling layer for downsampling to obtain the fourth deep feature map; The fourth deep feature map is input into the third multi-scale feature enhancement block, and multi-scale context information is extracted using convolutional kernels of different scales to obtain the third multi-scale feature map. The fourth deep feature map is input into the first bridging layer to extract deep semantic features, resulting in a semantic feature map; The semantic feature map is input into the channel spatial attention gating sequence construction module to fuse spatial and channel attention, resulting in the first enhanced semantic feature map; The first enhanced semantic feature map is input into the first deconvolution layer for deconvolution operation to obtain the first deconvolution feature map; The first deconvolutional feature map is input into the fourth convolutional activation normalization layer for refinement and semantic enhancement, resulting in the second deconvolutional feature map. The second deconvolution feature map and the third multi-scale feature map are input into the first gated fusion block for fusion to obtain the first fused feature map; The first fused feature map is input into the second deconvolution layer for deconvolution operation to obtain the third deconvolution feature map; The third deconvolutional feature map is input into the fifth convolutional activation normalization layer for refinement and semantic enhancement, resulting in the fourth deconvolutional feature map. The fourth deconvolution feature map and the second multi-scale feature map are input into the second gated fusion block for fusion to obtain the second fused feature map; The second fused feature map is input into the third deconvolution layer for deconvolution operation to obtain the fifth deconvolution feature map; The fifth deconvolutional feature map is input into the sixth convolutional activation normalization layer for refinement and semantic enhancement, resulting in the sixth deconvolutional feature map. The sixth deconvolutional feature map and the first multi-scale feature map are input into the third gated fusion block for fusion to obtain the third fused feature map. The third fused feature map is input into the seventh convolutional layer, and a 1×1 convolution is used to map the channels to the number of output channels to obtain the classification result.

Images