Real-time semantic segmentation method based on boundary feature guide fusion

Abstract

The invention provides a real-time semantic segmentation algorithm based on boundary feature guide fusion, and belongs to the field of real-time semantic segmentation. Firstly, a branch parallel strategy is adopted, a space branch extracts detail features, a semantic branch extracts deep semantic features, and a boundary feature extraction module is adopted by a boundary branch to concentrate on extraction of boundary features. Secondly, a feature enhancement fusion module is established in stages, and global features are gradually improved through local features; thirdly, an asymmetric-pixel attention pyramid pooling module is set at the tail end of a semantic branch to improve the learning ability of the model for complex scene features; and finally, a bidirectional attention fusion module is adopted to interact global features. According to verification, the performance of the method is better than that of many existing mainstream methods, and the problems that in real-time semantic segmentation of existing methods, boundary features of target objects in complex traffic scenes are difficult to extract, information guidance is lacked during feature fusion, and the segmentation effect is poor due to the fact that the features are prone to being interfered by information can be effectively solved.

Description

Technical Field

[0001] This invention relates to the field of real-time semantic segmentation technology, and specifically to a real-time semantic segmentation method based on boundary feature-guided fusion. Background Technology

[0002] Real-time semantic segmentation is an important topic in the field of computer vision. Its goal is to accurately classify every pixel in an image while maintaining real-time processing speed (i.e., a frame rate of at least 30 FPS), achieving pixel-level semantic understanding. With the rapid development of deep learning technology, this task has made significant progress in semantic segmentation and has been widely applied in various real-world scenarios such as autonomous driving, drone vision, smart healthcare, and remote sensing image processing.

[0003] In traffic scenarios, pedestrians, roads, and vehicles present diverse and complex information with blurred boundaries, making precise segmentation difficult. Early semantic segmentation methods largely relied on traditional image processing and machine learning techniques, such as edge detection, region growing, and classification methods based on handcrafted features. These methods typically extract features based on pre-defined information, only identifying some typical regions, and struggle to achieve accurate segmentation for complex and diverse targets.

[0004] In recent years, with the rapid development of deep learning technology in real-time semantic segmentation, neural network-based algorithms can automatically learn various object features and quickly and efficiently segment target images. In particular, the emergence of fully convolutional neural networks has achieved end-to-end pixel-level classification, effectively reducing the computational parameters of network models. Since then, numerous enhanced network models have emerged. Traditional encoder-decoder networks extract high-level semantic features from input images using an encoder. It gradually reduces the spatial resolution of feature maps through a series of convolutional layers, activation functions, and downsampling operations (such as pooling or stride convolution), while increasing their semantic abstraction capabilities. The decoder then gradually restores the low-resolution semantic features extracted by the encoder to the original image size, achieving fine pixel-level classification. However, its high computational cost and low inference speed also limit the development of models in fields with real-time requirements. Recently, dual-branch network models have abandoned the decoder, instead establishing spatial and semantic branches to extract spatial detail features and deep semantic features of the image, respectively, effectively improving the model's inference speed and achieving a new balance between segmentation accuracy and inference speed. However, it only focuses on the spatial and semantic features of the image, lacking attention to boundary features, resulting in blurred image edge segmentation. Furthermore, due to the complex shape of object boundaries, the extraction of boundary features is difficult, and different features have different characteristics, making them susceptible to information interference during feature fusion, which ultimately affects the segmentation performance of the model. Summary of the Invention

[0005] To overcome the shortcomings of the existing technologies, improve the ability to learn object boundary features in complex scenarios, and reduce information interference between different features, this invention provides a real-time semantic segmentation method based on boundary feature-guided fusion.

[0006] In embodiments of the present invention, the real-time semantic segmentation method based on boundary feature-guided fusion includes the following technical solutions, with specific steps as follows:

[0007] 1) Perform "fast downsampling" on the training set input to the model to share shallow features. After three downsampling operations, the image resolution is reduced to 1 / 8 of the original input image before being fed into each branch.

[0008] 2) To enable the model to efficiently extract features from different branches, this invention utilizes branch parallelism, extracting spatial detail information and deep semantic information of the image through spatial branches and semantic branches respectively. Addressing the difficulty in extracting boundary features, a boundary branch is established, and a boundary feature extraction module is employed to focus on extracting the image's boundary features.

[0009] 3) To reduce the loss of feature details and information interference between different features, feature enhancement and fusion modules are established at each stage of each branch. The local features of each branch are refined through the channel attention mechanism, and the features are guided to be fused efficiently, so as to gradually improve the global features with local features.

[0010] 4) To enhance the model's ability to learn multi-scale features, an asymmetric pixel attention pyramid pooling module is set up at the end of the semantic branch to learn multi-scale information of images in complex scenes.

[0011] 5) Establish a bidirectional attention fusion module, based on boundary features, to guide the fusion of spatial and semantic features and improve global information.

[0012] 6) Use a segmentation head consisting of residual blocks and convolutional blocks to efficiently segment the image features obtained by the model, and obtain the final segmentation result.

[0013] The real-time semantic segmentation method based on boundary feature-guided fusion described in the above steps is characterized by the following specific method of "fast downsampling": the input image is downsampled three times to reduce the resolution to 1 / 2, 1 / 4 and 1 / 8 of the original input image, respectively, and then sent to each branch for feature extraction.

[0014] The real-time semantic segmentation method based on boundary feature-guided fusion described in the above steps is characterized by the specific way each branch extracts features from the image: the spatial branch and the semantic branch use the mature Basicblock module as the basic module, the Bottleblock module as the bottleneck module for staged feature extraction, and the boundary branch adopts a boundary feature extraction module that focuses on extracting boundary features, and reuses decomposition convolution to extract deep boundary features.

[0015] The real-time semantic segmentation method based on boundary feature-guided fusion described in the above steps is characterized by the specific method of the asymmetric-pixel attention pyramid pooling module: basic multi-scale features are extracted by average pooling and global average pooling operations, then asymmetric convolution is used to enhance the feature expression capability, and finally, the importance score at the pixel level is calculated by the Sigmoid activation function at each pixel position of the multi-scale features to generate the feature weight of each pixel. Combined with the input features, the final multi-scale information is obtained.

[0016] The real-time semantic segmentation method based on boundary feature-guided fusion described in the above steps is characterized by the specific method of the bidirectional attention fusion module: based on boundary features, spatial detail features and deep semantic features are guided to interact globally through a bidirectional attention mechanism to obtain complete global features.

[0017] The real-time semantic segmentation method based on boundary feature-guided fusion described in the above steps is characterized by the specific way in which the segmentation head generates the image segmentation map: the global feature tensor is processed by a 3×3 convolution and a 1×1 convolution to finally generate the image segmentation map predicted by the model.

[0018] Compared with existing technologies, the beneficial effects of this invention are: it can efficiently extract image boundary features, improve global feature information, and effectively reduce information interference between different features. It has advanced learning and generalization capabilities, and its segmentation performance is better than many existing mainstream methods. It can effectively improve the problem of difficulty in extracting boundary features in segmentation. Due to the different characteristics of different features, they are easily affected by information interference during feature fusion, which ultimately leads to poor segmentation results. Attached Figure Description

[0019] Figure 1 This is the overall flowchart of a real-time semantic segmentation method based on boundary feature-guided fusion;

[0020] Figure 2 The process of processing information through spatial and semantic branches is as follows: (a): The specific process of extracting image features by the Basicblock module; (b): The specific process of extracting image features by the Bottleblock module.

[0021] Figure 3 This describes the specific process by which the boundary feature extraction module extracts image boundary features.

[0022] Figure 4 (a) is the information processing flow of the feature enhancement and fusion module. Figure 4 (b) is the specific process of channel attention processing information.

[0023] Figure 5 This is a flowchart of the asymmetric pixel attention pyramid pooling module.

[0024] Figure 6 This is a flowchart of the bidirectional attention fusion module. Detailed Implementation

[0025] Specific embodiments of the present invention will now be described. Examples of exemplary embodiments are shown in the accompanying drawings, which will help those skilled in the art to further understand the invention. It is worth noting that the following embodiments are only used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art without departing from the concept of the present invention are within the scope of protection of the present invention.

[0026] Please see Figures 1 to 6 This invention provides a real-time semantic segmentation method based on boundary feature-guided fusion, comprising the following steps:

[0027] Step 1: Perform "fast downsampling" on the training set input to the model. Each branch shares shallow features. After three downsampling operations, the image resolution is reduced to 1 / 8 of the original input image, and the output image size is 64×256×128. Then, it is fed into each branch for training.

[0028] Step 2: Each branch uses the feature extraction module for feature extraction, and the bottleneck blocks all use the Bottleblock module, such as... Figure 2 As shown in (b), combining 3×3 convolutions and 1×1 convolutions reduces computation while efficiently extracting image features. The basic modules for spatial and semantic branches use the relatively mature Basicblock module, such as... Figure 2 As shown in (a), image features are extracted through 3×3 convolution. Both Basicblock and Bottleblock use residual connections to promote deep training of the network. The image sizes output by each stage of the spatial branch are 64×256×128, 64×256×128, and 128×256×128, respectively, and the image sizes output by each stage of the semantic branch are 128×128×64, 256×64×32, and 512×32×16, respectively.

[0029] The boundary branch uses a boundary feature extraction module focused on boundary feature extraction as its basic module, and its structure is as follows: Figure 3 As shown, decomposition convolution is used to extract deep spatial features from the image. The output image sizes at each stage are 64×256×128, 64×256×128, and 128×256×128, respectively. The feature extraction process can be represented as follows:

[0030] (1)

[0031] (2)

[0032] (3)

[0033] (4)

[0034] (5)

[0035] In the formula: Input feature maps for boundary branches, It is a decomposed convolution with a 3×1 kernel. It is a decomposed convolution with a kernel of 1×3. This is for channel splicing operations. For 1×1 convolution, For element addition operations, Output feature maps for the module.

[0036] First, decomposition convolution is used to obtain boundary features F1, F2, and F3 at different depths. Then, these features are concatenated through channels to obtain the complete boundary feature F4. Finally, residual connections are used to connect the feature map to the input feature map. Perform element-wise addition to obtain the boundary feature output. .

[0037] Step 3: As Figure 4 As shown in (a), this invention embeds feature enhancement and fusion modules at different stages of feature extraction in each branch. It refines features through channel attention and guides local feature fusion to reduce feature interference. The specific process of channel attention is as follows: Figure 4 As shown in (b), the spatial branch output of the first feature enhancement fusion module is 64×256×128, and the semantic branch output is 128×128×64; the spatial branch output of the second feature enhancement fusion module is 64×256×128, and the semantic branch output is 256×64×32. The specific process can be represented as follows:

[0038] (6)

[0039] (7)

[0040] (8)

[0041] In the formula: For ReLU activation function, For 1×1 convolution, For bilinear interpolation upsampling operation, Adapt For adaptive max pooling, Adapt For adaptive average pooling, For element-wise multiplication, For element addition, For the output of the module in the spatial branch, This is a downsampling operation. This is for channel splicing operations. This is the Sigmoid activation function.

[0042] First, the feature maps of the semantic branch are upsampled to the same spatial dimension as the other two branches. Then, channel attention is used to fuse spatial detail features, semantic features, and boundary features. Adaptive average pooling and adaptive maximum pooling are employed to refine the image features, and the sigmoid activation function is used to generate corresponding feature weights. Next, the spatial detail features and semantic features learn these weights separately, enabling the model to focus more on key regions. Finally, residual connections are used to fuse the spatial and semantic branches with the features of the other two branches, respectively, to obtain the final module output.

[0043] Step 4: In this invention, an asymmetric pixel attention pyramid pooling module is established at the end of the semantic branch, and its structure is as follows: Figure 5 As shown, this module helps the model learn multi-scale features in complex scenes, improving its segmentation performance. The output feature map size is 128×256×128. Specifically, it can be represented as:

[0044] (9)

[0045] (10)

[0046] (11)

[0047] (12)

[0048] In the formula: For 1×1 convolution, It is a 1×3 asymmetric convolution. It is a 3×1 asymmetric convolution. For element addition, For element-wise multiplication, up represents bilinear interpolation upsampling. This is average pooling with a kernel size of m and a step size of n. This is global average pooling. This is for channel splicing operations. This is the Sigmoid activation function.

[0049] First, the input feature map X is subjected to average pooling and global average pooling operations to extract multi-scale feature representations. Then, these features are subjected to channel dimensionality reduction using 1×1 convolutional layers and upsampled to the same spatial resolution as the original input features. Next, asymmetric convolution operations (i.e., 3×1 and 1×3 convolutions sequentially) are performed on the dimensionality-reduced features and the multi-scale features to enhance feature expressiveness. All the obtained multi-scale feature maps are concatenated along the channel dimension, and then a 1×1 convolution is used to compress the number of channels, resulting in a fused compressed feature. Simultaneously, the input feature map is transformed using a 1×1 convolution to generate a feature representation at the same scale as the compressed feature. Then, the two are fused using element-wise addition to form a new feature map. At each pixel position of this fused feature map, a sigmoid activation function is used to calculate a pixel-level importance score, generating feature weights for each pixel, thus constructing a pixel attention mechanism for fine-tuning the multi-scale features. Finally, the pixel weight map is multiplied element-wise with the compressed features and the transformed input features respectively, and then added together to obtain the final multi-scale fusion feature output.

[0050] Step 5: Use the bidirectional attention fusion module to efficiently fuse global features, such as... Figure 6 As shown, based on boundary features, spatial detail features and deep semantic features are guided to interact globally through a bidirectional attention mechanism to obtain comprehensive global features. The output feature map size of this module is 128×256×128. Specifically:

[0051] (13)

[0052] (14)

[0053] In the formula: For the input boundary information, It is the sigmoid activation function. Pay attention to the weights of the generated boundary features. It is a spatial branching feature. For semantic branch features, This is an element addition operation. For 1×3 decomposition convolution, For 3×1 decomposition convolution, For 1×1 convolution, This is an element-wise multiplication operation.

[0054] First, attention mechanisms are introduced in both the horizontal and vertical directions to extract feature weights for boundary information. A 1×3 decomposition convolution is used in the horizontal direction, while a 3×1 decomposition convolution is used in the vertical direction to extract more refined boundary features. Then, the attention maps in both directions are normalized using a sigmoid activation function to obtain the corresponding attention weights. Next, the boundary attention weights at different scales are fused with spatial detail features and deep semantic features, guiding the network to focus more on the boundary information of key regions, thus obtaining the final global feature representation.

[0055] Step 6: Use the segmentation head to segment the global features learned by the model, and use 3×3 convolution and 1×1 convolution to output the image segmentation results. The output image size is 19×256×128, and the final segmentation result image is obtained.

[0056] In summary, to address the challenges of extracting object boundary features and information interference caused by a lack of information guidance between different features in real-time semantic segmentation, this invention relates to a real-time semantic segmentation algorithm based on boundary feature-guided fusion. This algorithm employs parallel spatial, semantic, and boundary branches to extract spatial detail features, deep semantic features, and boundary features from the image. At different stages between branches, a feature enhancement fusion module is used, employing a channel attention mechanism to guide the model in fusing local features from different branches. An asymmetric-pixel attention pyramid pooling model is established at the end of the semantic branch to help the model learn multi-scale features in complex scenes. The global features obtained from each branch are then used to achieve global feature interaction through a bidirectional attention fusion module. Finally, the final segmentation result is obtained using a segmentation head.

[0057] The method involved in this invention has superior segmentation performance compared to many existing excellent models, and has advanced learning and generalization capabilities. It can effectively improve the problems of existing models in real-time semantic segmentation, such as difficulty in extracting object boundary features and information interference caused by lack of information guidance between different features.

[0058] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A real-time semantic segmentation method based on boundary feature-guided fusion, characterized in that, Includes the following steps: Step 1: "Fast downsampling" shares shallow features to accelerate model training; Step 2: Use parallel branches to extract features; the spatial branch extracts spatial detail features, the semantic branch extracts deep semantic features, and the boundary branch extracts boundary features. Step 3: Establish feature enhancement and fusion modules at different stages between branches to guide the efficient fusion of different features, and gradually improve the global features with local features; Step 4: Set up an asymmetric pixel attention pyramid pooling module at the end of the semantic branch to learn multi-scale information of the image in complex scenes; Step 5: Establish a bidirectional attention fusion module, based on boundary features, to guide the fusion of spatial and semantic features and improve global information interaction; Step 6: Use a segmentation head consisting of residual blocks and convolutional blocks to perform image segmentation and obtain the final segmentation result.

2. The real-time semantic segmentation method based on boundary feature-guided fusion according to claim 1, characterized in that, The specific method of "fast downsampling" is as follows: the input image is downsampled three times, reducing the resolution to 1 / 2, 1 / 4 and 1 / 8 of the original input image respectively, and then sent to each branch for feature extraction.

3. The real-time semantic segmentation method based on boundary feature-guided fusion according to claim 1, characterized in that, The specific methods for feature extraction in each branch are as follows: the spatial branch and the semantic branch use the mature Basicblock module as the basic module, the Bottleblock module as the bottleneck module for staged feature extraction, and the boundary branch adopts a boundary feature extraction module that focuses on extracting boundary features, and reuses decomposed convolution to extract deep boundary features.

4. The real-time semantic segmentation method based on boundary feature-guided fusion according to claim 1, characterized in that, The specific method of the asymmetric-pixel attention pyramid pooling module is as follows: basic multi-scale features are extracted by average pooling and global average pooling operations, then asymmetric convolution is used to enhance the feature expression capability, and finally, the importance score at the pixel level is calculated by the sigmoid activation function at each pixel position of the multi-scale features to generate the feature weight of each pixel. Combined with the input features, the final multi-scale information is obtained.

5. The real-time semantic segmentation method based on boundary feature-guided fusion according to claim 1, characterized in that, The specific method of the bidirectional attention fusion module is as follows: Based on boundary features, the spatial detail features and deep semantic features are guided to interact globally through a bidirectional attention mechanism to obtain complete global features.

6. The real-time semantic segmentation method based on boundary feature-guided fusion according to claim 1, characterized in that, The specific method by which the segmentation head generates the image segmentation map is as follows: the global feature tensor is processed by a 3×3 convolution and a 1×1 convolution to finally generate the image segmentation map predicted by the model.

Images