A real-time semantic segmentation method based on multi-scale feature interaction and enhancement

Abstract

The application provides a real-time semantic segmentation method based on multi-scale feature interaction and enhancement, and relates to the field of computer vision.The real-time semantic segmentation method based on multi-scale feature interaction and enhancement comprises the following steps: S1, obtaining rich spatial features and semantic features through a backbone network;S2, enhancing high-level semantic features;S3, information interaction of enhanced semantic features and shallow spatial features;S4, fusing multi-scale features to generate a segmentation prediction map.Through aggregating high-level semantic features, the representation ability of context features is effectively enhanced, so that more discriminative features are obtained for processing similar targets;and the fusion of shallow features supplements the spatial detail information of the model, and has good performance in segmenting small objects.The model solves the real-time semantic segmentation challenge through a single feature extraction flow, enhances the spatial and semantic information interaction with minimum calculation cost, and realizes the balance between high-precision segmentation and fast reasoning.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to a real-time semantic segmentation method based on multi-scale feature interaction and enhancement. Background Technology

[0002] Image semantic segmentation is a crucial task in computer vision, aiming to assign a semantic label to each pixel of an image. This process is essential for scene understanding because it yields rich details of information related to the current object, including its category and location. Therefore, semantic segmentation has found wide applications in fields such as autonomous driving, medical image analysis, and augmented reality.

[0003] Traditional semantic segmentation methods rely on manual feature extraction, which is time-consuming and labor-intensive, and can only extract relatively low-level information such as texture and edges. With the development of deep learning technology, semantic segmentation methods based on deep convolutional neural networks (CNNs) can automatically learn more and higher-level image features, making them more suitable for complex and varied real-world scenarios. In recent years, many semantic segmentation methods based on deep CNNs have demonstrated excellent performance. However, these high-performance methods often rely on deeper and larger network models, such as ResNet101, as encoders to extract spatial and semantic information, resulting in high memory consumption and high latency, which greatly limits their application in real-time systems. Currently, some real-time semantic segmentation methods based on deep learning use lightweight encoding modules or significantly reduce network depth, but lose a lot of detailed information; others use a dual-path structure, acquiring spatial and semantic information through multiple branches, but neglecting the communication between branches, leading to poor segmentation accuracy. Therefore, how to simultaneously consider segmentation performance and speed in the model remains a challenge for semantic segmentation tasks. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides a real-time semantic segmentation method based on multi-scale feature interaction and enhancement, which solves the problem of lack of communication between branches in existing multi-path structure real-time semantic segmentation methods.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution: a real-time semantic segmentation method based on multi-scale feature interaction and enhancement, comprising the following steps:

[0008] S1. First, hierarchical features are extracted by using ResNet18 pre-trained on ImageNet as the encoder network; the first three stages of ResNet18 are divided into spatial paths to obtain rich spatial features, while the entire encoding network is regarded as a semantic path to obtain high-level contextual features.

[0009] S2. The high-level contextual features obtained in S1 are injected into the semantic enhancement module, and the high-level semantic features are aggregated through spatial attention branch, channel attention branch and weight learning branch;

[0010] S3. Then, the enhanced semantic features obtained in S2 are sent to the information exchange module to interact with the low-level spatial features in S1 to obtain complementary and more comprehensive feature representations.

[0011] S4. Then, all the results obtained in S3 are fed into the attention fusion module to aggregate the refined spatial and semantic features. Finally, a 1×1 convolutional layer and an upsampling operator are used to generate the final segmentation prediction.

[0012] Preferably, the encoder network described in S1 replaces the 7×7 convolution with three 3×3 convolutional layers as the first residual block of ResNet18 during encoding to reduce computational cost. At the same time, a global average pooling layer is applied at the end of the encoder to obtain global semantic information, and edge maps are predicted based on the output features of residual blocks Res2 and Res3 to guide low-level spatial features to learn more boundary details. By setting the segmentation loss in the semantic enhancement module, high-level features can learn coarse object location information.

[0013] Preferably, the semantic enhancement module employs spatial attention and channel attention to enhance high-level features, and generates two learnable tensors through a weighted learning branch to combine the two types of attention.

[0014] Preferably, in the information exchange module, the three multi-scale features propagate information to each other through a bidirectional vertical propagation path, and the features are refined by an adaptive spatial attention module. The bidirectional vertical propagation path includes two vertical paths, one from bottom to top and one from top to bottom, implemented through concatenation operations and convolutional layers; the adaptive spatial attention module is a spatial attention vector.

[0015] Preferably, the attention fusion module uses three parallel fully connected layers to calculate attention vectors for selecting important features, and limits the sum of the weights of all channels to 1.

[0016] (III) Beneficial Effects

[0017] This invention provides a real-time semantic segmentation method based on multi-scale feature interaction and enhancement. It has the following beneficial effects:

[0018] This invention effectively promotes information exchange between high-level semantic features and low-level spatial features through multi-scale feature interaction, thereby obtaining complementary details and contextual information, providing the network with a more comprehensive feature representation. An attention mechanism is used to optimize multi-scale features, enhancing useful features and weakening useless ones. Through adaptively fused attention vectors, important information in both spatial and channel dimensions is accurately focused, improving the model's ability to distinguish similar objects. Furthermore, the model can refine edges through complementary information, achieving more precise segmentation. The proposed model achieves good performance on the Cityscapes dataset, with an intersection-over-union ratio (IoU) of up to 74.52%, reflecting segmentation accuracy. It also outperforms most current methods in terms of speed, achieving a segmentation speed of 82.56 frames per second, achieving a better balance between accuracy and speed. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating the overall framework structure of the present invention;

[0020] Figure 2 Here is a structural diagram of the semantic enhancement module;

[0021] Figure 3 This is a structural diagram of the information exchange module;

[0022] Figure 4 Here is a structural diagram of the attention fusion module;

[0023] Figure 5 Examples of experimental results in embodiments of the present invention Figure 1 ;

[0024] Figure 6 Examples of experimental results in embodiments of the present invention Figure 2 ;

[0025] Figure 7 For qualitative comparisons on the Cityscapes dataset. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1:

[0028] like Figure 1-4The embodiment of the present invention shown provides a real-time semantic segmentation method based on multi-scale feature interaction and enhancement, including the following steps:

[0029] S1. The input image first extracts hierarchical features through five residual blocks of ResNet18. To reduce computational cost, three 3×3 convolutional layers replace the 7×7 convolution as the first residual block of ResNet18, achieving the same receptive field size while significantly reducing the number of parameters. Then, a global average pooling layer is applied to the output of ResNet18 to obtain global semantic information. The spatial path of the network includes the first three stages of ResNet18 to preserve rich spatial information; while the semantic path uses the entire ResNet18 as an extractor to obtain high-level contextual features. The overall framework structure of this invention is as follows: Figure 1 As shown.

[0030] S2. Inject the high-level contextual features obtained in S1 into the semantic enhancement module, and effectively establish the internal association between adjacent semantic features by using channel and spatial attention mechanisms.

[0031] The semantic enhancement module structure proposed in this invention is as follows: Figure 2 As shown, f4, f5 and f g Three high-level semantic features are upsampled to the same size and concatenated along the channel dimension. They are then initially fused using 3×3 convolutions, batch normalization, and ReLU activation. Two parallel branches are used to compute attention vectors in the channel and spatial dimensions respectively, while another branch is designed to learn the correlation between the two attention vectors. For the channel attention branch, features are first processed through a global average pooling layer to generate global features, and then sequentially processed through two 1×1 convolutional layers to compute the channel attention vector. To reduce the number of parameters, these two convolutional layers are designed as bottleneck structures. The spatial attention branch consists of two 1×1 convolutional layers to obtain the spatial attention vector. The weight learning branch consists of 3×3 convolutional layers, ReLU activation, and a softmax function to generate two tensors, α and β, which are multiplied by the spatial and channel attention vectors respectively, to explore the intrinsic correlation between the spatial and channel attention vectors. The two attention vectors are then fused using element-wise addition and a sigmoid function, as shown in the following equation:

[0032]

[0033] Where σ represents the sigmoid function. It is element-wise multiplication, M S and M C Let $\mathbf$ represent the spatial attention vector and the channel attention vector, respectively. Then, the fused attention vector is multiplied by the feature map to complete the attention-guided semantic feature fusion. Finally, residual connections and convolutional layers are applied to generate the final enhanced semantic feature $f$.s .

[0034] S3. Then, the enhanced semantic feature f obtained in S2 is... s Together with the two low-level features f2 and f3 obtained in S1, they are sent to the information exchange module to realize information interaction and generate semantically and spatially enhanced features.

[0035] The information exchange module structure proposed in this invention is as follows: Figure 3 As shown, the three features first exchange information through two information propagation paths in a bidirectional vertical propagation structure: bottom-up and top-down. The bottom-up information transmission begins by upsampling high-level features, then concatenating them with low-level features along the channel dimension, followed by learning the interaction using convolutional layers. Similarly, the top-down path first transforms low-level features through convolutional layers, then downsamples them before concatenating them with high-level features. In this way, spatial and semantic features can fully exchange information to generate detail-guided semantic features and context-guided spatial features. After information interaction, the three output features are fed into an adaptive spatial attention module to further enhance their representational power. The adaptive spatial attention module first compresses the input features using global average pooling and global max pooling along the channel axis, then applies a 1×1 convolutional layer to generate an attention vector. Simultaneously, the input features generate transformed representation features through convolutional layers with batch normalization and ReLU activation. These transformed features are then multiplied by the attention vector, and the value normalized by the sigmoid function is used as the final attention vector.

[0036] S4. The features obtained in S3 are then processed by the attention fusion module to generate a multi-scale feature fusion result. The feature fusion module structure proposed in this invention is as follows: Figure 4As shown, firstly, small-scale features are upsampled and large-scale feature maps are downsampled. The three features output by the information exchange module are adjusted to 1 / 8 of the original image size, and then they are concatenated along the channel dimension. Choosing a 1 / 8 size is a result of comprehensive consideration, as it achieves a good balance between restoring details and minimizing computational requirements. The concatenated features are then processed through convolutional layers and global average pooling to generate global descriptive features. Next, three parallel fully connected layers capture the correlation between channels, and then the three parallel outputs are concatenated and processed by a softmax function, limiting the sum of the weights of all channels to 1. Afterwards, the weights are re-split into three attention vectors, selectively choosing the more important features from the input features, and aggregating the three attention-enhancing features by simple element-wise addition. Finally, the aggregated features are passed through a 1×1 convolutional layer and upsampled to the original input image size to generate the final segmentation prediction map. Regarding model training, multiple segmentation losses and edge losses are introduced. The segmentation loss uses cross-entropy to supervise the final predicted segmentation and the segmentation map generated by the semantic enhancement module, while the edge loss uses binary cross-entropy to supervise the edge prediction map. The total loss is calculated as follows:

[0037] L = L f +L SEM +λ(L edge2 +L ledge3 )

[0038] Where L f L represents the final segmentation loss. SEM L represents the segmentation loss of the semantic enhancement module. edge2 and L edge3 These are the edge prediction losses generated by Res2 and Res3, respectively. λ is a hyperparameter used to balance the segmentation loss and the edge loss, and is set to 1 in this invention. For ground truth edges, a 3x3 Laplacian operator is used to filter the ground truth segments, and all edge predictions are only retained during the training phase and discarded during the prediction phase.

[0039] Example 2:

[0040] like Figure 1-7The embodiments of the present invention provide a real-time semantic segmentation method with multi-scale feature interaction and enhancement. To verify the effectiveness of the proposed module, ablation experiments were conducted on the Cityscapes dataset. The present invention uses a stepwise addition of components to evaluate the effectiveness of each module; the experimental results are shown in Table 1. The baseline model excluding the semantic enhancement module, information exchange module, and attention fusion module achieved a mIoU of 68.41%. Adding only the semantic enhancement module improved the performance to 71.22%, demonstrating its effectiveness in aggregating high-level semantic features. Because the information exchange module facilitates communication between low-level spatial and high-level semantic features and enhances the representational power of features, it further improved the model accuracy by 5.55%. With the addition of all modules, the model's mIoU increased to 74.71%, and the model's inference speed was slightly lower than the baseline, but still far exceeded the real-time standard (>30fps).

[0041] Table 1. Validation of each module: SEM: Semantic Enhancement Module; IM: Information Exchange Module; AFM: Attention Fusion Module

[0042]

[0043] This invention demonstrates the effectiveness of the semantic enhancement module in aggregating high-level semantic information by comparing four different configurations: (1) using only the spatial attention branch; (2) using only the channel attention branch; (3) combining spatial and channel attention branches additively; and (4) the semantic enhancement module. The experimental results are shown in Table 2. The method combining spatial and channel attention branches additively achieves higher accuracy than the method using only the spatial attention branch, but slightly lower accuracy than the method using only the channel attention branch. This is because the two types of attention focus on different information, and simple element-wise summation may cause mutual interference. The semantic enhancement module uses a weighted learning branch to learn how to effectively combine the spatial and channel attention branches, achieving a 1.11% higher accuracy than the method combining spatial and channel attention branches additively. Figure 5 The diagram shows some segmentation results obtained using only the spatial attention branch, only the channel attention branch, and a combination of spatial and channel attention branches plus a semantic enhancement module. It can be observed that using only the spatial attention branch, only the channel attention branch, and the combination of spatial and channel attention branches incorrectly segment similar classes, for example, labeling a motorcycle as a bicycle. The method using the semantic enhancement module performs better in identifying similar objects, correctly labeling the motorcycle.

[0044] Table 2 Comparison of results for different configurations of aggregated advanced semantic information. SBM: Spatial attention branch only; CBM: Channel attention branch only; SBM+CBM: Combining spatial and channel attention branches through addition; SEM: Semantic enhancement module.

[0045]

[0046] To demonstrate the effectiveness of the information exchange module, this invention compares the use of and absence of the bidirectional vertical propagation structure and the adaptive attention module. As shown in Table 3, when the bidirectional vertical propagation path structure is removed from the information exchange module, the mIoU drops to 73.24%. Similarly, when the adaptive spatial attention module is removed from the information exchange module, the mIoU also decreases to 73.63%. However, when the complete information exchange module is used, important information is emphasized and noise is filtered, achieving an mIoU of 74.71%. These experiments demonstrate that the bidirectional vertical propagation path and adaptive spatial attention are essential components of the information exchange module. Figure 6 The document presents some visualizations comparing the input feature maps (first row) and output feature maps (second row) of the information exchange module. It can be observed that after the information exchange module, shallow spatial feature maps (e.g., f2, f3) focus more on contextual information, while high-level semantic features (f...)... s (Carry more details)

[0047] Table 3 Comparison of the effectiveness evaluation of the information exchange module. w / o BVP: removal of bidirectional vertical propagation path; w / o ASA: removal of adaptive spatial attention module; IM: information exchange module

[0048]

[0049] The results of this invention on the Cityscapes test set are compared with current state-of-the-art segmentation methods, as shown in Table 4. For non-real-time models, such as PSPNet and CCNet, their mIoU is higher than that of the model proposed in this invention. However, these methods have high memory and latency response, making them difficult to deploy in real-time systems. Compared with nine real-time segmentation methods, the model proposed in this invention achieves an mIoU accuracy of 74.52% at 82.56 fps, exhibiting higher segmentation accuracy and faster inference speed than most methods. Although PBSNet's mIoU is 0.18% lower than BiSeNet, which also uses ResNet18 as the encoder, it is more efficient in terms of parameters and speed.

[0050] Table 4 Comparison of different models on the Cityscapes test set

[0051]

[0052]

[0053] Figure 7This section showcases some visualizations of BiSeNet and PBSNet. It can be seen that PBSNet demonstrates more accurate segmentation of objects such as roads and sidewalks, cars and trucks. These classes often appear together and have similar features, making them difficult to distinguish. PBSNet utilizes SEM to explore the internal relationships of high-level semantic features, improving its ability to distinguish similar objects. Furthermore, PBSNet exhibits superior performance in segmenting small objects (such as people in the first column and poles in the second column). This is likely due to the IM module, which supplements the network with rich spatial details.

[0054] This invention delves into the enhancement and interaction of semantic and spatial features, proposing a novel real-time semantic segmentation method. At the decoding end, a semantic enhancement module explores the intrinsic relationships between high-level features, while spatial features retained by the encoder supplement the network with detailed information. An information exchange module enhances the interaction between spatial and semantic features, and finally, an attention fusion module aggregates the refined spatial and semantic features to generate the final segmentation prediction. This invention achieves competitive results on the Cityscapes dataset.

[0055] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time semantic segmentation method based on multi-scale feature interaction and enhancement, characterized in that: The real-time semantic segmentation method comprises the following steps: S1. Firstly, hierarchical features are extracted by taking ResNet18 pre-trained on ImageNet as an encoder network; the first three stages of ResNet18 are divided into a spatial path to obtain rich low-level spatial features, and the entire encoding network is taken as a semantic path to obtain high-level context features; S2. The high-level context features obtained in S1 are injected into a semantic enhancement module, and high-level semantic features are aggregated through a spatial attention branch, a channel attention branch and a weight learning branch to finally obtain enhanced semantic features; The spatial attention branch is composed of two 1x1 convolution layers; the channel attention branch is composed of a global average pooling layer and two 1x1 convolution layers; and the weight learning branch is composed of a 3x3 convolution layer, a ReLu activation and a softmax function; S3. Then, the enhanced semantic features obtained in S2 are sent to an information exchange module to interact with the low-level spatial features in S1, so as to realize information interaction and generate features with enhanced semantics and space; S4. The results obtained in S3 are all sent to an attention fusion module to aggregate refined spatial features and semantic features, and finally a 1x1 convolution layer and an upsampling operator are used to generate a final segmentation prediction. 2.The method of claim 1, wherein: The encoder network in S1 replaces a 7x7 convolution with three 3x3 convolution layers as the first residual block of ResNet18 to reduce the computational cost during encoding, and applies a global average pooling layer at the end of the encoder to obtain global semantic information. According to the output feature prediction of the residual blocks Res2 and Res3, an edge map is generated to guide the low-level spatial feature to learn more boundary details, and an auxiliary segmentation loss is used to allow the semantic enhancement module to capture more semantic information, so that the high-level features can learn rough object position information. 3.The method of claim 1, wherein: The semantic enhancement module uses spatial attention and channel attention to enhance high-level features, and generates two learnable tensors through a weight learning branch to combine the two attentions. 4.The method of claim 1, wherein: In the information exchange module, three multi-scale features propagate information to each other through a bidirectional vertical propagation path, and the features are refined through an adaptive spatial attention module. The bidirectional vertical propagation path includes a bottom-up and a top-down vertical path, which are realized through a concatenation operation and a convolution layer. The adaptive spatial attention module is a spatial attention mechanism.

5. The method of claim 1, wherein: The attention fusion module uses three parallel fully connected layers to calculate attention vectors for selecting important features, and limits the sum of weights of all channels to 1.

Images