Real-time semantic segmentation method for multi-scale information fusion based on attention guidance

Abstract

The invention discloses a real-time semantic segmentation method for attention-guided multi-scale information fusion, and the method comprises the following steps: 1, constructing a multi-scale feature aggregation module, and aggregating multi-scale detail information; 2, constructing an attention guiding feature correction module, and correcting information extracted by different branches to promote fusion of multi-scale information and local information; and 3, constructing an attention guiding feature reduction module, fusing feature maps extracted at different depths in the network, and making up image information lost in a feature transmission process. And 4, building an image segmentation network based on the three modules, efficiently extracting important features in the image, and accurately segmenting the complex scene image through a deep learning mode. A novel real-time scene semantic segmentation model is designed on the basis of an encoder-decoder architecture, the model precision and the model speed are better balanced, and under the conditions of low parameter and low calculation cost, accurate segmentation of image objects can be achieved, and meanwhile real-time performance is achieved.

Description

Technical Field

[0001] The present invention relates to the field of scene segmentation and autonomous driving, and in particular to a real-time semantic segmentation method in a road scene. Background Art

[0002] The semantic segmentation model takes each pixel of the original image as input and performs a phased local or global understanding in the deep neural network. Then the pixels are divided into multiple categories, and the objects to be segmented in the input image are located and assigned the same category of semantic labels according to the specific application tasks. The improvement of the semantic segmentation task effect is of great help to fields such as autonomous driving, scene monitoring and drones. For example, in the field of autonomous driving, the detection of lane lines or pedestrians on zebra crossings on the vehicle's route during the driving of unmanned vehicles ensures the safety of vehicle driving. It is also applicable to the field of drones. Semantic segmentation can help forest rangers detect and locate forest fires in the area to avoid casualties. However, similar fields such as virtual reality have extremely important requirements for the accuracy and real-time judgment of semantic segmentation models. Its huge computing cost and the model reasoning load brought by large parameters require researchers to achieve a better balance between reasoning speed and segmentation performance.

[0003] In order to balance this demand for performance and speed, researchers have proposed many strategies to improve the complex network models by optimizing convolution. ShuffleNet combines point-by-point group convolution and channel shuffle to achieve an efficient and lightweight mobile network design. DABNet combines the advantages of deep asymmetric convolution and dilated convolution to design a new bottleneck structure. MobileNet is built using deep separable convolution, which greatly reduces the amount of computation and parameters compared with standard convolution. The DeepLab series uses dilated convolution to solve the problem of loss of original image information caused by the reduction of feature map resolution and designs cascaded or parallel dilated convolution modules to solve the problem of multi-scale information fusion. The ICNet decoder uses dilated convolution to refine upsampled features. Compared with deconvolution, a smaller convolution kernel size obtains the same receptive field. However, more or less models have reduced speed while improving performance, and the opposite is true. Obviously, in order to achieve a better balance between inference speed and segmentation performance, more research work is still needed to solve this urgent problem.

[0004] Therefore, how to design a semantic segmentation network with fast inference speed, small capacity and high segmentation accuracy becomes a challenging problem. Summary of the invention

[0005] The present invention provides a real-time semantic segmentation method based on attention-guided multi-scale information fusion. The method is constructed on the basis of a multi-scale feature aggregation module, an attention-guided feature correction module and an attention-guided feature restoration module. Its encoder uses two means, multi-scale information feature aggregation and attention-guided feature correction, to achieve efficient feature extraction. In contrast, its decoder focuses on the fusion and restoration of features at different levels from two perspectives, space and channel, and realizes pixel-level dense prediction and classification with the help of adaptive refinement and interaction of features. Its technical solution is as follows

[0006] A real-time semantic segmentation method based on attention-guided multi-scale information fusion, comprising the following steps:

[0007] Step 1: Extract the initial local features of the network's initial input and downsample and splice the original image to output it to the channel attention module;

[0008] Step 2: Extract channel information features and input the data into the downsampling module;

[0009] Step 3: The data is processed by the downsampling module and then output to the attention-guided multi-scale information fusion convolution block;

[0010] Step 4: Process the data through the multi-scale feature aggregation module and the attention guidance feature correction module respectively, fuse the extracted image information, and input the data into the attention guidance feature restoration module;

[0011] Step 5: Sample the data to a resolution consistent with the original image and output the final network predicted scene segmentation result.

[0012] Furthermore, the initial convolution module obtains the original input of the network, which consists of three standard convolutions with a kernel size of 3, and plays the initial local feature extraction role. The stride of the first convolution is set to 2, and the rest are 1. The initial convolution module implements downsampling of the image, and the obtained initial feature map is used as the input of the lightweight channel attention module.

[0013] Furthermore, the channel attention module is derived from ECANet, which is a lightweight attention model based on the SENet module. Without significantly increasing the parameters of the model, the performance gain brought by cross-channel interaction enables our proposed method to achieve better feature extraction. This module obtains the global information of the channel through global average pooling, and then uses one-dimensional convolution to generate lightweight channel attention weights, thereby enhancing the feature expression ability of important channels.

[0014] Furthermore, the downsampling module is applied to all downsampling layers, and the module flexibly switches between two different operation modes according to the relationship between the number of output channels and the number of input channels. If the number of output channels is less than the number of input channels, the module is simplified to a 3×3 convolution layer with a step size of 2, and directly performs feature dimension reduction. On the contrary, if the number of output channels is not less than the number of input channels, the module first compresses the channel dimension through a 2×2 maximum pooling layer with a step size of 2, and then fuses the feature maps processed by the convolution layer and the pooling layer, and finally outputs the downsampled feature representation.

[0015] Furthermore, the attention-guided multi-scale information fusion convolutional block includes a multi-scale feature aggregation module and different numbers of attention-guided feature correction modules. The first convolutional block consists of one and two continuously stacked multi-scale feature aggregation modules, while the second one consists of one attention-guided correction module and five continuously stacked multi-scale feature aggregation modules.

[0016] The multi-scale feature aggregation module uses standard convolution and superimposed non-continuous dilated convolution to extract features. The number of output channels is halved successively to keep the resolution of the feature map unchanged. Through the splicing operation, different context information is input into a standard 3x3 convolution to integrate features and a 1x1 convolution to achieve channel interaction. Finally, the use of channel shuffling operation realizes the flow of information and improves the expression of features.

[0017] The attention-guided correction module first uses a standard convolution to reduce the size of the feature. It consists of three branches. The first branch is a multi-scale information branch, which is composed of asymmetric dilated convolutions in the depth direction. The second branch is a local information branch, which uses a standard 3x3 convolution to extract local features of the feature map. The third branch is an attention branch, which uses global average pooling and one-dimensional convolution to generate the channel attention weights of the original input to weight the feature maps output by the other two branches, guide the feature fusion of the two at the end of the stage and input the fused feature map to the next three branches to repeat the above operation. The feature maps output by the three three branches are spliced ​​in the channel dimension and then input into the ECA module. Next, the 1x1 convolution restores the number of channels input to the module and increases the nonlinear representation of the module to achieve linear combination of information between feature map channels. Finally, the bilinear interpolation method is used to restore the feature map to the corresponding size of the module input.

[0018] Furthermore, the attention-guided feature restoration module uses bilinear interpolation and 1x1 convolution to match the resolution and channels of features at different stages. First, the deep 1 / 8 feature map is integrated into the middle 1 / 4 feature map. Then it is upsampled and the number of channels is reconstructed to facilitate subsequent operations. Next, the middle 1 / 4 feature map and the shallow 1 / 2 feature map are spliced ​​in the channel dimension, and the combined feature map will be passed into the attention modules of the two branches. The left side is the spatial attention module in the CBAM module, and the right side is the attention branch used in the AFC unit. With the particularity of the attention mechanism, we pay attention to the spatial and channel dimensions of the combined feature map respectively, and compensate for the part of the information lost by the middle 1 / 4 feature map and the shallow 1 / 2 feature map through learning interaction from two angles. Attention weighted guided feature fusion enables the module to perform adaptive feature refinement. Next, 3x3 convolution is used to further promote the interaction between features, and finally the number of feature map channels is borrowed by 1x1 convolution to meet the requirements of semantic segmentation tasks.

[0019] Furthermore, the constructed real-time semantic segmentation model can achieve accurate segmentation of image objects with low parameters and low computational cost, while being real-time.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] Advantage 1: This invention takes advantage of decomposition convolution, depthwise separable convolution and dilated convolution, and based on the multi-scale feature aggregation module, captures richer image features at the shallow stage, making the network more capable of capturing information;

[0022] Advantage 2: The feature correction module of the present invention is based on attention guidance. The module extracts effective image features and reduces feature redundancy by guiding attention weights, and can realize the correction and fusion of multi-scale features and local features;

[0023] Advantage 3: The feature correction module based on attention guidance of the present invention fuses the feature maps output at different stages of the network, focusing on the restoration of features in the dimensions of space and channels;

[0024] Advantage 4: The present invention can accurately segment high-resolution complex road scene images at real-time speed, which is of great help to the fields of autonomous driving, scene monitoring and drones.

[0025] In summary, the technical solution of the present invention achieves a better balance between reasoning speed and segmentation performance. Based on the comprehensive use of the multi-scale feature aggregation module and the attention-guided feature correction module, the feature extraction, fusion and restoration of the model are guided by the attention weight. With less increase in computational cost, the performance of the model we constructed is improved to a good extent. Finally, with the help of the attention-guided feature restoration module, the feature maps extracted from different depths in the network are fused to make up for the image information lost during the feature transmission process. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 Schematic diagram of the structure of the channel attention module of the present invention;

[0028] Figure 2 It is a structural schematic diagram of the downsampling module of the present invention;

[0029] Figure 3 Schematic diagram of the overall structure of the network model of the lightweight semantic segmentation method of the present invention;

[0030] Figure 4 It is a structural schematic diagram of the multi-scale feature aggregation module of the present invention;

[0031] Figure 5 Schematic diagram of the structure of the attention guiding feature correction module of the present invention;

[0032] Figure 6 Schematic diagram of the structure of the attention guidance feature restoration module of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution in the example of the present invention will be clearly and completely described below in conjunction with the drawings in the example of the present invention. Obviously, the described example is only an embodiment of a part of the present invention, not all embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0034] like Figure 1 As shown, the real-time semantic segmentation method based on attention-guided multi-scale information fusion of the present invention comprises the following steps:

[0035] Step 1: Extract the initial local features of the network’s initial input and downsample and splice the original image to output it to the channel attention module.

[0036] The scene image to be segmented passes through an initialized convolution module consisting of three convolutions with a kernel size of 3. This reduces the image resolution and extracts the image information initially. It is then spliced ​​with the downsampled original image and input into the channel attention module.

[0037] Step 2: The features output from step 1 are used as the input of the channel attention module to extract the channel information features and input the data into the downsampling module.

[0038] Lightweight attention module, i.e. Figure 3 The ECA module in the method can increase the interaction of feature map channels and ensure the performance of the network.

[0039] Step 3: Process the output data of step 2 through a downsampling module, reduce the image resolution, and then output it to the attention-guided multi-scale information fusion convolution block.

[0040] At each stage of the downsampling layer (such as Figure 3 In Figure 1, a downsampling module is designed with two operation modes. When the number of output channels is less than the number of input channels, the module is simplified to a 3×3 convolution operation with a stride of 2. However, when the number of output channels is equal to or greater than the number of input channels, the module first performs a 2×2 max pooling operation with a stride of 2, and then fuses the convolution and pooling feature maps to generate the final downsampled output, as shown in Figure 1. Figure 2 shown.

[0041] In step 4, the output of step 3 is used as the input of the attention-guided multi-scale information fusion convolution block. The extracted image information is fused through the multi-scale feature aggregation module and the attention-guided feature correction module respectively to make up for the image information lost during data transmission, and the data is input into the attention-guided feature restoration module.

[0042] The first convolutional block consists of one and two consecutively stacked multi-scale feature aggregation modules, while the second one consists of an attention-guided correction module and five consecutively stacked multi-scale feature aggregation modules.

[0043] Multi-scale feature aggregation module (MFA Module), such as Figure 4As shown in the figure, standard convolution and superimposed non-continuous dilated convolution are used for feature extraction. The number of output channels is halved successively, keeping the resolution of the feature map unchanged. Through the splicing operation, context information of different scales is input into a standard 3x3 convolution to integrate features for better feature interaction and a 1x1 convolution to achieve channel interaction. Finally, the use of channel shuffling operation realizes the flow of information and improves the expression of features. The principle of the MFA module we designed can be summarized by equations (1)(2)(3)(4):

[0044] U 1 =F 3x3 (U) (1)

[0045]

[0046] U ′ = F Att (F 1x1 ( F 3x3 ( Cat(U 1 ,U 2 ,U 3 ))) (3)

[0047]

[0048] Assume that the module input is U, and the information output at different stages is U n , F 3x3 and F 1x1 is a standard convolution with different convolution kernels, is the depth-wise discontinuous dilated convolution with different dilation rates, F Att is the ECA attention module, F Shuffle is the channel shuffle operation, This is the information output by the last module.

[0049] Attention guidance correction module (AFC Module), such as Figure 5As shown in the figure, the AFC module can capture the long-distance dependencies of image features and perform attention-based feature correction and more refined feature extraction on feature maps. Because it is applied to large-resolution scene images, the step size of the first 3x3 convolution in the module is set to 2 to reduce the computational consumption of feature map transmission. The feature map transmitted later has three branches to continue to extract corresponding features. The first branch in the AFC unit is a multi-scale information branch, which is composed of asymmetric dilated convolutions in the depth direction. Combined with the low computational complexity brought by discontinuous convolutions and the advantages of the large receptive field of dilated convolutions, this branch can quickly extract richer and more comprehensive multi-scale information. The second branch is a local information branch, which uses a standard 3x3 convolution to extract local features of the feature map. This branch pays more attention to local close-range information. The third branch is an attention branch that captures more accurate information and avoids noise interference. The feature maps output by the three units are spliced ​​in the channel dimension and then input into the ECA module. Without significantly increasing the parameters of the model, the performance gain brought by cross-channel interaction enables the AFC module to better extract features. Next, the 1x1 convolution restores the number of channels input to the module and increases the nonlinear representation of the module. Finally, bilinear interpolation is used to restore the feature map to the corresponding size of the module input.

[0050]

[0051]

[0052] Assume that the input of the AFC module is U, where the input and output information of different AFC units are U n and U′ n . F conv1d is a one-dimensional convolution with a kernel size of 3, F 3x3 and F 1x1 is a standard two-dimensional convolution with different convolution kernels, Depthwise separated convolutions for different dilation rates. is the attention weight in different units. Similarly, F Att For ECA module. GAP is the global average pooling operation, σ is the Sigmoid activation function, F T Represents dimensional operations such as transposition and compression. bilnear It is bilinear interpolation. Cat means concatenating the outputs of three AFC units in the channel dimension. This is the information output by the last AFC module.

[0053] Since dilated convolution is used in the MFA module and the AFC module, the dilation rates are set to [[2,5],[2,5],[5,5,5]] and [[3,7],[3,7],[3,7],[9,11],[9,11],[11,11,11]].

[0054] In step 5, the output data of step 4 is upsampled to a resolution consistent with the original image through the attention-guided feature restoration module, and the final network-predicted scene segmentation result is output.

[0055] Multi-scale feature restoration module (AFR Module), such as Figure 6 As shown in Figure 1, there are three feature maps of different sizes in the network, namely the shallow 1 / 2, middle 1 / 4 and deep 1 / 8 feature maps. We use bilinear interpolation and 1x1 convolution to match the resolution and channels of features at different stages. First, the deep 1 / 8 feature map is integrated into the middle 1 / 4 feature map. At this time, the information of the middle 1 / 4 feature map is supplemented by the information of the deep 1 / 8 feature map, which means that the middle 1 / 4 feature map is Figure 1 It enhances the semantic features to a certain extent without losing a lot of spatial information. Then it is upsampled and the number of channels is reconstructed to facilitate subsequent operations. Next, the middle layer 1 / 4 feature map and the shallow layer 1 / 2 feature map are spliced ​​in the channel dimension, and the combined feature map will be passed into the attention modules of the two branches, the left side is the spatial attention module, and the right side is the attention branch used in the AFC unit. With the particularity of the attention mechanism, we pay attention to the spatial and channel dimensions of the combined feature map respectively, and make up for some of the information lost by the middle layer 1 / 4 feature map and the shallow layer 1 / 2 feature map through learning interaction from two angles. Attention weighting guides feature fusion so that the module can perform adaptive feature refinement. Next, 3x3 convolution is used to further promote the interaction between features, and finally the number of feature map channels borrows 1x1 convolution to meet the requirements of semantic segmentation tasks.

[0056] The above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0057] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain example, reference can be made to the relevant descriptions of other embodiments.

[0058] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions described in the above embodiments can still be modified, or some or all of the technical features can be replaced by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time semantic segmentation method based on attention-guided multi-scale information fusion, characterized by: The following steps are involved: Step 1: Extract the initial local features of the network's initial input and downsample and splice the original image to output it to the channel attention module; Step 2: Extract channel information features and input the data into the downsampling module; Step 3: The data is processed by the downsampling module and then output to the attention-guided multi-scale information fusion convolution block; Step 4: Process the data through the multi-scale feature aggregation module and the attention guidance feature correction module respectively, fuse the extracted image information, and input the data into the attention guidance feature restoration module; Step 5: Sample the data to a resolution consistent with the original image and output the final network predicted scene segmentation result.

2. The real-time semantic segmentation method based on attention-guided multi-scale information fusion as claimed in claim 1, characterized in that: In step 1, the scene image to be segmented is passed through an initialized convolution module consisting of three convolutions with a convolution kernel size of 3, which reduces the image resolution and initially extracts the image information.

3. The real-time semantic segmentation method based on attention-guided multi-scale information fusion as claimed in claim 1, characterized in that: The downsampling module in step 3 is simplified to a 3×3 convolution operation with a stride of 2 when the number of output channels is less than the number of input channels; when the number of output channels is equal to or greater than the number of input channels, a 2×2 maximum pooling operation with a stride of 2 is performed first, and then the convolution and pooling feature maps are fused to generate the final downsampled output.

4. The real-time semantic segmentation method based on attention-guided multi-scale information fusion as claimed in claim 1, characterized in that: In step 4, the multi-scale feature aggregation module uses standard convolution and superimposed non-continuous dilated convolution to perform feature extraction. The number of output channels is halved successively to keep the resolution of the feature map unchanged. Through the splicing operation, context information of different scales is input into a standard 3x3 convolution to integrate features for better feature interaction and a 1x1 convolution to achieve channel interaction. Finally, the channel shuffling operation is used to realize the flow of information and improve the expression of features.

5. The real-time semantic segmentation method based on attention-guided multi-scale information fusion as claimed in claim 1, characterized in that: The first branch of the attention-guided correction module in step 4 is a multi-scale information branch, which is composed of asymmetric dilated convolutions in the depth direction. The second branch is a local information branch, which uses a standard 3x3 convolution to extract local features of the feature map. The third branch is the attention branch.

6. The real-time semantic segmentation method based on attention-guided multi-scale information fusion as claimed in claim 1, characterized in that: In step 5, a multi-scale feature restoration module is used. First, the deep 1 / 8 feature map is integrated into the middle 1 / 4 feature map. Then, the middle 1 / 4 feature map and the shallow 1 / 2 feature map are concatenated in the channel dimension. The combined feature map is passed to the attention modules of two branches, namely the spatial attention module and the attention branch used in the AFC unit. Next, 3x3 convolution is used to further promote the interaction between features. Finally, the number of feature map channels is borrowed by 1x1 convolution to meet semantic segmentation.

Images