A real-time semantic segmentation method for traffic scene images for autonomous driving

Abstract

The present invention discloses a method for real-time semantic segmentation of traffic scene images for autonomous driving, which utilizes a data set of semantic segmentation of urban traffic scenes to construct a multi-channel ternary aggregation real-time semantic segmentation network through training; the multi-channel ternary aggregation real-time semantic segmentation network consists of a detail branch, a semantic branch and an aggregation layer; in the MCTANet network constructed in the method for real-time semantic segmentation of traffic scene images, a dynamic space enhancement module is designed in the detail branch, which makes full use of spatial information of different scales without introducing too much computational complexity; a global context sharing module is designed in the semantic branch, which can effectively enhance the ability of the semantic branch to obtain the global receptive field, and the network reasoning speed is almost unaffected; the aggregation layer adopts a ternary guided aggregation layer, which complements and fuses the features of the above two branches to improve the segmentation accuracy; compared with the existing BiSeNetV1 and BiSeNetV2 networks, the method has higher segmentation accuracy while maintaining the reasoning speed.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception fusion technology, and in particular to a real-time semantic segmentation method for traffic scene images used for autonomous driving. Background Art

[0002] Autonomous driving technology has developed rapidly and received widespread attention in today's world. With the advancement of artificial intelligence and computer vision, autonomous driving has become one of the key innovations that will change the future of transportation. One of the core challenges of this technology is the perception and understanding of the vehicle's surrounding environment. In this context, the importance of real-time semantic segmentation technology has become increasingly prominent. Real-time semantic segmentation is an advanced image processing technology that can segment images captured by tools such as on-board cameras or lidar into regions with semantic information, and identify and distinguish important elements such as roads, vehicles, and pedestrians. This technology is crucial for autonomous vehicles. It provides vehicles with key environmental perception capabilities and is one of the foundations for achieving safe and efficient autonomous driving.

[0003] With the continuous development of deep learning and computer vision, semantic segmentation algorithms are also evolving. High-precision semantic segmentation algorithms often focus on segmentation accuracy, but may have problems such as slow computing speed and inability to meet the needs of real-time applications. The emergence of real-time semantic segmentation algorithms fills this gap. It can not only achieve faster processing speed while maintaining high accuracy, but also meet the needs of real-time applications such as autonomous driving. The advantage of real-time semantic segmentation algorithms is that they can accurately and real-time analyze the environment around the vehicle within milliseconds, helping autonomous vehicles make timely and accurate decisions, and improving the safety and stability of autonomous vehicles in complex traffic scenarios.

[0004] See also Figure 2As an advanced real-time semantic segmentation network structure, BiSeNetV2 has made significant progress based on the improvement of BiSeNetV1. Compared with BiSeNetV1, BiSeNetV2 proposes a guided aggregation layer to replace the FFM module to effectively fuse features with different depths and resolutions. In addition, during the training stage, BiSeNetV2 introduces a new training strategy to enhance features, thereby improving the network segmentation accuracy. These improvements enable BiSeNetV2 to better capture multi-scale information, improve the ability to express semantic information, and increase processing speed while maintaining high accuracy, making it more suitable for real-time autonomous driving scenarios. However, with the increasing complexity of urban traffic scenes, the segmentation accuracy of BiSeNetV2 needs to be further improved. Complex urban traffic scenes contain more categories of backgrounds and more small object targets, but BiSeNetV2 does not perform well in the segmentation of small targets. Therefore, it is necessary to improve the segmentation accuracy based on the real-time performance of the BiSeNetV2 network, which has become a valuable research topic. Summary of the invention

[0005] The purpose of the present invention is to provide a real-time semantic segmentation method for traffic scene images for autonomous driving that solves the above-mentioned technical problems.

[0006] To this end, the technical solution of the present invention is as follows:

[0007] A real-time semantic segmentation method for traffic scene images for autonomous driving uses a semantic segmentation dataset for urban traffic scenes to train a constructed multi-channel ternary aggregation real-time semantic segmentation network MCTANet, so as to achieve real-time semantic segmentation results by inputting urban traffic scene images into the network: wherein the multi-channel ternary aggregation real-time semantic segmentation network MCTANet is composed of a detail branch, a semantic branch and an aggregation layer; wherein the detail branch is composed of a first 2x downsampling module, a second 2x downsampling module, a first splicing convolution module, a dynamic space enhancement module, a multiplication operation module and a third 2x downsampling module connected in sequence; the output end of the first 2x downsampling module is also connected to the first splicing convolution module, and the output end of the first splicing convolution module is also connected to the input end of the multiplication operation module; the dynamic space enhancement module adopts a multi-scale dynamic space attention mechanism to process the output result of the first splicing convolution module through dynamic attention weights; the semantic branch is composed of a first 2x downsampling module, a second 2x downsampling module, a third 2x downsampling module, a fourth 2x downsampling module, a fifth 2x downsampling module, a global upper and lower sharing module, a first The output end of the fourth 2-fold downsampling module is also connected to the input end of the sum operation module in the semantic branch; the global up-down sharing module is designed based on the deep convolutional pyramid module DCPP; the aggregation layer includes a detail branch spatial processing module and a detail branch spatial fusion module connected to the detail branch, and a semantic branch context processing module and a semantic branch context fusion module connected to the semantic branch; the detail branch spatial processing module and the semantic branch context fusion module, as well as the detail branch spatial fusion module and the semantic branch context processing module are cross-multiplied and input into the sequentially connected splicing convolution module, ternary fusion layer and S-Head module; the ternary fusion layer is based on the improvement of the channel dimension processing layer of the ternary attention mechanism, and the channel dimension processing layer is composed of a global average pooling layer with a pooling kernel of 1×1, a first View operation module, a four-branch fully connected layer, a Concat module, a second View operation module, a Sigmoid function and a multiplication operation module connected in sequence, and the input end of the multiplication operation module of the ternary fusion layer is also connected to the output end of the first splicing convolution module of the aggregation layer.

[0008] Furthermore, the method for constructing a semantic segmentation dataset of urban traffic scenes is as follows: 1) using vehicle-mounted cameras and cameras to collect urban traffic scene images containing a variety of different categories of features; 2) according to the classification standards of open source semantic segmentation datasets, the collected urban traffic scene images are labeled to obtain urban traffic scene labeled images; 3) random horizontal flipping method, random scaling method and random cropping to fixed size method are used in turn for data enhancement; 4) all urban traffic scene labeled images are constructed to form an urban traffic scene semantic segmentation dataset.

[0009] Furthermore, the dynamic space enhancement module is composed of a first processing module and a second processing module; the first processing module is composed of an average pooling layer, a maximum pooling layer, a second splicing convolution module and a 1×1 convolution layer; the input ends of the average pooling layer and the maximum pooling layer are connected to the output end of the first splicing convolution module in the detail branch, and the output ends of the two are connected to the input end of the second splicing convolution module of the detail branch, and the output end of the second splicing convolution module is connected to the input end of the 1×1 convolution layer; the second processing module includes a first processing module composed of a 3×3 convolution layer, a ReLU activation function and a Sigmoid function connected in sequence. A first branch, a second branch consisting of a 5×5 convolutional layer, a ReLU activation function and a Sigmoid function connected in sequence, a third branch consisting of a 7×7 convolutional layer, a ReLU activation function and a Sigmoid function connected in sequence, an addition operation module and a multiplication operation module; the convolution layer input ends of the first branch, the second branch and the third branch are connected to the output end of the 1×1 convolutional layer in the first processing module, and the output ends of the three are connected to the input end of the addition operation module; the output end of the addition operation module and the output end of the first splicing convolution module in the detail branch are connected to the input end of the multiplication calculation module.

[0010] Furthermore, the global upper and lower sharing module includes a DCPP module, a first splicing convolution module, a second 1×1 convolution layer, a BN layer, a ReLU activation function and an addition operation module, and a first 1×1 convolution layer connected in sequence; the input end of the first 1×1 convolution layer and the input end of the DCPP module are connected to the output end of the fifth 2x downsampling module, the output end of the first 1×1 convolution layer is connected to the input end of the first splicing convolution module in the global context sharing module, and the output end of the second 1×1 convolution layer is also connected to the input end of the addition operation module.

[0011] Further, the DCPP module includes: a first branch consisting of a global average pooling layer with a pooling kernel of 1×1, a third 1×1 convolution layer, a second addition operation module and a 3×3 convolution layer connected in sequence, a second branch consisting of a global average pooling layer with a pooling kernel of 2×2, a third 1×1 convolution layer, a second addition operation module and a 3×3 convolution layer connected in sequence, a third branch consisting of a global average pooling layer with a pooling kernel of 4×4, a third 1×1 convolution layer, a second addition operation module and a 3×3 convolution layer connected in sequence, a fourth 1×1 convolution layer and a second concatenation convolution module; the input end of the global average pooling layer and the input end of the fourth 1×1 convolution layer in the three branches are both connected to the output end of the fifth 2x downsampling module; the output end of the fourth 1×1 convolution layer is respectively connected to the input end of the second addition operation module in the three branches, and the output end of the fourth 1×1 convolution layer and the output end of the 3×3 convolution layer in the three branches are all connected to the input end of the second concatenation convolution module.

[0012] Furthermore, the 2x downsampling modules in the detail branch and the semantic branch are composed of a 3×3 convolutional layer, a BN layer, and a ReLU activation function connected in sequence; in the detail branch, the semantic branch, and the aggregation layer, the convolutional splicing module is composed of a Concat module and a 1×1 convolutional layer connected in sequence.

[0013] Furthermore, the 2x upsampling modules in the semantic branch are composed of a channel calculation module, a 1×1 convolution layer, and a pixel reorganization module connected in sequence; wherein the channel calculation module is provided with an output channel number calculation formula, which is expressed as follows:

[0014] out chan =input chan ×factor×factor,

[0015] In the formula, out chan is the number of output channels; input chan is the number of input channels; factor is the upsampling factor, which is set according to the multiple of the output feature map relative to the input feature map.

[0016] Further, in the polymer layer,

[0017] In the aggregation layer,

[0018] The detail branch spatial processing module is composed of a 3×3 deep convolution layer, a BN layer, a 1×1 convolution layer and a Sigmoid function connected in sequence. The detail branch spatial fusion module is composed of a 3×3 convolution layer, a BN layer, a maximum pooling layer with a convolution kernel of 3×3 and an addition operation module connected in sequence. The input end of the 3×3 deep convolution layer, the input end of the 3×3 convolution layer and the input end of the addition operation module are connected to the output end of the third 2x downsampling module of the detail branch.

[0019] The semantic branch context processing module is composed of a 3×3 deep convolutional layer, a BN layer, a 1×1 convolutional layer and a Sigmoid function connected in sequence. The semantic branch context fusion module is composed of a 3×3 convolutional layer, a BN layer, an average pooling layer with a convolution kernel of 3×3 and an addition operation module connected in sequence. The input of the 3×3 deep convolutional layer, the input of the 3×3 convolutional layer and the input of the addition operation module are connected to the output of the second 2x upsampling module of the semantic branch.

[0020] Furthermore, the training of the multi-channel ternary aggregation real-time semantic segmentation network MCTANet is completed based on the deep learning framework; the initial learning rate is set to 5e -2 , the weight decay value is 5e -4 ; According to the poly learning rate strategy, set the learning rate in subsequent iterations to: Where iter is the number of iterations in any subsequent training phase, iters max The number of iterations to find the minimum value of the loss function; the stochastic gradient descent method with momentum = 0.9 is set to update the weights and biases of each stage of the MCTANet network; the batch size is set to a positive integer ranging from 3 to 7.

[0021] Furthermore, the loss function of network training adopts the cross entropy loss (CE Loss) function.

[0022] Compared with the prior art, the real-time semantic segmentation method of traffic scene images for autonomous driving has the following beneficial effects:

[0023] (1) The MCTANet network constructed in the method of the present application has a dynamic spatial enhancement module in its detail branch that is designed based on a multi-scale dynamic spatial attention mechanism. It uses convolution kernels of different sizes to capture spatial features of different scales, which can enhance the network's perception of multiple spatial structural features, allowing the model to pay attention to local details and global structures at the same time, so that the network can dynamically adjust each scale attention map to adapt to different input and task requirements; and the dynamic spatial enhancement module adds all normalized attention weights through a simple summation operation to obtain the final attention weight combination method, which is simple and effective, can make full use of spatial information of different scales, and will not introduce too much computational complexity;

[0024] (2) The MCTANet network constructed in the method of the present application has a global context sharing module designed based on a deep convolutional pyramid module in its semantic branch, which can effectively enhance the ability of the semantic branch to obtain a global receptive field. Because the input feature is only 1 / 32 resolution and deep separable convolution is used for feature fusion, the network's reasoning speed is almost unaffected;

[0025] (3) The MCTANet network constructed in the method of the present application adopts a ternary guided aggregation layer designed based on a multi-channel ternary attention mechanism in its aggregation layer, which can complement and fuse the features of detail branches and semantic branches with semantic gaps. At the same time, the multi-channel ternary attention mechanism introduced by the ternary guided aggregation layer enables the aggregation layer to capture cross-dimensional features in the three dimensions of channel, width and height, and generate attention gates of different dimensions, and obtain the final fused features by a simple method of adding and averaging, so that the network can further refine the features and pay attention to more complex and subtle information, thereby improving the segmentation accuracy;

[0026] (4) In the training process of the MCTANet network in the method of the present application, the initial learning rate is set to 5e -2 , the weight decay value is 5e -4 , and the momentum=0.9 stochastic gradient descent (SGD) method is used to update the weights and biases of each stage of the MCTANet network, so that the training effect of the MCTANet network is significantly improved compared with the BiSeNetV1 and BiSeNetV2 networks in the prior art; among them, in terms of segmentation accuracy, MCTANet achieves 77.73% mIoU, which is 5.41% mIoU higher than BiSeNetV1 and 4.2% mIoU higher than BiSeNetV2; in terms of inference speed, MCTANet achieves 108.6FPS on a single NVIDIAGeForce GTX 1080Ti GPU, which is 43.1FPS higher than BiSeNetV1, which is basically the same as the inference speed of BiSeNetV2. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the real-time semantic segmentation method of traffic scene images for autonomous driving of the present invention;

[0028] Figure 2 It is a structural schematic diagram of a dynamic spatial enhancement module (DSE Block) of a multi-channel ternary aggregation real-time semantic segmentation network MCTANet in the real-time semantic segmentation method of traffic scene images for autonomous driving of the present invention;

[0029] Figure 3It is a structural schematic diagram of a global upper and lower sharing module (GCE Block) of a multi-channel ternary aggregation real-time semantic segmentation network MCTANet in the real-time semantic segmentation method of traffic scene images for autonomous driving of the present invention;

[0030] Figure 4 It is a structural schematic diagram of the aggregation layer of the multi-channel ternary aggregation real-time semantic segmentation network MCTANet in the real-time semantic segmentation method of traffic scene images for autonomous driving of the present invention;

[0031] Figure 5 It is an image processing flow chart of a multi-channel ternary aggregation real-time semantic segmentation network MCTANet in the real-time semantic segmentation method of traffic scene images for autonomous driving of the present invention;

[0032] Figure 6 A comparison diagram of a predicted segmented image and a real segmented image achieved by the real-time semantic segmentation method for traffic scene images for autonomous driving of the present invention for a traffic scene image;

[0033] Figure 7 This is a comparison chart of the output results of processing actual traffic scene images using the real-time semantic segmentation method for traffic scene images for autonomous driving of the present invention and the real-time semantic segmentation method based on BiSeNetV2. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention in any way.

[0035] See also Figure 1 and Figure 6 ,The specific implementation steps of the real-time semantic segmentation method of traffic scene images for autonomous driving are as follows:

[0036] S1. Collect multi-category urban traffic scene images to create a semantic segmentation dataset of urban traffic scenes. Specifically, the specific implementation steps of step S1 are as follows:

[0037] Step S11, using a vehicle-mounted camera or a camera to collect images of urban traffic scenes containing different categories of features, the category features including but not limited to roads, sidewalks, buildings, walls, fences, poles, traffic lights, traffic signs, vegetation, terrain, sky, pedestrians, riders, cars, trucks, buses, trains, motorcycles, and bicycles; wherein each urban traffic scene image should contain at least two category features;

[0038] Step S12: labeling the urban traffic scene images collected in step S11 according to the classification standard of the open source semantic segmentation dataset Cityscapes;

[0039] Specifically, the labeling processing method is: finely segment different category features in the urban traffic scene image using a semantic segmentation standard tool, and name the segmented category features using a unified naming method to obtain a real segmentation image of the urban traffic scene;

[0040] Step S13, sequentially adopting a random horizontal flipping method, a random scaling method, and a random cropping method to a fixed size method, to perform data enhancement on the real segmentation image of the urban traffic scene obtained in step S12, to enhance the generalization ability, and to form a semantic segmentation dataset of the urban traffic scene;

[0041] Specifically, the random horizontal flipping method can increase data diversity. In the dataset images, some categories of objects are symmetrical in the horizontal direction, and flipping the image will not change the category label of the object. Specifically, through random horizontal flipping, another image with the same semantic information but flipped in the horizontal direction can be generated, which can enable the MCTANet network to better adapt to target objects from different perspectives; the random scaling method can randomly reduce or enlarge the data image to simulate the appearance of objects at different scales, which helps to enhance the MCTANet network's ability to recognize changes in object scale; the random cropping to fixed size method can randomly select a certain area of ​​the data image and adjust it to a fixed size, which can help the MCTANet network focus on specific areas of the image and reduce sensitivity to object positions;

[0042] Step S14: divide the processed data set into test, training and validation sets according to a reasonable ratio for the images in the urban traffic scene semantic segmentation data set; in this embodiment, the test, training and validation sets are divided according to a ratio of 2:7:1.

[0043] S2. Construct a multi-channel ternary aggregation real-time semantic segmentation network MCTANet, which consists of a detail branch, a semantic branch, and an aggregation layer;

[0044] Specifically, the network of step S2 is improved based on the BiSeNetV2 network, and its specific construction steps are:

[0045] Step S21, build detail branch (Detail Branch):

[0046] See also Figure 6 , the detail branch consists of a first 2x downsampling module, a second 2x downsampling module, a first splicing convolution module, a dynamic space enhancement module, a multiplication module and a third 2x downsampling module connected in sequence; the output end of the first 2x downsampling module is also connected to the input end of the first splicing convolution module, and the output end of the first splicing convolution module is also connected to the input end of the multiplication module; wherein,

[0047] The three 2x downsampling modules in the detail branch have the same structure, which are composed of a 3×3 convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the convolutional splicing module is composed of a Concat module and a 1×1 convolutional layer connected in sequence.

[0048] The Dynamic Spatial Enhancement Module (DSE Block) is designed based on the multi-scale dynamic spatial attention mechanism; see Figure 3 The dynamic spatial enhancement module is composed of a first processing module and a second processing module; wherein the first processing module is composed of an average pooling layer, a maximum pooling layer, a second splicing convolution module and a 1×1 convolution layer; the input ends of the average pooling layer and the maximum pooling layer are connected to the output end of the first splicing convolution module of the detail branch, and the output ends of the two are connected to the input end of the second splicing convolution module of the detail branch, and the output end of the second splicing convolution module is connected to the input end of the 1×1 convolution layer; the second processing module includes a 3×3 convolution layer, a ReLU activation function and a Sigmoid function connected in sequence The first branch is composed of a 5×5 convolution layer, a ReLU activation function and a Sigmoid function connected in sequence, a third branch is composed of a 7×7 convolution layer, a ReLU activation function and a Sigmoid function connected in sequence, an addition operation module and a multiplication operation module; the convolution layer input ends of the first branch, the second branch and the third branch are connected to the output end of the 1×1 convolution layer in the first processing module, and the output ends of the three are connected to the input end of the addition operation module to add the output results of the three branches to obtain the final attention weight Dynamic weight; the output end of the addition operation module and the output end of the first splicing convolution module are connected to the input end of the multiplication calculation module to perform multiplication calculation on the output result of the first splicing convolution module and the output result of the addition operation module.

[0049] Specifically, after the urban traffic scene image is input into the detail branch, it is first downsampled twice by 2 to obtain detail feature maps of 1 / 2 resolution and 1 / 4 resolution of the urban traffic scene image; then, the detail feature maps of 1 / 2 resolution and 1 / 4 resolution are jointly input into the first splicing convolution module for splicing processing, and then input into the dynamic spatial enhancement module, which uses convolution kernels of different sizes (3×3, 5×5 and 7×7) to capture spatial features of different scales, and uses ReLu activation function and Sigmoid function to process different spatial features in turn to obtain dynamic attention weights; then, the sum operation module adds all normalized attention weights to obtain the final attention weight; finally, the final attention weight is multiplied with the splicing result obtained by the first splicing convolution module and output. In summary, in terms of structural design, the detail branch first dynamically adjusts the importance of the attention map of each scale, and uses the dynamic spatial enhancement module to enhance the detail branch's ability to focus on spatial features. It considers that detail features of different levels and depths are very important for generating attention weights, and adopts a processing method in which the 1 / 2 resolution detail feature map is downsampled by 2 times and then concatenated with the 1 / 4 resolution detail feature map, and directly inputs it into the dynamic spatial enhancement module. Since there is no upsampling and complex convolution operations, the computational complexity of the dynamic spatial enhancement module is very low.

[0050] Step S22: constructing a semantic branch:

[0051] See also Figure 6 The semantic branch is composed of a first 2x downsampling module, a second 2x downsampling module, a third 2x downsampling module, a fourth 2x downsampling module, a fifth 2x downsampling module, a global up and down sharing module (GCS Block), a first 2x upsampling module, a summing operation module and a second 2x upsampling module connected in sequence; the output end of the fourth 2x downsampling module is also connected to the input end of the summing operation module in the semantic branch; wherein,

[0052] The five 2x downsampling modules in the semantic branch have the same structure, which are composed of a 3×3 convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the convolutional splicing module is composed of a Concat module and a 1×1 convolutional layer connected in sequence;

[0053] The global up-down sharing module (GCS Block) is designed based on the deep convolutional pyramid module DCPP; see Figure 4, the global upper and lower sharing module includes a DCPP module, a first splicing convolution module, a second 1×1 convolution layer, a BN layer, a ReLU activation function and a first summing operation module, and a first 1×1 convolution layer connected in sequence; the input end of the first 1×1 convolution layer and the input end of the DCPP module are respectively connected to the output end of the fifth 2x downsampling module, the output end of the first 1×1 convolution layer is connected to the input end of the first splicing convolution module in the global context sharing module, and the output end of the second 1×1 convolution layer is also connected to the input end of the summing operation module;

[0054] Among them, the DCPP module includes: a first branch consisting of a global average pooling layer with a pooling kernel of 1×1, a third 1×1 convolution layer, a second addition operation module and a 3×3 convolution layer connected in sequence, a second branch consisting of a global average pooling layer with a pooling kernel of 2×2, a third 1×1 convolution layer, a second addition operation module and a 3×3 convolution layer connected in sequence, a third branch consisting of a global average pooling layer with a pooling kernel of 4×4, a third 1×1 convolution layer, a second addition operation module and a 3×3 convolution layer connected in sequence, a fourth 1×1 convolution layer and a second concatenation convolution module; the input end of the global average pooling layer and the input end of the fourth 1×1 convolution layer in the three branches are both connected to the output end of the fifth 2x downsampling module; the output end of the fourth 1×1 convolution layer is respectively connected to the input end of the second addition operation module in the three branches, and the output end of the fourth 1×1 convolution layer and the output end of the 3×3 convolution layer in the three branches are all connected to the input end of the second concatenation convolution module.

[0055] The first 2x upsampling module and the second 2x upsampling module have the same structure, both of which are composed of a channel calculation module, a 1×1 convolution layer and a pixel reorganization module connected in sequence, so as to first use the convolution layer to correct the number of channels of the input feature map, and then use the pixel reorganization upsampling method to upsample the convolution feature map to obtain the upsampled feature map;

[0056] Among them, the channel calculation module is provided with an output channel number calculation formula, and its expression is:

[0057] out chan =input chan ×factor×factor,

[0058] In the formula, out chan is the number of output channels; input chan is the number of input channels; factor is the upsampling factor, which is set according to the multiple of the output feature map relative to the input feature map. If a feature map that is twice as large as the input feature map is required, then factor = 2.

[0059] Specifically, after the urban traffic scene image is input into the semantic branch, it is first downsampled five times in a row by a factor of 2 to obtain semantic feature maps of the urban traffic scene image with resolutions of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively. Then, in order to enhance the ability of the MCTANet network to obtain the global receptive field, a global context sharing module is introduced after the semantic feature map with a resolution of 1 / 32 to perform three global average pooling operations with different pooling kernels on the input semantic feature map, and output semantic feature maps with resolutions of 1×1, 2×2, and 4×4. Next, 1×1 convolution and upsampling operations are performed to adjust the pooled feature map. The global context sharing module is reorganized to the same resolution; then, based on the semantic feature maps with the same resolution obtained above, multi-scale feature fusion is performed through a 3×3 convolution layer. Since the resolution of the semantic feature maps input and output by the global context sharing module is 1 / 32, and the 3×3 convolution layer of the fused feature uses depthwise separable convolution instead of standard convolution, the global context sharing module does not increase the amount of additional calculations and has almost no effect on the reasoning speed of the MCTANet network; finally, based on the semantic feature map with a resolution of 1 / 32 output by the global context sharing module, semantic feature maps with resolutions of 1 / 16 and 1 / 8 are obtained through two upsampling processes. In order to enhance features in the training stage to assist network training, the semantic feature maps with resolutions of 1 / 4 and 1 / 32 obtained in the downsampling process, and the semantic feature maps with resolutions of 1 / 8 and 1 / 16 obtained in the subsequent upsampling process are inserted into the segmentation head to update the loss function to assist training.

[0060] Step S23: constructing an aggregation layer:

[0061] See also Figure 5 and Figure 6 The aggregation layer includes a detail branch spatial processing module, a detail branch spatial fusion module, a semantic branch context processing module, a semantic branch context fusion module and a fusion processing module; among which, the detail branch spatial processing module and the detail branch spatial fusion module are connected to the detail branches respectively, and the semantic branch context processing module and the semantic branch context fusion module are connected to the semantic branches.

[0062] The detail branch spatial processing module is composed of a 3×3 deep convolution layer, a BN layer, a 1×1 convolution layer and a Sigmoid function connected in sequence; the detail branch spatial fusion module is composed of a 3×3 convolution layer, a BN layer, a maximum pooling layer with a pooling kernel of 3×3 and an addition operation module connected in sequence; wherein, the input end of the 3×3 deep convolution layer in the detail branch spatial processing module, the input end of the 3×3 convolution layer in the detail branch spatial fusion module and the input end of the addition operation module in the detail branch spatial fusion module are respectively connected to the output end of the third 2x downsampling module of the detail branch;

[0063] The semantic branch context processing module is composed of a 3×3 deep convolution layer, a BN layer, a 1×1 convolution layer and a Sigmoid function connected in sequence; the semantic branch context fusion module is composed of a 3×3 convolution layer, a BN layer, an average pooling layer with a convolution kernel of 3×3 and an addition operation module connected in sequence; wherein, the input end of the 3×3 deep convolution layer in the semantic branch context processing module, the input end of the 3×3 convolution layer in the semantic branch context fusion module and the input end of the addition operation module in the semantic branch context fusion module are connected to the output end of the second 2x upsampling module of the semantic branch;

[0064] The fusion processing module includes a first multiplication operation module, a second multiplication operation module, and a concatenated convolution module, a ternary fusion layer (MCTriplet Fusion) and an S-Head module connected in sequence; the output end of the Sigmoid function in the detail branch space processing module and the output end of the sum operation module in the semantic branch context fusion module are respectively connected to the input end of the first multiplication operation module, so as to perform a product operation on the two output results through the first multiplication operation module; the output end of the sum operation module of the detail branch space fusion module and the output end of the Sigmoid function in the semantic branch context processing module are respectively connected to the input end of the second multiplication operation module, so as to perform a product operation on the two output results through the second multiplication operation module; the output ends of the first multiplication operation module and the second multiplication operation module are connected to the input end of the concatenated convolution module; the convolution concatenation module is composed of a Concat module and a 1×1 convolution layer connected in sequence;

[0065] The ternary fusion layer (MCTriplet Fusion) is improved based on the ternary attention mechanism. Specifically, the channel dimension processing layer in the ternary attention mechanism is replaced by: a global average pooling layer with a pooling kernel of 1×1, a first View operation module, a four-branch fully connected layer, a Concat module, a second View operation module, a Sigmoid function and a third multiplication operation module connected in sequence. The input end of the third multiplication operation module is also connected to the output end of the splicing convolution module in the fusion processing module; wherein, in the channel dimension processing layer, a feature image with B×C×H×W is converted into a feature image of B×C×1×1 through a global average pooling layer with a pooling kernel of 1×1, and H and W are removed after the first View operation module to form a feature image of B×C, and four feature maps are obtained after the four-branch fully connected layer, and then converted into a feature image of B×C×1×1 through the splicing processing of the Concat module and the inverse transformation processing of the second View operation module, and then through the Sigmoid function and product operation processing, a new feature image of B×C×H×W is output; wherein B is bitch size, that is, the image input batch, C is the channel dimension, H is the height dimension, and W is the width dimension.

[0066] Specifically, since the detail branch and the semantic branch have different depths and widths, there is a semantic gap between the features of the two branches; therefore, it is necessary to introduce a ternary guided aggregation layer to complement and fuse the features of the two branches; specifically,

[0067] First, the 1 / 8 resolution detail feature map output by the detail branch is processed by the detail branch spatial processing module to achieve channel number conversion and early feature processing, and obtains detail feature 1; the 1 / 8 resolution detail feature map output by the semantic branch is processed by the semantic context processing module to obtain semantic guided feature 1; at the same time, the 1 / 8 resolution detail feature map output by the detail branch is processed by the detail branch spatial fusion module to obtain detail feature 2; the 1 / 8 resolution detail feature map output by the semantic branch is processed by the semantic context fusion module to obtain semantic guided feature 2;

[0068] Then, based on the detail features and semantic guidance features obtained above, the detail feature 1 and the semantic guidance feature 2 are multiplied to obtain the fusion feature 1, and the detail feature 2 and the semantic guidance feature 1 are multiplied to obtain the fusion feature 2, and then the fusion feature 1 and the fusion feature 2 are spliced ​​and input into the ternary fusion layer; the ternary fusion layer uses a multi-channel ternary attention mechanism to capture cross-dimensional features, which performs pooling and 7×7 convolution processing on the fusion features in the width and height dimensions, and then generates the width attention gate and the height attention gate through the Sigmoid function; compared with the original ternary attention mechanism, the multi-channel ternary attention mechanism uses the second-generation channel attention mechanism SENetV2 to generate the channel attention gate in the channel dimension, instead of using the width and height dimension processing method;

[0069] Next, by introducing the multi-channel ternary attention mechanism, the ternary guided aggregation layer can further refine the features and pay attention to more complex and subtle information, thereby improving the segmentation accuracy of the network; then, based on the channel, width and height attention gates obtained above, the final fused features are obtained by a simple method of adding and averaging, and then the final fused features are integrated by the segmentation head S-Head according to the number of channels, so that the output dimension of the segmentation head is consistent with the number of categories of the images of the urban traffic scene dataset to generate the final prediction results.

[0070] S3, training the multi-channel ternary aggregation real-time semantic segmentation network MCTANet constructed in step S2 to accurately output the real-time semantic segmentation results of the urban traffic scene image;

[0071] The specific implementation steps of step S3 are:

[0072] Step S31: Configure the training environment:

[0073] In this embodiment, the multi-channel ternary aggregation real-time semantic segmentation network MCTANet is trained based on the Linux-Ubuntu18.04 operating system environment, using pytorch as the deep learning framework, the CUDA version is 12.2, and a single NVIDIA GeForce GTX 1080Ti GPU is used to complete the model training; the open source semantic segmentation dataset Cityscapes is used as a training sample in the training stage;

[0074] Step S32: Setting network training parameters:

[0075] 1) The initial learning rate is set to 5e -2 , the weight decay value is set to 5e -4 ; The dataset size in the training phase is cropped to 512×1024 resolution, and the dataset size in the validation phase is cropped to 1024×1024 resolution;

[0076] 2) According to the poly learning rate strategy, set the learning rate in subsequent iterations to:

[0077]

[0078] Where iter is the number of iterations in any subsequent training phase, iters max The number of iterations to find the minimum value of the loss function;

[0079] At the same time, the stochastic gradient descent (SGD) method with momentum = 0.9 is set to update the weights and biases of each stage of the MCTANet network;

[0080] 3) Dynamically set the batch size according to the real-time situation of the training platform. The batch size is a positive integer between 3 and 7.

[0081] S33. Set the network training loss function and train the MCTANet network:

[0082] See also Figure 6 , the MCTANet network is trained; during the training process, the output ends of the second 2x downsampling, the fifth 2x downsampling, the sum operation module and the second 2x upsampling module in the semantic branch are respectively connected to the input end of an S-Head module, and the output results of the output ends of each S-Head module, together with the output structure of the S-Head module connected to the ternary fusion layer, are input to the loss function to calculate the loss function value; wherein the loss function adopts the cross entropy loss (CE Loss) function; in this embodiment, the number of iterations of the warm-up learning strategy is 1K, and the number of iterations for finding the minimum value of the loss function is at least 80K.

[0083] S4. Input the urban traffic scene image into the multi-channel ternary aggregation real-time semantic segmentation network MCTANet trained in step S3 to output the real-time semantic segmentation result of the urban traffic scene image.

[0084] In order to further demonstrate the effective semantic segmentation performance of the real-time semantic segmentation method for traffic scene images for autonomous driving of the present application, and that it has significant progress over the prior art, the method of the present application is used for visualized semantic segmentation on the Cityscapes validation set; as a comparative example, the BiSeNetV2 network is used to replace the MCTANet network, and visualized semantic segmentation is performed on the same validation set, and the visualized semantic segmentation results of the two methods are compared.

[0085] like Figure 7 The figure shows a comparison of the output results of processing 12 actual traffic scene images using the real-time semantic segmentation method for traffic scene images for autonomous driving of the present invention and the real-time semantic segmentation method based on BiSeNetV2, specifically including: input image (a), BiSeNetV2 predicted image (b), MCTANet (c) and real segmentation image (d).

[0086] like Figure 7 As shown in the middle row (a), the BiSeNetV2 network does not fully segment cyclists and cannot effectively identify poles and traffic lights, while the MCTANet network proposed in the present invention effectively solves the above problems;

[0087] like Figure 7 As shown in the middle row (b), the BiSeNetV2 network does not fully segment pedestrians and cannot effectively identify distant poles and traffic lights. The MCTANet network proposed in this invention effectively solves the above problems;

[0088] like Figure 7 As shown in the middle row (c), the BiSeNetV2 network does not fully segment pedestrians and vegetation on both sides of the road. The MCTANet network proposed in the present invention effectively solves the above problem;

[0089] like Figure 7 As shown in the middle row (d), the BiSeNetV2 network does not fully segment the cyclist, pole, traffic sign, and traffic light. The MCTANet network proposed in the present invention effectively solves the above problem.

[0090] like Figure 7 As shown in the middle row (e), the BiSeNetV2 network does not fully segment the vegetation on both sides of the road, and cannot effectively identify the poles, traffic signs and traffic lights. The MCTANet network proposed in the present invention effectively solves the above problems;

[0091] like Figure 7 As shown in the middle row (f), the BiSeNetV2 network does not fully segment the cyclist and cannot effectively identify the poles and walls. The MCTANet network proposed in the present invention effectively solves the above problems;

[0092] like Figure 7 As shown in the middle row (g), the BiSeNetV2 network does not fully segment the poles and cannot effectively identify the walls and traffic signs on both sides of the road. The MCTANet network proposed in the present invention effectively solves the above problems;

[0093] like Figure 7 As shown in the middle row (h), the BiSeNetV2 network does not fully segment pedestrians, cars, trucks, vegetation and terrain, and cannot effectively identify poles and traffic signs. The MCTANet network proposed in the present invention effectively solves the above problems;

[0094] like Figure 7 As shown in the middle row (i), the BiSeNetV2 network does not fully segment vegetation and cannot effectively identify poles. The MCTANet network proposed in the present invention effectively solves the above problems;

[0095] like Figure 7 As shown in the middle row (j), the BiSeNetV2 network does not fully segment the car and the wall, and cannot effectively identify the pole and traffic sign. The MCTANet network proposed in the present invention effectively solves the above problems;

[0096] like Figure 7 As shown in the middle row (k), the BiSeNetV2 network does not fully segment pedestrians, cyclists, and short-distance traffic signs, and cannot effectively recognize poles and long-distance traffic signs. The MCTANet network proposed in this invention effectively solves the above problems;

[0097] like Figure 7 As shown in the middle row (l), the BiSeNetV2 network does not fully segment pedestrians and cyclists and cannot effectively recognize long-distance traffic signs. The MCTANet network proposed in the present invention effectively solves the above problems.

[0098] As shown in Table 1 and Table 2 (continued from Table 1), the segmentation accuracy of different categories of features in the semantic segmentation results output after real-time semantic segmentation of the same traffic scene image using the MCTANet network of the present application and the BiSeNetV1 network and BiSeNetV2 network of the prior art, as well as the comparison chart of the overall segmentation accuracy and inference speed.

[0099] Table 1:

[0100]

[0101] Table 2 (Continued from Table 1):

[0102]

[0103] From the comparison results shown in Table 1 and Table 2, it can be seen that compared with the BiSeNetV1 and BiSeNetV2 networks, the MCTANet network proposed in the present invention has achieved a significant improvement in the segmentation accuracy of the 19 categories of objects contained in the Cityscapes dataset, especially for objects such as poles, traffic lights, traffic signs, trucks, buses, trains and motorcycles. The segmentation accuracy is more obvious; it can be seen that the real-time semantic segmentation method of traffic scene images for autonomous driving in the present application is superior to the existing BiSeNetV1 and BiSeNetV2 technologies in semantic segmentation accuracy, and can maintain a faster reasoning speed.

[0104] In addition, in actual applications, the method of the present application has a preheating learning strategy iteration number of 1K, and a number of iterations for finding the minimum value of the loss function of at least 80K; compared to the BiSeNetV2 network application, the preheating learning strategy iteration number is 1K, and the number of iterations for finding the minimum value of the loss function is at least 150K. The present application only requires half the number of iterations to achieve a semantic segmentation effect that exceeds the semantic segmentation accuracy of the BiSeNetV2 network.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time semantic segmentation method for traffic scene images for autonomous driving, characterized in that: Using the urban traffic scene semantic segmentation dataset, the constructed multi-channel ternary aggregation real-time semantic segmentation network MCTANet is trained to achieve the output of real-time semantic segmentation results by inputting urban traffic scene images into the network: wherein, the multi-channel ternary aggregation real-time semantic segmentation network MCTANet is composed of a detail branch, a semantic branch and an aggregation layer; wherein, the detail branch is composed of a first 2-fold downsampling module, a second 2-fold downsampling module, a first splicing convolution module, a dynamic space enhancement module, a multiplication operation module and a third 2-fold downsampling module connected in sequence; the output end of the first 2-fold downsampling module is also connected to the first splicing convolution module, and the output end of the first splicing convolution module is also connected to the input end of the multiplication operation module; the dynamic space enhancement module adopts a multi-scale dynamic space attention mechanism to process the output result of the first splicing convolution module through dynamic attention weights; the semantic branch is composed of a first 2-fold downsampling module, a second 2-fold downsampling module, a third 2-fold downsampling module, a fourth 2-fold downsampling module, a fifth 2-fold downsampling module, a global upper and lower sharing module, a first 2-fold upsampling module, a summation operation module The output end of the fourth 2-fold downsampling module is also connected to the input end of the sum operation module in the semantic branch; the global up-down sharing module is designed based on the deep convolutional pyramid module DCPP; the aggregation layer includes a detail branch spatial processing module and a detail branch spatial fusion module connected to the detail branch, and a semantic branch context processing module and a semantic branch context fusion module connected to the semantic branch; the detail branch spatial processing module and the semantic branch context fusion module, as well as the detail branch spatial fusion module and the semantic branch context processing module are cross-multiplied and then input into the sequentially connected splicing convolution module, the ternary fusion layer and the segmentation head S-Head module; the ternary fusion layer is based on the improvement of the channel dimension processing layer of the ternary attention mechanism, and the channel dimension processing layer is composed of a global average pooling layer with a pooling kernel of 1×1, a first View operation module, a four-branch fully connected layer, a Concat module, a second View operation module, a Sigmoid function and a multiplication operation module connected in sequence, and the input end of the multiplication operation module of the ternary fusion layer is also connected to the output end of the first splicing convolution module of the aggregation layer; The dynamic space enhancement module is composed of a first processing module and a second processing module; the first processing module is composed of an average pooling layer, a maximum pooling layer, a second splicing convolution module and a 1×1 convolution layer; the input ends of the average pooling layer and the maximum pooling layer are connected to the output end of the first splicing convolution module in the detail branch, and the output ends of the two are connected to the input end of the second splicing convolution module of the detail branch, and the output end of the second splicing convolution module is connected to the input end of the 1×1 convolution layer; the second processing module includes a first processing module composed of a 3×3 convolution layer, a ReLU activation function and a Sigmoid function connected in sequence. The first branch is composed of a first branch, a second branch consisting of a 5×5 convolution layer, a ReLU activation function and a Sigmoid function connected in sequence, a third branch consisting of a 7×7 convolution layer, a ReLU activation function and a Sigmoid function connected in sequence, an addition operation module and a multiplication operation module; the convolution layer input ends of the first branch, the second branch and the third branch are connected to the output end of the 1×1 convolution layer in the first processing module, and the output ends of the three are connected to the input end of the addition operation module; the output end of the addition operation module and the output end of the first splicing convolution module in the detail branch are connected to the input end of the multiplication calculation module.

2. The method for real-time semantic segmentation of traffic scene images for autonomous driving according to claim 1, characterized in that: The method for constructing a semantic segmentation dataset of urban traffic scenes is as follows: 1) using vehicle-mounted cameras and cameras to collect urban traffic scene images containing a variety of different categories of features; 2) according to the classification standards of open source semantic segmentation datasets, the collected urban traffic scene images are labeled to obtain labeled images of urban traffic scenes; 3) random horizontal flipping method, random scaling method and random cropping to fixed size method are used in turn for data enhancement; 4) All labeled images of urban traffic scenes are constructed to form a semantic segmentation dataset of urban traffic scenes.

3. The method for real-time semantic segmentation of traffic scene images for autonomous driving according to claim 1, characterized in that: The global upper and lower sharing module includes a DCPP module, a first splicing convolution module, a second 1×1 convolution layer, a BN layer, a ReLU activation function and an addition operation module, and a first 1×1 convolution layer connected in sequence; the input end of the first 1×1 convolution layer and the input end of the DCPP module are connected to the output end of the fifth 2x downsampling module, the output end of the first 1×1 convolution layer is connected to the input end of the first splicing convolution module in the global context sharing module, and the output end of the second 1×1 convolution layer is also connected to the input end of the addition operation module.

4. The method for real-time semantic segmentation of traffic scene images for autonomous driving according to claim 3, characterized in that: The DCPP module includes: a first branch consisting of a global average pooling layer with a pooling kernel of 1×1, a third 1×1 convolution layer, a second addition operation module and a 3×3 convolution layer connected in sequence, a second branch consisting of a global average pooling layer with a pooling kernel of 2×2, a third 1×1 convolution layer, a second addition operation module and a 3×3 convolution layer connected in sequence, a third branch consisting of a global average pooling layer with a pooling kernel of 4×4, a third 1×1 convolution layer, a second addition operation module and a 3×3 convolution layer connected in sequence, a fourth 1×1 convolution layer and a second concatenation convolution module; the input end of the global average pooling layer and the input end of the fourth 1×1 convolution layer in the three branches are both connected to the output end of the fifth 2x downsampling module; the output end of the fourth 1×1 convolution layer is respectively connected to the input end of the second addition operation module in the three branches, and the output end of the fourth 1×1 convolution layer and the output end of the 3×3 convolution layer in the three branches are all connected to the input end of the second concatenation convolution module.

5. The method for real-time semantic segmentation of traffic scene images for autonomous driving according to claim 1, characterized in that: The 2x downsampling modules in the detail branch and the semantic branch are composed of a 3×3 convolutional layer, a BN layer, and a ReLU activation function connected in sequence; in the detail branch, the semantic branch, and the aggregation layer, the convolutional splicing module is composed of a Concat module and a 1×1 convolutional layer connected in sequence.

6. The method for real-time semantic segmentation of traffic scene images for autonomous driving according to claim 1, characterized in that: The 2x upsampling modules in the semantic branch are composed of a channel calculation module, a 1×1 convolution layer, and a pixel reorganization module connected in sequence; the channel calculation module is provided with an output channel number calculation formula, which is expressed as follows: out chan =input chan ×factor×factor, In the formula, out chan is the number of output channels; input chan is the number of input channels; factor is the upsampling factor, which is set according to the multiple of the output feature map relative to the input feature map.

7. The method for real-time semantic segmentation of traffic scene images for autonomous driving according to claim 1, characterized in that: In the aggregation layer, the detail branch spatial processing module is composed of a 3×3 deep convolution layer, a BN layer, a 1×1 convolution layer and a Sigmoid function connected in sequence, and the detail branch spatial fusion module is composed of a 3×3 convolution layer, a BN layer, a maximum pooling layer with a convolution kernel of 3×3 and an addition operation module connected in sequence, wherein the input end of the 3×3 deep convolution layer, the input end of the 3×3 convolution layer and the input end of the addition operation module are connected to the output end of the third 2x downsampling module of the detail branch; the semantic branch context processing module is composed of a 3×3 deep convolution layer, a BN layer, a 1×1 convolution layer and a Sigmoid function connected in sequence, and the semantic branch context fusion module is composed of a 3×3 convolution layer, a BN layer, an average pooling layer with a convolution kernel of 3×3 and an addition operation module connected in sequence, wherein the input end of the 3×3 deep convolution layer, the input end of the 3×3 convolution layer and the input end of the addition operation module are connected to the output end of the second 2x upsampling module of the semantic branch.

8. The method for real-time semantic segmentation of traffic scene images for autonomous driving according to claim 1, characterized in that: The training of the multi-channel triple aggregation real-time semantic segmentation network MCTANet is completed based on the deep learning framework; the initial learning rate is set to 5e -2 , the weight decay value is 5e -4 ; According to the poly learning rate strategy, set the learning rate in subsequent iterations to: Where iter is the number of iterations in any subsequent training phase, iters max The number of iterations to find the minimum value of the loss function; the stochastic gradient descent method with momentum = 0.9 is set to update the weights and biases of each stage of the MCTANet network; the batch size is set to a positive integer ranging from 3 to 7.

9. The method for real-time semantic segmentation of traffic scene images for autonomous driving according to claim 1, characterized in that: In step S3, the loss function of network training adopts the cross entropy loss function CE Loss.

Images