Autonomous driving method based on real-time semantic segmentation network

Abstract

The present invention discloses an automatic driving method based on a real-time semantic segmentation network, comprising obtaining a training data set; constructing a real-time semantic segmentation initial model and training to obtain a real-time semantic segmentation model; obtaining image data information around a target vehicle in real time and performing semantic segmentation using the real-time semantic segmentation model, and realizing automatic driving of the target vehicle in real time. The automatic driving method based on a real-time semantic segmentation network provided by the present invention, through the design of an innovative real-time semantic segmentation model, not only realizes automatic driving based on a real-time semantic segmentation network, but also has a small amount of calculation, a fast operation speed, a smooth segmentation process, a strong model expression ability, high reliability, good accuracy and good real-time performance.

Description

Technical Field

[0001] The present invention belongs to the field of digital image processing, and in particular relates to an automatic driving method based on a real-time semantic segmentation network. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, autonomous driving assistance technology has gradually been applied to people's production and life, bringing endless convenience to people's production and life. Therefore, ensuring the safety and reliability of the autonomous driving process has become the focus of researchers.

[0003] Semantic segmentation is to segment images generated in a specific scene, that is, to classify each pixel in the image according to pre-defined semantic categories, so as to achieve the purpose of segmenting the image. Real-time semantic segmentation is the core part of the autonomous driving process; therefore, real-time semantic segmentation is particularly important in the autonomous driving process.

[0004] At present, the performance of traditional semantic segmentation models in terms of accuracy is getting better and better. However, this type of method often requires the use of complex network structures and huge amounts of calculations to achieve good segmentation effects, which is a huge consumption of GPU resources and is not conducive to achieving model segmentation effects on mobile devices; secondly, the complex calculations of this type of method will lead to slow reasoning speeds, which are obviously not suitable for autonomous driving applications that have extremely high requirements for both accuracy and speed.

[0005] Therefore, in order to solve the application problems of traditional semantic segmentation models in the field of autonomous driving, researchers proposed the following semantic segmentation solution suitable for autonomous driving:

[0006] The first is to design a lightweight module structure: This type of solution reduces the computational complexity of the model and improves network efficiency, but the computational effect of the lightweight convolution is not as good as the original convolution, especially for small models. Due to the significant reduction in the number of parameters, the performance of the model will be seriously lost; when applied to real-time autonomous driving, it will cause dual problems of accuracy and reliability;

[0007] The second is to design a new network paradigm: In addition to the classic encoding and decoding structure, the semantic segmentation model can be divided into a dual-branch structure and a multi-branch structure; however, if this type of solution is not designed properly, the information of each layer extracted in the early stage will be lost, resulting in unsatisfactory segmentation effect; or, this type of solution may lose a lot of detailed information, which is not conducive to the extraction of boundary information; in the actual autonomous driving process, the problem of unclear semantic segmentation may occur, thus affecting the safety of autonomous driving;

[0008] The third method is to use a lightweight basic network to extract feature information: Although this type of method has been widely used, the overly large receptive field of this type of solution is not conducive to the feature extraction of other small-scale targets, nor is it conducive to the refinement of target boundaries; in actual autonomous driving applications, there may be problems with the recognition of small-scale targets, which in turn causes driving safety issues. Summary of the invention

[0009] The object of the present invention is to provide an automatic driving method based on a real-time semantic segmentation network with high reliability, good accuracy and good real-time performance.

[0010] The automatic driving method based on the real-time semantic segmentation network provided by the present invention comprises the following steps:

[0011] S1. Obtain training data set;

[0012] S2. Build the initial model of real-time semantic segmentation based on resnet18 network, pyramid pooling module, feature fusion module and boundary reinforcement module;

[0013] S3 uses the training data set obtained in step S1 to train the initial real-time semantic segmentation model constructed in step S2 to obtain a real-time semantic segmentation model;

[0014] S4 real-time acquisition of image data information around the target vehicle, and the real-time semantic segmentation model obtained in step S3 for semantic segmentation;

[0015] S5. According to the semantic segmentation result obtained in step S4, automatic driving of the target vehicle is realized in real time.

[0016] The step S2 specifically includes the following steps:

[0017] The real-time semantic segmentation initial model includes an encoder, a pyramid pooling module, a first feature fusion module, a second feature fusion module, a first boundary reinforcement module, and a second boundary reinforcement module;

[0018] The trained backbone network resnet18 is used as the encoder;

[0019] The output features of layer 4 of the backbone network resnet18 are processed by the pyramid pooling module; the output features of layer 3 of the backbone network resnet18 are processed by the first feature fusion module; the output features of the pyramid pooling module are fused with the output features of the first feature fusion module and then input into the second feature fusion module;

[0020] After fusing the output features of layer2 of the backbone network resnet18 with the output features of the first feature fusion module, the fused high-level features are obtained;

[0021] The output features of layer1 of the backbone network resnet18 and the fused high-level features are used as the input of the first boundary reinforcement module, and the output of the first boundary reinforcement module is the first enhanced primary feature map; the output features of the maximum pooling layer of the backbone network resnet18 and the fused high-level features are used as the input of the second boundary reinforcement module, and the output of the second boundary reinforcement module is the second enhanced primary feature map;

[0022] After upsampling and convolution operations are performed on the fused high-level features, they are element-wise added to the first enhanced primary feature map and the second enhanced primary feature map, and the addition result is upsampled and convolved again to obtain the final real-time semantic segmentation initial model output.

[0023] The backbone network resnet18 includes an input convolution layer, a maximum pooling layer, a layer1 layer, a layer2 layer, a layer3 layer and a layer4 layer; the input convolution layer, the maximum pooling layer, the layer1 layer, the layer2 layer, the layer3 layer and the layer4 layer are connected in series in sequence.

[0024] The pyramid pooling module specifically includes the following contents:

[0025] The input features of the pyramid pooling module are simultaneously input to the five convolution submodules: the first convolution submodule adopts The convolution of the convolution is expanded to a scale of 1, and the second convolution submodule adopts The convolution of the convolution is expanded to a scale of 3, and the third convolution submodule adopts The convolution of the convolution is expanded to a scale of 5, and the fourth convolution submodule adopts The convolution of the convolution is expanded to a scale of 7, and the fifth convolution submodule adopts The convolution is global average pooling;

[0026] The output features of the five convolutional submodules are processed by one attention enhancement submodule respectively, and the processed five output features are fused in a cascade manner;

[0027] The fused features are then used The convolution is performed, and the processed features are fused with the input features of the module by element-wise addition to obtain the output features of the pyramid pooling module.

[0028] The attention enhancement submodule specifically processes the input features of the attention enhancement submodule through global average pooling and then The processed features are processed by convolution and fused with the input features by element-wise multiplication to obtain the output features of the attention enhancement submodule.

[0029] The feature fusion module specifically includes the following contents:

[0030] The high-level features of the input feature fusion module are upsampled to restore the feature map, and then the restored feature map is used The convolution is processed to obtain the high-level processing feature map;

[0031] For the low-level features of the input feature fusion module, we use The convolution is processed to obtain the low-level processing feature map;

[0032] The obtained high-level processing feature map and the low-level processing feature map are subjected to feature fusion by element-level addition, and then the fused features are input into the channel shuffling module for processing to obtain a shuffled feature map;

[0033] The obtained shuffled feature map is processed by the content submodule and the detail submodule respectively; finally, the output features of the content submodule and the output features of the detail submodule are fused by element-wise addition to obtain the output features of the feature fusion module.

[0034] The content submodule specifically includes the following contents:

[0035] For the input features of the content submodule, we use The convolution is processed with a dilation factor of 5 to obtain the content processing sub-feature;

[0036] The content processing sub-feature is processed by average pooling and linearly activated by the sigmoid function. The processed feature is fused with the content processing sub-feature by element-wise multiplication to obtain the content fusion feature. The content fusion feature is then fused with the content processing sub-feature again by element-wise multiplication to obtain the output feature of the content sub-module.

[0037] The details submodule specifically includes the following contents:

[0038] For the input features of the detail submodule, we use The convolution is processed with a dilation factor of 3 to obtain the detail processing sub-features;

[0039] The detail processing sub-feature is processed by maximum pooling and linearly activated by sigmoid function. The processed feature is fused with the detail processing sub-feature by element-wise multiplication to obtain the detail fusion feature. The detail fusion feature is then fused with the detail processing sub-feature again by element-wise multiplication to obtain the output feature of the detail sub-module.

[0040] The first boundary reinforcement module and the second boundary reinforcement module both specifically include the following contents:

[0041] The input features of the boundary reinforcement module include high-level features and low-level features;

[0042] After upsampling the high-level features, the upsampled features are fused with the low-level features in a cascade manner, and the fused sub-features are obtained after calculating the auxiliary loss;

[0043] The fusion sub-features are adopted The convolution operation is then performed, and then the maximum value is obtained by the max operation to obtain the maximum value feature map; the maximum value feature map is processed by Laplace convolution to obtain the Laplace feature map; during the Laplace convolution processing, the Laplace operator is used as the edge detection operator;

[0044] The fused sub-features are then processed using the spatial attention module to obtain spatial attention sub-features;

[0045] The spatial attention sub-feature and the maximum feature map are fused by element-wise addition, and the Laplacian feature map is fused with the fused sub-feature by element-wise addition. Finally, the two fused features are fused again by element-wise addition to obtain the output features of the boundary enhancement module.

[0046] The spatial attention module specifically includes the following contents:

[0047] The input features of the spatial attention module are processed by the global average pooling layer and the global maximum pooling layer at the same time, and the two features are fused in a cascade manner to obtain the fused sub-features;

[0048] The fusion sub-features are passed The convolution is processed, and then linear activation is performed through the sigmoid function to obtain the activation feature;

[0049] Finally, the activation features are fused with the input features using element-wise multiplication to obtain the output features of the spatial attention module.

[0050] The training described in step S3 specifically includes the following steps:

[0051] During training:

[0052] Calculate the output loss for the output features of layer4 of the backbone network resnet18;

[0053] Calculate the output loss for the output features of layer3 of the backbone network resnet18;

[0054] Calculate auxiliary loss for the fused sub-features of the first boundary enhancement module;

[0055] For the fused sub-features of the first boundary enhancement module, the auxiliary loss is calculated.

[0056] The automatic driving method based on the real-time semantic segmentation network provided by the present invention, through the design of an innovative real-time semantic segmentation model, not only realizes the automatic driving based on the real-time semantic segmentation network, but also has small calculation amount, fast operation speed, smooth segmentation process, strong model expression ability, high reliability, good accuracy and good real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0058] Figure 2 It is a structural schematic diagram of the real-time semantic segmentation model of the present invention.

[0059] Figure 3 It is a structural schematic diagram of the pyramid pooling module of the present invention.

[0060] Figure 4 Schematic diagram of the structure of the attention enhancement submodule of the pyramid pooling module of the present invention.

[0061] Figure 5 It is a structural schematic diagram of the feature fusion module of the present invention.

[0062] Figure 6 It is a structural schematic diagram of the content submodule of the feature fusion module of the present invention.

[0063] Figure 7 It is a schematic diagram of the structure of the detail submodule of the feature fusion module of the present invention.

[0064] Figure 8 It is a structural schematic diagram of the boundary reinforcement module of the present invention.

[0065] Fig. 9 Schematic diagram of the effect of Laplace convolution of the boundary enhancement module of the present invention.

[0066] Fig.10 Schematic diagram of the structure of the spatial attention module of the boundary enhancement module of the present invention.

[0067] Fig.11It is a schematic diagram of the visualization effect of the ablation experiment of the present invention.

[0068] Fig.12 The figure is a visualization diagram of the comparative experiment of the present invention on the Cityscapes dataset.

[0069] Fig.13 It is a visualization diagram of the comparative experiment of the present invention on the CamVid dataset.

[0070] Fig.14 It is a visualization diagram of the comparative experiment of the present invention on the BDD100K dataset. DETAILED DESCRIPTION

[0071] like Figure 1 The figure shows a flow chart of the method of the present invention: the automatic driving method based on real-time semantic segmentation network provided by the present invention comprises the following steps:

[0072] S1. Obtain training data set;

[0073] S2. Build the initial model of real-time semantic segmentation based on resnet18 network, pyramid pooling module, feature fusion module and boundary reinforcement module;

[0074] The structure of the constructed real-time semantic segmentation initial model is as follows Figure 2 As shown:

[0075] The initial model of real-time semantic segmentation includes an encoder, a pyramid pooling module (labeled araspp in the figure), a first feature fusion module (labeled CSFM in the figure), a second feature fusion module (labeled CSFM in the figure), a first boundary reinforcement module (labeled LABRM in the figure), and a second boundary reinforcement module (labeled LABRM in the figure);

[0076] The trained backbone network resnet18 is used as the encoder;

[0077] The output features of layer 4 of the backbone network resnet18 are processed by the pyramid pooling module, and the pyramid pooling module does not change the size of the feature map; the output features of layer 3 of the backbone network resnet18 are processed by the first feature fusion module; the output features of the pyramid pooling module are fused with the output features of the first feature fusion module, and then input into the second feature fusion module;

[0078] After fusing the output features of layer2 of the backbone network resnet18 with the output features of the first feature fusion module, the fused high-level features are obtained;

[0079] In order to guide the low-level features to strengthen the edge representation, the output features of layer1 of the backbone network resnet18 and the fused high-level features are used as the input of the first boundary reinforcement module, and the output of the first boundary reinforcement module is the first enhanced primary feature map; the output features of the maximum pooling layer of the backbone network resnet18 and the fused high-level features are used as the input of the second boundary reinforcement module, and the output of the second boundary reinforcement module is the second enhanced primary feature map;

[0080] After upsampling and convolution operations are performed on the fused high-level features, they are then element-wise added to the first enhanced primary feature map and the second enhanced primary feature map, and the addition result is upsampled and convolved again to obtain the output of the final real-time semantic segmentation initial model;

[0081] When implementing:

[0082] (1) The backbone network resnet18 includes input convolution layer, maximum pooling layer, layer1 layer, layer2 layer, layer3 layer and layer4 layer; the input convolution layer, maximum pooling layer, layer1 layer, layer2 layer, layer3 layer and layer4 layer are connected in series in sequence;

[0083] (2) Pyramid pooling module (araspp, structure as Figure 3 Specifically include:

[0084] The input features of the pyramid pooling module are simultaneously input to the five convolution submodules: the first convolution submodule adopts The convolution of the convolution is expanded to a scale of 1, and the second convolution submodule adopts The convolution of the convolution is expanded to a scale of 3, and the third convolution submodule adopts The convolution of the convolution is expanded to a scale of 5, and the fourth convolution submodule adopts The convolution of the convolution is expanded to a scale of 7, and the fifth convolution submodule adopts The convolution of the first to fourth convolution submodules is used to obtain target features of different scales, and to ensure that the number of parameters of the module does not increase excessively, the number of channels of the feature map after the dilated convolution is only 1 / 4 of the original feature map; and the fifth convolution submodule is used to obtain the global information of all feature maps;

[0085] The output features of the five convolutional submodules are processed by an attention enhancement submodule (marked ARM in the figure) to enhance the feature expression ability, highlight important channels and suppress noise; and the processed five-way output features are fused in a cascade manner;

[0086] In order to facilitate the subsequent fusion with the input features, the fused features are then used The processed features are processed by convolution; and the processed features are fused with the input features of the module by element-level addition to make up for the feature information lost after parallel convolution, so as to obtain the output features of the pyramid pooling module;

[0087] Among them, the attention enhancement submodule (structured as Figure 4 As shown in the figure, the high-level features contain rich global information and have many channels. Therefore, the input features of the attention enhancement submodule are processed by global average pooling (at this time, the feature vector represents the global information contained in each channel) and then The convolution is processed (due to the trainability of convolution parameters, after successive iterative training, the parameters of the convolution are gradually optimized, so that more attention is focused on important channels, while suppressing unimportant channels), and the processed features are fused with the input features by element-wise multiplication, so that the original feature map can be adaptively adjusted according to the weights of each channel, and finally the output features of the attention enhancement submodule are obtained;

[0088] The following formula is used to represent the processing process of the pyramid pooling module:

[0089] represents the input feature map, represents the size of the feature map, c is the number of channels, h is the height of the feature map, w is the width of the feature map, avgool represents global average pooling, represents the convolution of kernel_size=1, represents element-wise multiplication, represents the dilated convolution with dilation=1 and kernel_size=3. cat Indicates cascade;

[0090] The processing process of the pyramid pooling module is expressed as: In the formula represents the dilated convolution with different dilation rates; the second formula represents the operation performed by the attention enhancement submodule, Xi It is the input of the attention reinforcement submodule; for Convolution; Xj is the output of the attention reinforcement submodule; Xout is the output of the pyramid pooling module;

[0091] (3) The first feature fusion module and the second feature fusion module (structure as Figure 5As shown in the figure, they specifically include: there is a semantic gap between high-level and low-level features. If only simple addition or cascading is used to fuse high-level and low-level features, certain erroneous information will be introduced, causing mutual interference between features and unsatisfactory model segmentation effect. Therefore, this application proposes a feature fusion module;

[0092] Since the size of high-level features is smaller than that of low-level features, the high-level features (marked high in the figure) of the input feature fusion module are upsampled to restore the feature map, and then the restored feature map is used convolution (making the number of high-level feature map channels equal to the number of low-level features) Figure 1 and adopt The convolution can further extract higher-level information contained in high-level features) to obtain high-level processing feature maps;

[0093] For the low-level features of the input feature fusion module (marked low in the figure), use The convolution is processed to obtain the low-level processing feature map;

[0094] The obtained high-level processing feature map and low-level processing feature map are subjected to feature fusion by element-level addition (the purpose is to ensure that the number of channels after fusion is small and reduce the number of parameters for subsequent calculations). The fused features are then input into the channel shuffle module (marked shuffle in the figure) for processing to strengthen the information interaction between feature map channels and prevent overfitting, and finally a shuffled feature map is obtained;

[0095] The obtained shuffled feature map is processed by the content submodule (labeled ContextR in the figure) and the detail submodule (labeled DetailR in the figure); finally, the output features of the content submodule and the output features of the detail submodule are fused by element-wise addition to obtain the output features of the feature fusion module;

[0096] Among them, the content submodule (structured as Figure 6 ) is used to strengthen the contextual representation of the fused features, including:

[0097] For the input features of the content submodule, we use The convolution is processed with a dilation factor of 5 to obtain the content processing sub-feature; this convolution has a larger receptive field and can extract more context information;

[0098] The content processing sub-feature is processed by average pooling and linearly activated by the sigmoid function. The processed feature (the feature map vector represents the weight of each channel in all channels, which is conducive to the redistribution of model attention) is fused with the content processing sub-feature by element-wise multiplication to obtain the content fusion feature. The content fusion feature is then fused with the content processing sub-feature again by element-wise multiplication (to highlight the key channels, suppress the non-key channels, and prevent the loss of the original feature information), thereby obtaining the output feature of the content sub-module;

[0099] Details submodule (structured as Figure 7 Specifically include:

[0100] For the input features of the detail submodule, we use The convolution is processed with a dilation factor of 3 (a smaller receptive field is more conducive to extracting accurate detail information) to obtain the detail processing sub-feature;

[0101] The detail processing sub-feature is processed by maximum pooling (used to extract the discriminative information of the feature map, which is conducive to the extraction of boundary detail information) and linearly activated by the sigmoid function. The processed features are fused with the detail processing sub-features by element-wise multiplication to obtain the detail fusion feature. The detail fusion feature is then fused with the detail processing sub-feature by element-wise multiplication again (to give full play to each other's advantages and complement each other), thereby obtaining the output feature of the detail sub-module.

[0102] The following formula is used to represent the processing process of the feature fusion module:

[0103] low represents the low-level feature map, high represents the high-level feature map, up represents upsampling, represents a normal convolution with kernel_size=3, shuffle represents channel shuffle (channel confusion), avgpool represents global average pooling, maxpool represents global maximum pooling;

[0104] The processing process of the feature fusion module is expressed as: In the formula express The convolution of represents dilated convolution, It is sigmoid linear activation;

[0105] (4) The first boundary reinforcement module and the second boundary reinforcement module each specifically include:

[0106] When the extracted semantic information is rich enough, it is beneficial for the model to segment large targets; however, if the boundary detail information is missing, it is not conducive to the classification of target boundary pixels, resulting in blurred boundaries and low segmentation performance; the primary feature map has a small receptive field, and most of the extracted features are small target object features, which also contain certain noise. If the primary features are directly fused with the high-level features, noise will be introduced, resulting in classification errors; therefore, this application proposes a boundary enhancement module (structure such as Figure 8 As shown), used to strengthen primary features;

[0107] The input features of the boundary reinforcement module include high-level features and low-level features;

[0108] Since the high-level features are not the same size as the low-level features, the high-level features are upsampled, and then the upsampled features are fused with the low-level features in a cascade manner, and the fused sub-features are obtained after the auxiliary loss is calculated (marked Xcat in the figure);

[0109] The fusion sub-features are adopted The convolution is processed (changing its number of channels to the number of categories of the final segmentation of the model, for example, Cityscapes contains 19 categories, then the returned features Figure X c is 19×h×w in size, and then the max operation is used to take the maximum value to obtain the maximum feature map (labeled Xmax in the figure, and the pixel value at each position of Xmax indicates the category to which the pixel belongs); the maximum feature map is processed by Laplace convolution to obtain the Laplace feature map (labeled Xlap in the figure); during the Laplace convolution processing, the Laplace operator is used as the edge detection operator;

[0110] Among them, since Xmax is the maximum value of Xc along the channel, the point with a larger value in Xmax has a higher probability of belonging to a certain semantic class. Conversely, the point with a smaller value in Xmax has a lower probability of belonging to a certain class, that is, the smaller the value, the higher the probability of belonging to the edge of the target. In order to improve the spatial representation of the edge of the segmented target, the boundary enhancement module uses Laplace convolution on Xmax. The Laplace operator is a classic edge detection operator, such as Fig. 9 As shown ( Fig. 9 (a) is the original image, Fig. 9 (b) is the feature map after Laplace convolution. Compared with the original image, the edge of the feature map after Laplace convolution is obvious.

[0111] In order to further enhance the spatial representation of the low-level feature map, the fused sub-features are processed by the spatial attention module to obtain spatial attention sub-features;

[0112] The spatial attention sub-feature and the maximum feature map are fused by element-wise addition, and the Laplacian feature map is fused by element-wise addition with the fusion sub-feature. Finally, the two fused features are fused again by element-wise addition to obtain the output feature of the boundary reinforcement module.

[0113] The boundary enhancement module obtains the edge enhancement features Figure X The advantages of cl also include strengthening the spatial position representation of the features Figure X The spatial information of sl fully utilizes the advantages of low-level features;

[0114] Among them, the spatial attention module (labeled SAM in the figure, with the structure as Fig.10 ), specifically including:

[0115] The input features of the spatial attention module are processed by the global average pooling layer and the global maximum pooling layer at the same time to obtain the global information of the feature map; and the two features are fused in a cascade manner to obtain the fused sub-features;

[0116] The fusion sub-features are passed The convolution is processed, and then linear activation is performed through the sigmoid function to obtain the activation feature;

[0117] Finally, the activation features are fused with the input features by element-wise multiplication to obtain the output features of the spatial attention module. Weights are applied to each position of the input feature map to strengthen the representation of important positions and suppress the representation of redundant positions.

[0118] The following formula is used to represent the processing process of the boundary reinforcement module: In the formula represents auxiliary loss calculation, for The convolution of is the Laplace convolution, Processing function for the spatial attention module;

[0119] S3 uses the training data set obtained in step S1 to train the initial real-time semantic segmentation model constructed in step S2 to obtain a real-time semantic segmentation model;

[0120] During specific training, the output loss is calculated for the output features of layer4 of the backbone network resnet18 (labeled aux1 in the figure);

[0121] Calculate the output loss for the output features of layer3 of the backbone network resnet18 (labeled aux0 in the figure);

[0122] For the fused sub-features of the first boundary enhancement module, calculate the auxiliary loss (labeled aux2 in the figure);

[0123] For the fused sub-features of the second boundary enhancement module, calculate the auxiliary loss (labeled aux3 in the figure);

[0124] S4 real-time acquisition of image data information around the target vehicle, and the real-time semantic segmentation model obtained in step S3 for semantic segmentation;

[0125] S5. According to the semantic segmentation result obtained in step S4, automatic driving of the target vehicle is realized in real time.

[0126] The following is a description of the effects of the method of the present application in combination with some embodiments (since the performance of the autonomous driving method of the present application depends entirely on the semantic segmentation grid model constructed by the present application, the following mainly compares the effects of the semantic segmentation network model used in the method of the present application):

[0127] Dataset:

[0128] The Cityscapes dataset is an image segmentation dataset for unmanned driving environments. It contains street scenes in 50 cities around the world in different environments, backgrounds, and seasons. The dataset provides 5,000 finely annotated images, 2,000 roughly annotated images, and 30 categories of annotated objects. This training only focuses on 19 of them. The dataset is divided into training set, validation set, and test set. Each set includes 2,972, 500, and 1,000 images, respectively, and each image is an RGB three-channel image with a resolution of 1024×2048.

[0129] The CamVid dataset is the earliest semantic segmentation dataset used in the field of autonomous driving. It includes 701 images and 32 types of labeled objects. This training only focuses on 11 of them. The set is divided into training set, validation set and test set. Each set includes 367, 101 and 233 images respectively. Each image has RGB three channels and a resolution of 960×720.

[0130] The BDD100K dataset covers real driving scenarios, is diverse, and contains information about weather conditions, daytime, and scene locations. The data has 8,000 images annotated for semantic segmentation, of which 7,000 images are used for training and 1,000 images are used for validation. The resolution of each image is 720×1280, and 19 annotated classes are used for semantic segmentation.

[0131] Experimental details:

[0132] Evaluation Metrics

[0133] Use mean intersection over union (mIoU) as the accuracy measure;

[0134] Use frames per second (FPS) as a speed metric;

[0135] Use Params to represent each network parameter;

[0136] GFLOPS is used to represent the computational effort of each network;

[0137] Experimental setup

[0138] The preprocessing of the input image during the training process includes: random cropping, horizontal rotation, and random changes in the brightness, hue, and contrast of the image. The batch size (batchsize) is set to 4, and the stochastic gradient descent (SGD) with momentum is used for model training. The following formula is the learning rate reduction formula: In the formula is the initial learning rate, set to 0.01; iter is the current number of iterations; max iter The maximum number of iterations is set to 80000; the momentum power is set to the default value of 0.9, and the decay coefficient is 0.00005;

[0139] This application is implemented using the deep learning framework pytorch-1.3, using a Tesla-T4 GPU and 16 CPU cores for model training and testing. The weighted cross entropy loss function is used for main loss and auxiliary loss calculation. The specific formula is as follows: In the formula is the loss of the entire network; aux0~aux3 are Figure 2 The loss in is the weight coefficient set and is set to 3;

[0140] Ablation experiment:

[0141] For araspp module:

[0142] In order to verify the effectiveness of the araspp module in this application, this application compares the performance of the original ASPP module with the araspp module proposed in this application. The experimental results are shown in Table 1:

[0143] Table 1 Schematic diagram of araspp module comparison experiment

[0144]

[0145] After adding the arASPP module, the network parameters only increased by 3M, the segmentation speed decreased by 62 frames, but the average intersection-over-union ratio increased by 10%. Compared with the ASPP module, araspp adds parallel branches, and the number of feature map channels output by each branch is only 1 / 4 of the number of feature map channels in ASPP. Therefore, araspp has relatively small parameters and calculations, and the segmentation speed is 6 frames faster than the network with ASPP. Due to araspp's ability to capture multi-scale target features, an attention enhancement submodule ARM is added behind each parallel branch to enhance feature representation, so the module can further extract feature information useful to the model, and the segmentation accuracy is also higher.

[0146] For the feature fusion module CSFM:

[0147] The data of the ablation experiment of the feature fusion module CSFM is shown in Table 2:

[0148] Table 2 Schematic diagram of CSFM module comparison experiment

[0149]

[0150] According to Table 2, it can be seen that the average intersection-over-union ratio achieved by the encoding network resnet18 alone is 61.31%. After using the multi-scale pooling module araspp, the segmentation accuracy of the model is improved to 71.47%. Since the CSFM module in this application is composed of two modules, the content submodule ContextR and the detail submodule DetailR, in order to verify the effectiveness of these two submodules, this application conducts ablation experiments on both modules separately. Compared with the network that does not use the DetailR module, the network that only uses the DetailR module has an increase of 2.7M parameters, an increase of 24G calculations, and a decrease of 5 frames in segmentation speed, but its re-extraction of low-level detail information helps the network improve the segmentation accuracy by 0.62%. The network using the ContextR module also has a certain improvement in the number of parameters and calculations. Because it uses an expansion convolution with an expansion factor of 5, the segmentation speed is reduced more, but it also strengthens the network's further extraction of global information, and the network segmentation accuracy is improved to 72.54%. The CSFM module is assembled by DetailR and ContextR. The CSFM module absorbs the advantages of both DetailR and ContextR modules, and the average intersection-over-union ratio is improved to 72.92%. The model inference speed is still 102.9 frames.

[0151] For the boundary reinforcement module LABRM:

[0152] In order to verify the effectiveness of the boundary reinforcement module LABRM module in this application, an ablation experiment is conducted on the LABRM module and its submodules. The specific experimental data are shown in Table 3:

[0153] Table 3 Schematic diagram of LABRM module comparison experiment

[0154]

[0155] From Table 3, we can see that the segmentation accuracy of the network composed of res18+araspp reaches 71.47%. Only the LAP branch in the LABRM module is used to fuse the lowest two layers of features. Due to its powerful edge feature extraction capability, the model segmentation accuracy is improved to 73.12%, but the use of Laplacian convolution also reduces the network inference speed to 105.9 frames. Only the spatial attention mechanism is used to fuse the lowest two layers of features, and the network segmentation accuracy is 73.44%. There are fewer convolution operations in the spatial attention mechanism, so the network processing frames per second are faster than the network segmentation speed of the Res18+araspp+LAP combination. The LABRM module is obtained by fusing the Laplacian convolution and the spatial attention submodule SAM. This module has both the edge extraction capability of the Laplacian operator and the position reweighting redistribution capability of the spatial attention mechanism. The segmentation accuracy reaches 73.78%, and the segmentation speed is still 104.4.

[0156] Decoder used in this application:

[0157] In order to verify the impact of various decoding methods on network performance, this application summarizes the experimental data shown in Table 4.

[0158] Table 4 Decoding structure comparison experimental data table

[0159]

[0160] From Table 4, we can see that the encoding network of Res18+arASPP achieves a segmentation accuracy of 71.47%. The segmentation accuracy obtained by decoding using element-level addition and cascade is not as good as the encoding network itself, which shows that these two decoding methods are not suitable for the network. However, the decoding structure in this application can still improve the segmentation accuracy by 6.2% based on res18+araspp.

[0161] In order to better compare the performance impact of each module, this application summarizes the ablation experiment data as shown in Table 5:

[0162] Table 5. Summary of ablation experiment comparison data

[0163]

[0164] The araspp module improves segmentation accuracy by 10% due to its powerful ability to capture multi-scale features, but the use of a large number of convolutions in the five branches also reduces the network inference speed by 62 frames. The network segmentation accuracy of the network that only uses the CSFM module for decoding is 1.5% higher than that of res18+araspp, and the streaming speed is reduced by 14 frames; the network segmentation accuracy of the network that only uses the LABRM module to fuse low-level features is also improved by 2.3%, and the inference speed is reduced by 12 frames. After fusing CSFM and LABRM as the decoding structure, the segmentation accuracy of the network is improved to 77.63%, and the inference speed is still 101.9 frames. In order to compare the segmentation accuracy of each module more objectively, the ablation experiment results are visualized, and the effects are shown as follows. Fig.11 As shown (the FBRNet marked in the figure is the method of this application).

[0165] Comparison of baseline auxiliary loss experiments:

[0166] During the network training process, in order to obtain the optimal auxiliary loss coefficient, this application attempts to set the auxiliary loss at different positions of the encoder, as shown in Table 6:

[0167] Table 6 Comparison data of baseline auxiliary loss experiment

[0168]

[0169] It can be seen from Table 6 that the segmentation accuracy obtained by setting the auxiliary loss only at layer4 is 72.02%, and the segmentation accuracy obtained by setting the auxiliary loss at both layer4 and layer3 is 72.92%, which is 0.9% higher than the segmentation accuracy when the auxiliary loss is set only at layer4. This application also attempts to set the auxiliary loss at layer4, layer3 and layer2 at the same time, but the average intersection-over-union ratio decreases by 0.5%. Therefore, this application only sets the auxiliary loss at layer4 and layer3 to obtain the optimal performance of the model.

[0170] Auxiliary loss coefficient experiment:

[0171] This application sets auxiliary loss at LABRM. In order to obtain the best experimental effect, this application explores the influence of different auxiliary loss parameters on the experimental effect. The data results are shown in Table 7:

[0172] Table 7 LABRM auxiliary loss coefficient experimental comparison data table

[0173]

[0174] It can be seen from Table 7 that when the auxiliary loss coefficients in both LABRM modules are set to 3, the optimal segmentation accuracy can be achieved.

[0175] Comparative experiment:

[0176] To verify the effectiveness of the proposed method, experiments were conducted on three datasets:

[0177] Dataset 1: Cityscapes dataset

[0178] Table 8 lists the segmentation performance of excellent models on the Cityscapes dataset in recent years:

[0179] Table 8 Comparison of network performance on the Cityscapes dataset

[0180]

[0181] Since some networks are not open source, the corresponding network properties cannot be obtained, and are represented by — in Table 8. The number of frames processed per second 1 represents the image segmentation speed in the original paper of the network, and the number of frames processed per second 2 represents the segmentation speed tested using the same experimental environment as this application;

[0182] Compared with BiSeNetV1 using the same backbone network, this application has superior segmentation effects on both the validation set and the test set, and the segmentation speed is nearly 20 frames faster than BiSeNetV1. Compared with BiSeNetV2, the segmentation accuracy of this application on the Cityscapes test set is 2.4% higher, and the segmentation speed is fast. Compared with other classic algorithms ICNet, ERFNet, DFANet, and FasterSeg, this application has the highest segmentation accuracy and the fastest inference speed under the same hardware conditions.

[0183] In order to more intuitively compare the segmentation performance of each network, Fig.12 (The FBRNet marked in the figure is the method of this application) shows the visualization effect of each network on the Cityscapes dataset; Fig.12 It can also be seen that the actual effect of the present application method is better.

[0184] Dataset 2: CamVid Dataset

[0185] Table 9 shows the segmentation performance of each network on the CamVid dataset:

[0186] Table 9. Comparison of network performance on the CamVid dataset

[0187]

[0188] This application achieved a segmentation accuracy of 72.4% on the CamVid dataset, and the number of frames processed per second reached 89.79. Compared with all the open source networks listed, this application has the best effect in terms of segmentation accuracy and segmentation speed. Fig.13 (The FBRNet marked in the figure is the method of this application) shows the visual segmentation effect of each network on the CamVid dataset; Fig.13 It can also be seen that the actual effect of the present application method is better.

[0189] Dataset 3: BDD100K dataset

[0190] Table 10 shows the segmentation performance of each network on the BDD100K dataset:

[0191] Table 10 Network performance comparison data on the BDD100k dataset

[0192]

[0193] The DRN series network achieved a high segmentation accuracy on this dataset, but due to its complex network structure, its reasoning speed was extremely slow; while this application was able to achieve the same accuracy with a reasoning speed of 100.75 frames. Compared with other networks, this application has the best segmentation accuracy and speed. Fig.14 (The FBRNet marked in the figure is the method of this application) shows the visual segmentation effect of each network on the BDD100K dataset; Fig.14 It can also be seen that the actual effect of the present application method is better.

[0194] Therefore, in summary, this application proposes a new real-time semantic segmentation network based on feature fusion and boundary refinement; the pyramid pooling module proposed in this application can be used to enhance the model's ability to extract multi-scale features, and compared with the original ASPP, the operation speed is faster and the accuracy is higher; the feature fusion module proposed in this application further extracts feature information from the perspectives of context and details, and better integrates high-level and low-level features; the boundary reinforcement module proposed in this application is based on the efficient edge detection Laplacian operator and spatial attention mechanism, which enhances the model's ability to extract edge features. Through ablation experiments, this application proves the effectiveness of these modules; comparative experiments were conducted in three data sets: Cityscapes, CamVid, and BDD100k, and the results showed that the method of this application achieved a good balance between segmentation accuracy and inference speed; therefore, the semantic segmentation network proposed in this application is particularly suitable for today's autonomous driving scenarios, and the autonomous driving method provided by this application has higher reliability, better accuracy, and better real-time performance.

Claims

1. An autonomous driving method based on a real-time semantic segmentation network, comprising the following steps: S1. Obtain training data set; S2. Build the initial model of real-time semantic segmentation based on the resnet18 network, pyramid pooling module, feature fusion module and boundary reinforcement module; The pyramid pooling module specifically includes the following contents: The input features of the pyramid pooling module are simultaneously input to the five-way convolution submodule: The output features of the five convolutional submodules are processed by one attention enhancement submodule respectively, and the processed five output features are fused in a cascade manner; The fused features are processed again using 1×1 convolution, and the processed features are fused with the input features of the module using element-wise addition to obtain the output features of the pyramid pooling module; The attention enhancement submodule specifically processes the input features of the attention enhancement submodule through global average pooling, then processes them through 1×1 convolution, and fuses the processed features with the input features using element-wise multiplication to obtain the output features of the attention enhancement submodule. The boundary reinforcement module includes a first boundary reinforcement module and a second boundary reinforcement module; the first boundary reinforcement module and the second boundary reinforcement module both specifically include the following contents: The input features of the boundary reinforcement module include high-level features and low-level features; After upsampling the high-level features, the upsampled features are fused with the low-level features in a cascade manner, and the fused sub-features are obtained after calculating the auxiliary loss; The fused sub-features are processed by 1×1 convolution, and then the max operation is used to obtain the maximum value to obtain the maximum feature map; The maximum feature map is processed by Laplace convolution to obtain a Laplace feature map; during the Laplace convolution processing, the Laplace operator is used as an edge detection operator; The fused sub-features are then processed using the spatial attention module to obtain spatial attention sub-features; The spatial attention sub-feature and the maximum feature map are fused by element-wise addition, and the Laplacian feature map is fused by element-wise addition with the fusion sub-feature. Finally, the two fused features are fused again by element-wise addition to obtain the output feature of the boundary reinforcement module. S3. Using the training data set obtained in step S1, the real-time semantic segmentation initial model constructed in step S2 is trained to obtain a real-time semantic segmentation model; S4. Acquire image data information around the target vehicle in real time, and perform semantic segmentation using the real-time semantic segmentation model obtained in step S3; S5. According to the semantic segmentation result obtained in step S4, automatic driving of the target vehicle is realized in real time.

2. The automatic driving method based on real-time semantic segmentation network according to claim 1, characterized in that The step S2 specifically includes the following steps: The real-time semantic segmentation initial model includes an encoder, a pyramid pooling module, a first feature fusion module, a second feature fusion module, a first boundary reinforcement module, and a second boundary reinforcement module; The trained backbone network resnet18 is used as the encoder; The output features of layer 4 of the backbone network resnet18 are processed by the pyramid pooling module; the output features of layer 3 of the backbone network resnet18 are processed by the first feature fusion module; the output features of the pyramid pooling module are fused with the output features of the first feature fusion module and then input into the second feature fusion module; After fusing the output features of layer2 of the backbone network resnet18 with the output features of the first feature fusion module, the fused high-level features are obtained; The output features of layer1 of the backbone network resnet18 and the fused high-level features are used as the input of the first boundary reinforcement module, and the output of the first boundary reinforcement module is the first enhanced primary feature map; the output features of the maximum pooling layer of the backbone network resnet18 and the fused high-level features are used as the input of the second boundary reinforcement module, and the output of the second boundary reinforcement module is the second enhanced primary feature map; After upsampling and convolution operations are performed on the fused high-level features, they are element-wise added to the first enhanced primary feature map and the second enhanced primary feature map, and the addition result is upsampled and convolved again to obtain the output of the final real-time semantic segmentation initial model.

3. The automatic driving method based on real-time semantic segmentation network according to claim 2, characterized in that The backbone network resnet18 includes an input convolution layer, a maximum pooling layer, a layer1 layer, a layer2 layer, a layer3 layer and a layer4 layer; the input convolution layer, the maximum pooling layer, the layer1 layer, the layer2 layer, the layer3 layer and the layer4 layer are connected in series in sequence.

4. The automatic driving method based on real-time semantic segmentation network according to claim 3 is characterized in that The pyramid pooling module specifically includes the following contents: The input features of the pyramid pooling module are simultaneously input to the five convolution submodules: the first convolution submodule uses 1×1 convolution for convolution expansion of scale 1, the second convolution submodule uses 3×3 convolution for convolution expansion of scale 3, the third convolution submodule uses 3×3 convolution for convolution expansion of scale 5, the fourth convolution submodule uses 3×3 convolution for convolution expansion of scale 7, and the fifth convolution submodule uses 1×1 convolution for global average pooling; The output features of the five convolutional submodules are processed by one attention enhancement submodule respectively, and the processed five output features are fused in a cascade manner; The fused features are processed again using 1×1 convolution, and the processed features are fused with the input features of the module using element-wise addition to obtain the output features of the pyramid pooling module.

5. The automatic driving method based on real-time semantic segmentation network according to claim 4 is characterized in that The feature fusion module specifically includes the following contents: The high-level features of the input feature fusion module are upsampled to restore the feature map, and then the restored feature map is processed by 3×3 convolution to obtain the high-level processed feature map; The low-level features of the input feature fusion module are processed using 3×3 convolution to obtain the low-level processing feature map; The obtained high-level processing feature map and the low-level processing feature map are subjected to feature fusion by element-level addition, and then the fused features are input into the channel shuffling module for processing to obtain a shuffled feature map; The obtained shuffled feature map is processed by the content submodule and the detail submodule respectively; finally, the output features of the content submodule and the output features of the detail submodule are fused by element-wise addition to obtain the output features of the feature fusion module.

6. The automatic driving method based on real-time semantic segmentation network according to claim 5, characterized in that The content submodule specifically includes the following contents: The input features of the content submodule are processed by 3×3 convolution with a dilation factor of 5 to obtain the content processing subfeatures; The content processing sub-feature is processed by average pooling and linearly activated by a sigmoid function. The processed feature is fused with the content processing sub-feature by element-wise multiplication to obtain a content fusion feature. The content fusion feature and the content processing sub-feature are fused again using element-wise multiplication to obtain the output feature of the content sub-module; The details submodule specifically includes the following contents: The input features of the detail submodule are processed by 3×3 convolution with a dilation factor of 3 to obtain the detail processing sub-features; After the detail processing sub-feature is processed by maximum pooling and linearly activated by sigmoid function, the processed feature is fused with the detail processing sub-feature by element-wise multiplication to obtain the detail fusion feature; The detail fusion feature and the detail processing sub-feature are fused again using element-wise multiplication to obtain the output feature of the detail sub-module.

7. The automatic driving method based on real-time semantic segmentation network according to claim 6, characterized in that The spatial attention module specifically includes the following contents: The input features of the spatial attention module are processed by the global average pooling layer and the global maximum pooling layer at the same time, and the two features are fused in a cascade manner to obtain the fused sub-features; The fused sub-features are processed by 1×1 convolution, and then linearly activated by the sigmoid function to obtain the activated features; Finally, the activation features are fused with the input features using element-wise multiplication to obtain the output features of the spatial attention module.

8. The automatic driving method based on real-time semantic segmentation network according to claim 7, characterized in that The training described in step S3 specifically includes the following steps: During training: Calculate the output loss for the output features of layer4 of the backbone network resnet18; Calculate the output loss for the output features of layer3 of the backbone network resnet18; Calculate auxiliary loss for the fused sub-features of the first boundary enhancement module; For the fused sub-features of the first boundary enhancement module, the auxiliary loss is calculated.

Images