An automatic driving multi-task visual perception method

Abstract

The application relates to an automatic driving multitask visual perception method and belongs to the field of automatic driving. A method of simultaneously utilizing target detection and semantic segmentation is adopted in a hard constraint type multitask network to comprehensively perceive an image; a single-stage target detection network yolov5 is used as a backbone network and a feature pyramid network is used as an encoder of the network to realize extraction and fusion of features of the image. A target detection decoder and a semantic segmentation decoder are used to respectively realize target detection of pedestrians and vehicles and semantic segmentation tasks of lane lines and drivable areas; convolution and feature map splicing are used to replace addition in a traditional residual module to fuse position information in a shared network layer, loss of target edge positioning information is reduced, and thus the problem of edge detection blur between lane lines and drivable areas is improved. Hidden information between different detection tasks is mined by using task learning, so that the overall detection precision of the model is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of automatic driving, and relates to an automatic driving multi-task visual perception method. BACKGROUND

[0002] The components of an automatic driving vehicle mainly include an environment perception system, a path planning system and a decision control system. Visual perception is an indispensable part of the perception system. Compared with other sensors, a visual camera sensor is inexpensive, can accurately obtain road ground indication lines, traffic signs, front obstacles and the like by being installed at the front of a vehicle, and can realize road multi-target detection and drivable area segmentation by using an image processing algorithm, thereby effectively improving the driving safety of an intelligent vehicle, so that the application of this kind of technology has a broad development prospect and has very important application value for the research of intelligent vehicles. Visual perception technology is a key technology of an automatic driving vehicle, and a good visual perception algorithm can help an automatic driving vehicle make reasonable and correct decisions when facing complex road conditions. Target detection and semantic segmentation are two most basic tasks in visual perception. In recent years, with the continuous development of deep learning technology, visual perception technology has made great breakthroughs in target detection and semantic segmentation.

[0003] In the field of target detection, target detection is mainly divided into single-stage detection algorithms and two-stage target detection algorithms. The two-stage target detection algorithm divides the target detection task into two steps of operation, the first step extracts the target candidate box of the target to be detected from the image, and the second step classifies and performs boundary box regression on the selected candidate box. This kind of algorithm has high detection accuracy, but the cumbersome steps can easily lead to low detection speed, poor real-time performance and the like, and the main representative algorithm is R-CNN and its improved algorithm Faster R-CNN. The single-stage target detection algorithm represented by YOLO (You Only See Once) can simultaneously realize boundary box regression and target classification, can be trained end-to-end, has high detection speed, and has a satisfactory detection accuracy due to the ingenious network structure design. However, these algorithms are all detection methods based on rectangular detection boxes, and cannot find suitable rectangular boxes to wrap the targets to be detected when facing lane lines and drivable areas which are irregularly shaped targets.

[0004] In the field of image semantic segmentation, the essence of image semantic segmentation is to classify the target of each pixel in the image and determine the category of each pixel point, so as to divide the region. This region division method is very effective for detecting lane lines and drivable areas which are irregularly shaped, unevenly distributed and large-area targets. The representative algorithms of semantic segmentation are U-Net, PSPNet and the like.

[0005] The most important task of the visual perception part of the automatic driving system is to detect and identify pedestrians, vehicles, traffic lights, traffic signs, lane lines and drivable areas. The target detection algorithm can be used to detect pedestrians, vehicles and traffic lights, and the semantic segmentation algorithm can be used to detect lane lines and drivable areas. However, this algorithm deployment scheme of "one task corresponding to one algorithm" not only occupies a large amount of computing resources, but also ignores the hidden correlation between the detection tasks. SUMMARY

[0006] Therefore, the purpose of the present application is to provide an automatic driving multi-task visual perception method.

[0007] To achieve the above purpose, the present application provides the following technical scheme:

[0008] An automatic driving multi-task visual perception method, comprising the following steps:

[0009] S1: The image is preprocessed by a Resize module before being sent to the detection network, and then the preprocessed image set is sent to the detection network;

[0010] S2: The image sent to the network is processed, the feature is extracted through the backbone network, and the feature is fused through the neck network;

[0011] S3: The fused features are input into the target detection decoder and the semantic segmentation decoder respectively to realize target detection of the obstacles of vehicles and pedestrians and semantic segmentation of the drivable area and the lane line;

[0012] S4: An improved residual connection module is designed in the semantic segmentation head, the convolution and feature splicing method is used instead of the addition of the traditional residual module to reduce the loss of positioning information in the bottom layer feature, and a decoder capable of performing semantic segmentation on the lane line and the drivable area is designed based on the improved residual module, which solves the problem of fuzzy segmentation edge and improves the detection accuracy of semantic segmentation.

[0013] Optionally, in S4, the decoder capable of performing semantic segmentation on the lane line and the drivable area is: firstly, a pyramid pooling module PPM is used to aggregate the context information of different regions in the feature map processed in S3 to improve the ability of the network to obtain global information; then the processed feature map is superimposed with the feature map in the backbone network in S1, and finally a 1*1 convolution is used to fuse the two parts of features; in order to restore the position information of the target in the image without increasing the calculation amount, a nearest neighbor difference method NI is used to upsample to restore the resolution of the image, thereby realizing the semantic segmentation of the lane line and the drivable area.

[0014] Optionally, in the S4, the semantic segmentation decoder adopts an improved deep residual connection module, and feature fusion is realized by using splicing and convolution instead of addition of a traditional residual module.

[0015] Optionally, the improved residual connection module is specifically as follows:

[0016] The conventional residual module is as shown in formula (1):

[0017]

[0018] Wherein Z i represents a feature map after feature fusion, X i and Y i respectively represent two feature maps needing to be fused, c1 and c2 respectively represent channel numbers of the feature maps X i and Y i , and here c1 and c2 are equal to complete feature fusion in the form of addition; the improved residual module is as shown in formula (2):

[0019]

[0020] T i is a feature map after splicing, the relationship among T i , X i , Y i is as shown in formula (3) and (4):

[0021]

[0022]

[0023] Here, c1 and c2 do not need to be equal, wherein the reserved is a bottom layer feature map containing rich positioning information, and in subsequent resolution recovery by upsampling, this part of information is used for accurate segmentation of different regions; the spliced feature map is subjected to feature fusion and restored to the original dimension by using 1*1 convolution.

[0024] The present application has the following advantages:

[0025] (1) Compared with deployment of a one-to-one single task detection scheme, the use of a multi-task learning network not only saves a large amount of computing resources and computing time, but also mines relevant hidden information among detection tasks and improves the performance of each detection task;

[0026] (2) The decoder designed based on the improved residual module solves the problem of fuzzy segmentation edges and improves the detection accuracy of semantic segmentation.

[0027] Additional advantages, objects, and features of the application will be apparent to those skilled in the art upon examination of the following specification. It is intended that the application not be limited by any of the details of the specification, unless so expressly desired, but instead be controlled by the full breadth permitted by the claims. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the drawings, in which:

[0029] Figure 1 is an overall block diagram of the algorithm;

[0030] Figure 2 is a specific detail diagram of the algorithm;

[0031] Figure 3 is a network structure diagram of the semantic segmentation decoder;

[0032] Figure 4 is a diagram of the improved residual module; Figure 4 (a) is a diagram of a conventional residual module structure; Figure 4 (b) is a diagram of an improved residual module structure; Figure 4 (c) is a diagram of a splicing mode. DETAILED DESCRIPTION

[0033] The embodiments of the present application will be described below through specific concrete examples, and other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the present specification. The present application can also be implemented or applied through other different specific embodiments, and various modifications or changes can be made to the details in the present specification based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concepts of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0034] The drawings are only used for exemplary illustration, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation on the present application. In order to better illustrate the embodiments of the present application, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0035] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it is understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back" and the like are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0036] 1. Multi-task learning based unmanned visual perception network structure design

[0037] The key of multi-task learning is to find the relationship between tasks. If the relationship between tasks is properly measured, different tasks can provide each other with additional useful information. Using this additional information, a model with better performance and higher precision can be trained. The detection tasks of traffic targets such as vehicles and pedestrians and the semantic segmentation task of drivable areas have such a correlation, such as the positions of traffic targets such as vehicles and pedestrians must be outside the drivable area. The semantic segmentation task of the drivable area and the lane line segmentation task also have a correlation, such as lane lines are generally within the drivable area and generally exist at the edge of the drivable area.

[0038] Currently, there are two modes of multi-task learning in the field of deep learning, parameter hard constraint and parameter soft constraint. The hard constraint network requires different tasks to share the same bottom neural network, and uses its own parameters and features in the high-level branches of each sub-task. This way can reduce the risk of overfitting. Moreover, the multi-task network with parameter hard constraint has fewer introduced calculation parameters than the multi-task network with parameter soft constraint, so it has faster detection speed. In autonomous driving, the faster the detection speed, the more time the intelligent car has to make decisions, and the safer the autonomous car is, so the detection speed is also an important performance evaluation index. Based on the parameter hard constraint, this paper proposes a multi-task network YOLOT that can simultaneously perform target detection and semantic segmentation. The overall network architecture of YOLOT is as shown in Figure 1 The structure of our YOLOT is composed of an encoder and three decoders, the encoder is a shared network layer, and the three decoders perform three detection tasks respectively. The three detection tasks can learn from each other using the shared network layer, thereby improving their detection accuracy.

[0039] 2. Improved residual connection module

[0040] While the backbone network is extracting features from the input image by continuous convolution, the resolution of the image is continuously reduced, which leads to the loss of edge feature information of the segmentation boundary between different regions. However, the upsampling method cannot recover these feature information. Using a residual module to fuse the bottom features is an effective solution. However, the traditional residual module uses an addition method to fuse different features. This addition method cannot completely preserve the position information from the bottom features.

[0041] The structure of the conventional residual module is shown in Figure 4 (a), which uses an addition method to fuse different features. As shown in equation (1),

[0042]

[0043] where Z represents the feature map after feature fusion, X and Y represent two feature maps to be fused, and c1 and c2 represent the channel numbers of the feature maps X and Y, respectively. At this time, c1 and c2 must be equal to complete the feature fusion in the form of addition. This method does not increase the dimension of the feature map, but each dimension contains more features, which is an efficient choice for ordinary classification tasks. However, this direct addition method does not completely preserve the edge features in the bottom layer, so the edge part between different target regions in the segmentation result will still appear blurred. Therefore, this paper designs an improved residual module, the specific structure is shown in Figure 4 (b). We use a combination of concatenation and convolution to replace addition to achieve feature fusion. The concatenation method completely preserves the bottom feature information, making the edge information between the segmentation regions more complete. As shown in equation (2),

[0044]

[0045] T is the feature map after concatenation, and the relationship between T, X, and Y is shown in equations (3) and (4).

[0046]

[0047]

[0048] Using a combination of concatenation and convolution to fuse different feature layers only requires that the size of the fused feature layers be the same, and there is no requirement for the channel number of the feature layers, i.e., c1 and c2 do not need to be equal at this time, and there is no need to adjust the channel number of the feature layer in advance through convolution, which is more flexible than the addition method. The specific concatenation method is shown in Figure 4 (c), the dimension of the concatenated feature map is increased, and more positioning information is preserved, where the preserved For the underlying feature map containing rich positioning information, when recovering the resolution in the subsequent upsampling, this part of information can be used to accurately segment different regions. Finally, we use 1*1 convolution to fuse the features after splicing and restore the original dimension.

[0049] 3. The specific process of the unmanned visual perception network is as follows

[0050] (1) The input image information is first adjusted in size by the Resize module, and then input into the backbone network. The backbone network is composed of four layers of down-sampling convolution and a Focus module, which mainly extracts effective feature information in the image.

[0051] (2) The feature map after feature extraction is input into the neck network. The neck network is composed of a feature pyramid network and a spatial pyramid pooling module. The main function is to fuse the extracted features, thereby improving the model's ability to model nonlinearity and improving the model's performance.

[0052] (3) The fused features are respectively input into the target detection decoder and the semantic segmentation decoder.

[0053] (4) The target detection decoder is a multi-scale target detector that can detect vehicles, pedestrians, traffic lights, and other targets. The semantic segmentation decoder can achieve semantic segmentation of lane lines and drivable areas.

[0054] Finally, it should be explained that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the present technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. An automatic driving multi-task visual perception method, characterized in that: The method comprises the following steps: S1: The image is preprocessed by a Resize module before being sent into a detection network, and then the preprocessed image set is sent into the detection network; S2: The image sent into the network is processed, feature extraction is performed through a backbone network, and feature fusion is performed through a neck network; S3: The fused features are respectively input into a target detection decoder and a semantic segmentation decoder to realize target detection of vehicle and pedestrian obstacles and semantic segmentation of drivable areas and lane lines; S4: An improved residual connection module is designed in the semantic segmentation head, convolution and feature splicing are used to replace the addition of the traditional residual module, the loss of positioning information in the bottom layer features is reduced, and a decoder capable of performing semantic segmentation on lane lines and drivable areas is designed based on the improved residual module, which solves the problem of fuzzy segmentation edges and improves the detection accuracy of semantic segmentation; The improved residual connection module is specifically: The conventional residual module is shown in formula (1): （1） wherein Z represents the feature map after feature fusion, X i and Y i respectively represent two feature maps needing to be fused, c 1 and c 2 respectively represent the channel numbers of the feature maps X i and Y i , here c 1 and c 2 are equal to complete feature fusion in the form of addition; the improved residual module is shown in formula (2): （2） T i For the feature map after splicing, T i 、 X i 、 Y i The relationship between them is shown in equations (3) and (4): （3） （4） Here c 1 and c 2 need not be equal, where the is a bottom feature map containing rich positioning information, and when the resolution is recovered in the subsequent upsampling, this part of information is used to accurately segment different regions; the 1*1 convolution is used to fuse the features after splicing and restore the original dimension. 2.The automatic driving multi-task visual perception method of claim 1, wherein: In S4, the decoder capable of performing semantic segmentation on lane lines and drivable areas is: firstly, a pyramid pooling module PPM is used to aggregate the context information of different regions in the feature map processed in S3, to improve the ability of the network to obtain global information; then the processed feature map is superimposed with the feature map in the backbone network in S1, and finally 1*1 convolution is used to fuse the two parts of features; to restore the position information of the target in the image without increasing the calculation amount, the nearest neighbor difference method NI is used for upsampling to restore the resolution of the image, thereby realizing semantic segmentation of lane lines and drivable areas. 3.The automatic driving multi-task visual perception method of claim 1, wherein: In S4, the decoder of semantic segmentation uses an improved deep residual connection module, which uses splicing and convolution to replace the addition of the traditional residual module to realize feature fusion; the splicing method completely preserves the bottom layer feature information, making the edge information between the segmentation regions more complete.

Images