An autonomous driving perception system and method based on multi-task learning

Abstract

The present invention discloses an autonomous driving perception system based on multi-task learning, comprising a shared encoder for feature extraction and three decoders for processing specific tasks, wherein the shared encoder comprises a backbone structure and a neck structure, wherein the three decoders comprise an independently operated traffic target detection decoder, a drivable area segmentation decoder and a lane line detection decoder, wherein the backbone structure adopts three identical multi-layer pyramid backbone networks for parallel grouping, and each parallel backbone network is connected by a composite residual structure between layers; the neck structure adopts a pyramid network with the same number of layers as the backbone network, and each input end of the pyramid network is provided with a FEM feature enhancement module. At the same time, the present invention also provides an autonomous driving perception method based on multi-task learning. The present invention enhances the detail of feature extraction, saves computational cost and shortens reasoning time.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving perception method and system based on multi-task learning. Background Art

[0002] In recent years, extensive autonomous driving research has revealed the importance of panoramic driving perception systems. It can extract visual information from images captured by the camera and assist the decision-making system in controlling the vehicle's actions, playing an important role in autonomous driving. In order to better assist vehicle driving, the visual perception system should be able to understand the scene and provide information to the decision-making system, including: the location of obstacles, whether the road is drivable, the location of lanes, etc. Among them, object detection can help vehicles avoid obstacles and comply with traffic rules, while drivable area segmentation and lane detection are the key to planning vehicle routes.

[0003] For driving perception systems, accuracy and real-time performance are the two most critical requirements, which are related to whether autonomous vehicles can make accurate and timely decisions to ensure safety. However, for actual autonomous driving systems, computing resources are often limited. Therefore, it is very challenging to consider these two requirements in real scenarios. Many current methods can handle these tasks separately, such as researchers have proposed Faster R-CNN and YOLOv4 for object detection, ENet and PSPNet for semantic segmentation of driving systems, SCNN and SADENet for lane detection. Although these methods achieve excellent performance, it takes longer to process these tasks one by one than to process all tasks at once. When deploying panoramic driving perception systems on embedded devices commonly used in autonomous vehicles, limited computing resources and latency need to be taken into account. In addition, in the process of traffic scene perception, different tasks often have a lot of related information, such as lanes are often the boundaries of drivable areas, and drivable areas are usually closely surrounded by traffic objects.

[0004] At present, open source technologies have been provided for the application of multi-task learning in autonomous driving perception. A panoramic driving perception network (YOLOP) is proposed to simultaneously perform traffic target detection, drivable area segmentation and lane detection. It is a feedforward network consisting of an encoder for feature extraction and three decoders for processing specific tasks. There are no complex and redundant shared blocks between different decoders, which reduces the computational consumption and enables the network to be easily trained end-to-end. However, in the feedforward network, the feature extraction processing is not detailed enough, and the efficiency of feature extraction is relatively weak, which is not conducive to the subsequent decoder to analyze and utilize the data, and the computational cost is high and the inference time is long. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to address the deficiencies of the prior art and provide an autonomous driving perception system based on multi-task learning, which enhances the detail of feature extraction, saves computing costs and shortens reasoning time.

[0006] At the same time, the present application also provides an autonomous driving perception method based on multi-task learning to solve the above problems.

[0007] The technical solution provided by the present invention is: an autonomous driving perception system based on multi-task learning, comprising a shared encoder for feature extraction and three decoders for processing specific tasks, wherein the shared encoder comprises a backbone structure and a neck structure, and the three decoders comprise an independently operated traffic target detection decoder, a drivable area segmentation decoder and a lane line detection decoder.

[0008] The backbone structure adopts three identical multi-layer pyramid backbone networks in parallel grouping, and each parallel backbone network is connected through a composite residual structure between layers;

[0009] The neck structure adopts a pyramid network with the same number of layers as the backbone network, and a FEM feature enhancement module is set at the input end of each layer of the pyramid network.

[0010] The technical solution of the present invention is further defined as follows: in the backbone structure, the output of each stage in the parallel grouped backbone network flows to the intermediate level stage of its subsequent sibling node, and then the characteristics of the lead backbone are input to the neck structure.

[0011] Furthermore, the FEM feature enhancement module includes two parallel branches. Branch one uses deformable convolution to introduce additional degrees of freedom at specific positions, optimizes the shape of the convolution kernel according to local features, and handles geometric changes of the target; Branch two uses depth convolution and point convolution to form depth-separable convolution, wherein the depth convolution only uses one convolution kernel to perform convolution operations on each input channel, while the point convolution uses a 1x1 convolution kernel to convolve the output of the depth convolution.

[0012] Furthermore, the traffic target detection decoder includes a subtask network with the same number of layers as the backbone network, and multiple subtask networks process data in parallel. Each subtask network includes a parallel convolution module and a region perception module. The convolution module and the region perception module are combined after parallel processing of the data to output the detection result.

[0013] Furthermore, the drivable area segmentation decoder receives the bottom-level feature output information of the neck structure, calculates the features in two parallel paths, and processes them using cosine convolution and dilated convolution respectively. The cosine convolution limits the convolution operation to a specific direction by using a cosine function as a convolution kernel; the dilated convolution expands the receptive field by introducing holes in the convolution kernel.

[0014] Furthermore, the lane detection decoder adopts a stepped network architecture, using 1-3 FEM feature enhancement modules from top to bottom to process image features, and then the three branches simultaneously input depth-separable convolution.

[0015] Furthermore, the depthwise separable convolution includes a data serial depthwise convolution, a point convolution and an activation function, the depthwise convolution uses a convolution kernel to perform a convolution operation on each input channel, and the point convolution uses a 1x1 convolution kernel to convolve the output of the depthwise convolution.

[0016] Another technical solution provided by the present invention is: using the autonomous driving perception method based on multi-task learning of the system as described above, the following steps are performed:

[0017] S1. Input the video or image to be processed into the backbone structure of the shared encoder. The backbone structure is grouped in parallel through three identical multi-layer pyramid backbone networks. After processing, the features of the video or image of three different scales are obtained, and the receptive field is gradually expanded.

[0018] S2, high-level and low-level features of the video or image are input into the neck structure, and the FEM feature enhancement module of the neck structure uses two parallel branches to process the feature details to obtain enhanced features of three different scales;

[0019] S3. Using the enhanced features obtained in step S2, three specific subtasks are processed:

[0020] S3.1 inputs the enhanced features of three different scales into the subtask network of the traffic target detection decoder in parallel, and multiple subtask networks process the data in parallel. Each subtask network includes a parallel convolution module and a region perception module. The convolution module and the region perception module combine the data after parallel processing to output the traffic target detection result.

[0021] S3.2 outputs the bottom-level features of the neck structure to a drivable area segmentation decoder, which calculates the features in two parallel paths, using cosine convolution and dilated convolution respectively. The cosine convolution uses a cosine function as a convolution kernel to limit the convolution operation to a specific direction and outputs a drivable area result.

[0022] S3.3 outputs the bottom-level features of the neck structure to the lane detection decoder. The lane detection decoder adopts a stepped network architecture and uses 1-3 FEM feature enhancement modules from top to bottom to process image features. Subsequently, the three branches simultaneously input the depth-separable convolution to output the lane detection results.

[0023] The technical solution is further limited as follows: the processing method of the FEM feature enhancement module in step S2 is: branch one uses deformable convolution to introduce additional degrees of freedom at a specific position, optimizes the shape of the convolution kernel according to local features, and processes the geometric changes of the target; branch two uses depth convolution and point convolution to form depth separable convolution, wherein the depth convolution uses only one convolution kernel to perform a convolution operation on each input channel, and the point convolution uses a 1x1 convolution kernel to convolve the output of the depth convolution.

[0024] Furthermore, the depth-wise separable convolution in step S3.3 includes a data serial depth convolution, a point convolution and an activation function, wherein the depth convolution uses a convolution kernel to perform a convolution operation on each input channel, and the point convolution uses a 1x1 convolution kernel to convolve the output of the depth convolution.

[0025] Beneficial effects: The autonomous driving perception system and method based on multi-task learning described in the present invention consists of an encoder for feature extraction and three decoders for processing specific tasks. The image analysis process can be accelerated by processing multiple tasks simultaneously instead of sequentially, and the multi-task network shares the same feature extraction backbone, which can realize information sharing between multiple tasks. The present invention redesigns the backbone and neck structure of the mainstream neural network, proposes a feature extraction module FEM, effectively utilizes the advantages of variable convolution and depth-separable convolution, and significantly improves the efficiency of feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a model diagram of an autonomous driving perception system based on multi-task learning provided by the present invention;

[0027] Figure 2 It is a model diagram of the FEM feature enhancement module of the autonomous driving perception system provided by the present invention;

[0028] Figure 3 is a model diagram of a traffic target detection decoder of an autonomous driving perception system provided by the present invention;

[0029] Figure 4 is the original data input in the embodiment of the present invention;

[0030] Figure 5 It is the output data after the original data is processed in the embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0032] Embodiment 1: The present invention provides an autonomous driving perception system based on multi-task learning, the model diagram is as follows Figure 1As shown in the figure, it includes a shared encoder for feature extraction and three decoders for processing specific tasks. This system can simultaneously complete traffic target detection, drivable area segmentation, and lane detection tasks. There are no complex and redundant shared blocks between different decoders, which reduces computational consumption and allows our network to be easily trained end-to-end.

[0033] The shared encoder includes a backbone structure and a neck structure. The backbone structure adopts three identical multi-layer pyramid backbone networks grouped in parallel, and each parallel backbone network is connected by a composite residual structure between layers. It fuses the high- and low-level features of multiple identical backbones, gradually expands the receptive field, and extracts features more effectively. In the backbone structure, in the parallel grouped backbone network, from left to right, the output of each stage flows to the intermediate level stage of its subsequent sibling node, and then the features of the lead backbone are input into the neck structure for feature extraction. Unlike simple network deepening or widening, this network integrates the high- and low-level features of multiple backbone networks, gradually expands the receptive field, and effectively improves the efficiency of image feature extraction.

[0034] The neck structure adopts a pyramid network with the same number of layers as the backbone network, and each input end of the pyramid network is provided with a FEM feature enhancement module. The model diagram of the FEM feature enhancement module is shown in FIG. Figure 2 As shown in the figure, it includes two parallel branches. Branch 1 uses deformable convolution to introduce additional degrees of freedom at specific locations, optimizes the shape of the convolution kernel according to local features, and handles the geometric changes of the target, making the convolution operation more sensitive to complex textures, boundaries and detail features, thereby improving the perception ability and feature expression ability. At the same time, deformable convolution can effectively handle the geometric changes of the target, such as the shape change, posture change and scale change of the target. Traditional convolution operations perform feature extraction within a fixed convolution kernel range, but cannot adapt to the non-rigid deformation of the target. By introducing a deformable convolution kernel, the convolution kernel can be deformed according to the features at a specific location to adapt to the geometric changes of the target. Branch 2 uses depth convolution and point convolution to form a depth separable convolution, where the depth convolution only uses one convolution kernel to perform convolution operations on each input channel, while the point convolution uses a 1x1 convolution kernel to convolve the output of the depth convolution. This decomposition can greatly reduce the number of parameters and computational complexity, especially when there are many input channels. At the same time, it supports feature propagation and feature reuse, thereby reducing parameters and computation. Therefore, it is helpful to ensure the real-time performance of the network.

[0035] The three decoders include a traffic target detection decoder, a drivable area segmentation decoder and a lane line detection decoder which operate independently.

[0036] The model diagram of the traffic target detection decoder is as follows Figure 3As shown in the figure, it includes subtask networks with the same number of layers as the backbone network. Multiple subtask networks process data in parallel to better process image detail texture features and improve detection accuracy. Each subtask network includes parallel convolution modules and region perception modules. The convolution modules and region perception modules process data in parallel and then combine them to output detection results. Within each scale branch, the convolution operation and the region perception module (RPM) are processed in parallel and then combined to output detection results, as shown in the figure. Figure 3 As shown in the figure. Inside the regional perception module, convolution calculations of three different sizes of convolution kernels are performed in parallel to obtain a larger image feature receptive field. After the calculation, the features are combined and the RPM is output after activation. The traffic target detection subnetwork not only effectively utilizes feature information of different scales, but also uses RPM to efficiently extract features to be detected, significantly improving the final detection accuracy of traffic targets.

[0037] The drivable area segmentation decoder receives the bottom-level feature output information of the neck structure, calculates the features in two parallel paths, and processes them respectively using cosine convolution and dilated convolution. The cosine convolution limits the convolution operation to a specific direction by using the cosine function as the convolution kernel; the dilated convolution expands the receptive field by introducing holes in the convolution kernel. By making full use of the directional sensitivity of the cosine convolution, the convolution operation can be limited to a specific direction by using the cosine function as the convolution kernel, so as to better extract the directional features in the image, such as the edge direction or the texture direction. At the same time, the cosine convolution has a certain rotation invariance. Since the cosine convolution kernel is a function with an angle as a parameter, when performing a convolution operation in the image, features at different angles can be identified and matched. The dilated convolution can effectively expand the receptive field by introducing holes (i.e., the dilated rate or the dilation rate) in the convolution kernel. By increasing the number of holes, a wider range of contextual information can be obtained without increasing the parameters and the amount of calculation, so as to better understand and capture the global features in the image. The influence of obstacles and construction areas on the segmentation of the drivable area is significantly reduced. Finally, global pooling is used to reduce computational complexity, improve feature processing efficiency, and obtain the image after segmentation of the drivable area.

[0038] The lane detection decoder adopts a stepped network architecture, and uses 1-3 FEM feature enhancement modules from top to bottom to process image features. Subsequently, the three branches are simultaneously input into the depth separable convolution. The depth separable convolution includes the depth convolution, point convolution and activation function of the data serial. The depth convolution uses a convolution kernel to perform a convolution operation on each input channel, and the point convolution uses a 1x1 convolution kernel to convolve the output of the depth convolution. This decomposition can greatly reduce the number of parameters and computational complexity, especially when there are many input channels. Therefore, the depth separable convolution has advantages on devices with limited computing resources, such as mobile devices and embedded systems. The three-branch results are combined and output to obtain a lane detection image.

[0039] The driving perception system is an important part of autonomous driving. The high-precision real-time perception system can help the vehicle make reasonable decisions during driving. First, it can help the driver monitor the environment and obstacles around the vehicle in real time, avoid collisions and dangerous situations, and improve driving safety. Secondly, the driving perception system can identify road signs, lane lines, traffic lights, etc., and provide real-time navigation guidance to help the driver stay in the correct lane and road, etc. However, mainstream driving perception algorithms are often independent of each other. When deploying a panoramic driving perception system on embedded devices commonly used in autonomous vehicles, limited computing resources and delays need to be considered. In addition, in traffic scenes, different tasks often have a lot of related information. Therefore, multiple models need to be deployed in a driving system, which greatly reduces the perception efficiency of the driving system. In response to this problem, the present invention proposes a driving perception algorithm based on multi-task learning. Traffic target detection (including vehicles and pedestrians), drivable area segmentation and lane detection can be performed simultaneously. This model consists of an encoder for feature extraction and three decoders for processing specific tasks. The image analysis process can be accelerated by processing multiple tasks simultaneously instead of sequentially, and the multi-task network shares the same feature extraction backbone, which can realize information sharing between multiple tasks. Therefore, the present invention is of great significance.

[0040] The specific method provided in this embodiment is carried out according to the following steps:

[0041] S1. The video or image to be processed is input into the backbone structure of the shared encoder. The backbone structure is grouped in parallel through three identical multi-layer pyramid backbone networks. After processing, the features of videos or images of three different scales are obtained, and the receptive field is gradually expanded.

[0042] S2. The high-level and low-level features of the video or image are input into the neck structure. The FEM feature enhancement module of the neck structure uses two parallel branches to process the feature details to obtain enhanced features of three different scales.

[0043] The processing method of the FEM feature enhancement module is as follows: Branch 1 uses deformable convolution to introduce additional degrees of freedom at specific positions, optimizes the shape of the convolution kernel according to local features, and handles the geometric changes of the target; Branch 2 uses depth convolution and point convolution to form depth-separable convolution, where the depth convolution only uses one convolution kernel to perform convolution operations on each input channel, while the point convolution uses a 1x1 convolution kernel to convolve the output of the depth convolution.

[0044] S3. Using the enhanced features obtained in step S2, three specific subtasks are processed.

[0045] In general, this system fully considers the correlation between multiple tasks and selects traffic target detection, drivable area segmentation and lane line detection. The reason is that the lane line is often the boundary of the drivable area, and the drivable area is usually closely surrounded by traffic objects. The three subtasks are related to each other, which significantly improves the efficiency of shared backbone feature extraction and the processing effect of each subtask. In the training stage, the strategy of three-task joint training is adopted. Since the three tasks are interrelated, the model can pay attention to multiple models at the same time, making full use of the advantages of subtask selection and accelerating model training and processing.

[0046] S3.1 inputs the enhanced features of three different scales in parallel into the subtask network of the traffic target detection decoder. Multiple subtask networks process the data in parallel. Each subtask network includes a parallel convolution module and a region perception module. The convolution module and the region perception module combine the data after parallel processing to output the traffic target detection result.

[0047] S3.2 outputs the bottom-level features of the neck structure to the drivable area segmentation decoder. The drivable area segmentation decoder calculates the features in two parallel paths, using cosine convolution and dilated convolution respectively. The cosine convolution uses the cosine function as the convolution kernel to limit the convolution operation to a specific direction and outputs the drivable area result.

[0048] S3.3 outputs the bottom-level features of the neck structure to the lane detection decoder. The lane detection decoder adopts a stepped network architecture and uses 1-3 FEM feature enhancement modules from top to bottom to process image features. Subsequently, the three branches simultaneously input the depth-separable convolution to output the lane detection results.

[0049] The depth-wise separable convolution includes a data serial depth convolution, a point convolution and an activation function, wherein the depth-wise convolution uses a convolution kernel to perform a convolution operation on each input channel, and the point convolution uses a 1x1 convolution kernel to convolve the output of the depth-wise convolution.

[0050] This algorithm can simultaneously complete traffic target detection, drivable area segmentation, and lane detection tasks. The network contains a shared encoder and three subsequent decoders to solve specific tasks. There are no complex and redundant shared blocks between different decoders, which reduces the computational cost and allows our network to be easily trained end-to-end. In this embodiment, the original data of the video / image to be processed is as follows Figure 4 As shown, the output data of the result after the three tasks are combined can be obtained as follows Figure 5 As shown, taking the road image actually taken as an example, after inputting into the present invention, it passes through the traffic target detection algorithm, the drivable area segmentation algorithm, and the lane line detection algorithm at the same time, and the results after the three algorithms are processed are merged into one image to obtain an assisted driving perception image with rich traffic information. Among them, object detection can help vehicles avoid obstacles and comply with traffic rules, and drivable area segmentation and lane detection are the key to planning vehicle routes.

[0051] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. An autonomous driving perception system based on multi-task learning, comprising a shared encoder for feature extraction and three decoders for processing specific tasks, wherein the shared encoder comprises a backbone structure and a neck structure, and the three decoders comprise an independently operated traffic target detection decoder, a drivable area segmentation decoder and a lane line detection decoder, characterized in that: The backbone structure adopts three identical multi-layer pyramid backbone networks in parallel grouping, and each parallel backbone network is connected through a composite residual structure between layers; The neck structure adopts a pyramid network with the same number of layers as the backbone network, and each input end of the pyramid network is provided with a FEM feature enhancement module; In the backbone structure, the output of each stage in the parallel grouped backbone network flows to the intermediate level stage of its subsequent sibling node, and then the characteristics of the lead backbone are input to the neck structure; The FEM feature enhancement module includes two parallel branches. The first branch uses deformable convolution to introduce additional degrees of freedom at specific locations, optimizes the shape of the convolution kernel according to local features, and handles geometric changes of the target. The second branch uses depthwise convolution and point convolution to form depthwise separable convolution, where the depthwise convolution uses only one convolution kernel to perform convolution operations on each input channel, while the point convolution uses a 1x1 convolution kernel to perform convolution on the output of the depthwise convolution. The traffic target detection decoder includes a subtask network with the same number of layers as the backbone network, and the multiple subtask networks process data in parallel. Each subtask network includes a parallel convolution module and a region perception module. The convolution module and the region perception module combine the data after parallel processing and output a detection result. The drivable area segmentation decoder receives the bottom-level feature output information of the neck structure, calculates the features in two parallel paths, and processes them respectively using cosine convolution and dilated convolution. The cosine convolution limits the convolution operation to a specific direction by using a cosine function as a convolution kernel; the dilated convolution expands the receptive field by introducing holes in the convolution kernel. The lane detection decoder adopts a stepped network architecture, using 1-3 FEM feature enhancement modules from top to bottom to process image features, and then the three branches simultaneously input depth-separable convolution.

2. The multi-task learning-based autonomous driving perception system according to claim 1, characterized in that: The depth-wise separable convolution includes a data serial depth convolution, a point convolution and an activation function, wherein the depth-wise convolution uses a convolution kernel to perform a convolution operation on each input channel, and the point convolution uses a 1x1 convolution kernel to convolve the output of the depth-wise convolution.

3. An autonomous driving perception method based on multi-task learning using the system of claim 1 or 2, characterized in that: Proceed as follows: S1. Input the video or image to be processed into the backbone structure of the shared encoder. The backbone structure is grouped in parallel through three identical multi-layer pyramid backbone networks. After processing, the features of the video or image of three different scales are obtained, and the receptive field is gradually expanded. S2, high-level and low-level features of the video or image are input into the neck structure, and the FEM feature enhancement module of the neck structure uses two parallel branches to process the feature details to obtain enhanced features of three different scales; S3. Using the enhanced features obtained in step S2, three specific subtasks are processed: S3.1 inputs the enhanced features of three different scales into the subtask network of the traffic target detection decoder in parallel, and multiple subtask networks process the data in parallel. Each subtask network includes a parallel convolution module and a region perception module. The convolution module and the region perception module combine the data after parallel processing to output the traffic target detection result. S3.2 outputs the bottom-level features of the neck structure to a drivable area segmentation decoder, which calculates the features in two parallel paths, using cosine convolution and dilated convolution respectively. The cosine convolution uses a cosine function as a convolution kernel to limit the convolution operation to a specific direction and outputs a drivable area result. S3.3 outputs the bottom-level features of the neck structure to the lane detection decoder. The lane detection decoder adopts a stepped network architecture and uses 1-3 FEM feature enhancement modules from top to bottom to process image features. Subsequently, the three branches simultaneously input the depth-separable convolution to output the lane detection results.

4. The autonomous driving perception method based on multi-task learning according to claim 3, characterized in that: The processing method of the FEM feature enhancement module in step S2 is as follows: branch one uses deformable convolution to introduce additional degrees of freedom at specific positions, optimizes the shape of the convolution kernel according to local features, and handles the geometric changes of the target; branch two uses depth convolution and point convolution to form depth-separable convolution, where the depth convolution only uses one convolution kernel to perform convolution operations on each input channel, while the point convolution uses a 1x1 convolution kernel to convolve the output of the depth convolution.

5. The autonomous driving perception method based on multi-task learning according to claim 3, characterized in that: The depth-wise separable convolution in step S3.3 includes a data serial depth convolution, a point convolution, and an activation function. The depth-wise convolution uses a convolution kernel to perform a convolution operation on each input channel, and the point convolution uses a 1x1 convolution kernel to convolve the output of the depth-wise convolution.

Images