Autonomous driving visual multi-task perception method and system based on bird's-eye view features

Abstract

The present invention discloses a multi-task perception method and system for autonomous driving vision based on bird's-eye view features, which belongs to the field of autonomous driving technology. It includes: acquiring images from the surround-view camera of the autonomous driving system and extracting features; using a multi-task distillation module to collaboratively predict dense depth features, semantic features and content features to generate dense depth prediction results and dense semantic prediction results; splicing content features and dense semantic prediction results, using dense depth prediction results as the depth weighted value of content features to generate semantically enhanced bird's-eye view features; optimizing bird's-eye view features; predicting the probability of occupied grid category distribution, fusing the probability distribution of foreground categories and the optimized bird's-eye view features; generating target detection prediction results based on the fused features. The present invention fully exploits the correlation between target detection tasks and occupied grid prediction tasks, improves model performance, reduces deployment redundancy caused by multiple independent tasks, and thus reduces training and operation and maintenance costs.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and in particular relates to an autonomous driving visual multi-task perception method and system based on bird's-eye view features. Background Art

[0002] Over the past decade, with the rapid development of computer vision and artificial intelligence technologies, autonomous driving (AD) technology has also surged. In recent years, the rapid emergence of large language models (LLMs) has made end-to-end autonomous driving possible. A complete autonomous driving system typically consists of three major modules: perception, decision planning, and control execution. Decision planning and control execution are already trending towards being replaced by large models. As the first step in autonomous driving, the perception module remains difficult to directly replace and will continue to exist and evolve for a long time to come. The primary goal of perception is to utilize data (such as point clouds and images) acquired by a series of on-board sensors to understand and predict the surrounding driving environment and potential changes. Accurate and efficient perception is a crucial prerequisite and guarantee for the safe and stable operation of autonomous driving systems. Autonomous driving perception tasks can be subdivided into more than a dozen different tasks based on the required functionality, but can generally be categorized into two types: driving scene reconstruction and object recognition. Scene reconstruction primarily aims to understand the autonomous vehicle's surroundings and generate a comprehensive understanding of the current driving scene by modeling surrounding objects, roads, and obstacles. Object recognition focuses on detecting and identifying surrounding dynamic and static targets, such as pedestrians, vehicles, and traffic signs, ensuring the vehicle can accurately assess its relationship with surrounding objects and make safe decisions. These two tasks complement each other and together constitute the autonomous driving system's comprehensive perception of the environment. Currently, high-performance autonomous driving perception models often rely heavily on lidar equipment and use a variety of different models to complete different perception tasks. As a result, current autonomous driving solutions suffer from a series of issues such as high cost, difficult deployment, and limited flexibility and scalability.

[0003] Compared to LiDAR, surround-view cameras are less expensive and can capture dense color information. However, because projection loses depth information, a single camera cannot accurately obtain three-dimensional information. Stereo cameras require high resolution and complex computation to indirectly obtain depth information. Despite this, due to the camera's relatively low cost, vision solutions are still favored by some companies.

[0004] Currently, mainstream autonomous driving technologies typically use multiple independent deep learning models for different tasks. This technical solution has at least three drawbacks: 1) Data feature extraction is a common deep learning network structure across all tasks, and feature extraction networks are one of the fundamental structures of modern neural network models. Training different feature extraction networks for multiple independent models increases training costs; 2) Different perception tasks have certain correlations, and it is difficult for several independent models to leverage these correlations to improve the accuracy of their own tasks; 3) Multiple independent models add unnecessary redundancy to the deep learning model, increasing the actual deployment cost of the algorithm. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of traditional autonomous driving technology that deploys multiple independent perception models and to improve existing bird's-eye view-based perception solutions. A method and system for autonomous driving visual multi-task perception based on bird's-eye view features is proposed. The model proposed in this invention can effectively utilize the synergy between target detection and occupancy grid prediction tasks, reducing training costs and deployment difficulty. By leveraging the rich color information of the image, the semantic features of the image are predicted, thereby enhancing the representation capability of the bird's-eye view features. The cross-attention mechanism designed for explicit region of interest enhancement can fully utilize the results of layer-by-layer prediction to improve the model's final prediction results.

[0006] The technical solutions adopted in the present invention are as follows:

[0007] The variable subscripts "det" and "occ" are used to distinguish between detection and occupancy prediction tasks, and the variable superscripts are used to distinguish between intermediate variables within a submodule. "F" represents the intermediate features of the network, and the variable subscript "prev" represents the output of the previous layer in the Transformer structure.

[0008] In a first aspect, the present invention proposes a multi-task visual perception method for autonomous driving based on bird's-eye view features, comprising the following steps:

[0009] (1) Obtain a set of images from the surround view camera of the autonomous driving system and extract image features;

[0010] (2) Based on the image features obtained in step (1), a multi-task distillation module is used to collaboratively predict dense depth features, dense semantic features, and content features, and dense depth prediction results and dense semantic prediction results are generated by combining the dense depth features and dense semantic features;

[0011] (3) Splicing content features and dense semantic prediction results, using dense depth prediction results as the depth weighted value of content features, to generate semantically enhanced bird's-eye view features;

[0012] (4) Optimizing the bird’s-eye view features using a bird’s-eye view feature optimization encoder with a deformable attention mechanism;

[0013] (5) Predict the probability distribution of the occupied grid categories based on the optimized bird's-eye view features, extract the probability distribution of the foreground category, and fuse the probability distribution of the foreground category with the optimized bird's-eye view features;

[0014] (6) Use the target detection neural network to process the features fused in step (5) to generate target detection prediction results.

[0015] The step (2) comprises:

[0016] (2.1) using three networks with the same structure to preliminarily process the image features to obtain dense depth features, dense semantic features, and content features; the parameters of the three networks with the same structure are not shared;

[0017] (2.2) A multi-task distillation module is used to fuse dense depth features and dense semantic features to obtain dense depth prediction results and dense semantic prediction results.

[0018] Furthermore, the calculation process of the multi-task distillation module is as follows:

[0019] G=σ(F2)

[0020] R=Conv(F1)G

[0021] Among them, R is the output result of the multi-task distillation module, F1 and F2 are one of the dense depth features and dense semantic features respectively, F1≠F2, when F1 is a dense depth feature, the calculated R is the dense depth prediction result, when F1 is a dense semantic feature, the calculated R is the dense semantic prediction result; σ is the activation function, G is the weighted probability between [0, 1], and Conv(.) represents convolution.

[0022] Furthermore, the semantically enhanced bird's-eye view feature generation method is:

[0023] LSS technology is used to back-project the concatenation of content features and dense semantic prediction results into the camera coordinate system according to the depth weighted value, and then the coordinate system is transformed into the vehicle coordinate system to obtain the frustum pseudo point cloud.

[0024] The frustum pseudo point cloud is converted into voxel features, and the dimension is reduced along the height to obtain the semantically enhanced bird's-eye view features.

[0025] Furthermore, the step (4) includes:

[0026] (4.1) Downsampling and encoding the semantically enhanced bird's-eye view features step by step, with the downsampling encoding results at each level forming a feature pyramid; the resolution of the encoded features in the feature pyramid is gradually reduced;

[0027] (4.2) The encoding layer composed of a deformable attention mechanism and a convolutional output head is used to process the encoded features in the feature pyramid step by step, gradually restoring them from low resolution to high resolution, and finally obtaining the optimized bird's-eye view features with the same resolution as the semantically enhanced bird's-eye view features.

[0028] Furthermore, the step-by-step decoding process is as follows:

[0029]

[0030] in, is the encoding feature of the kth layer of the feature pyramid, K is the number of layers of the feature pyramid, DeformAttn is the deformable attention mechanism, is the feature weighted value of the k-th layer encoding feature in the feature pyramid, UpConv is the upconvolution layer, and UpSample is the nearest neighbor interpolation operation.

[0031] Furthermore, in the step (5), a convolutional neural network semantic output head is used to generate the occupancy grid category distribution probability with the optimized bird's-eye view features as input.

[0032] Furthermore, the calculation process of the target detection neural network in step (6) includes:

[0033] (6.1) Initialize the target query Q:

[0034]

[0035] Among them, q k is the kth initial detection box within the predefined perception range, N is the number of detection boxes, (x, y, z) is the geometric center of the detection box in three-dimensional space, (w, l, h) are the length, width, and height of the detection box, raw is the deflection angle of the detection box from a bird's-eye view, and Contfeat is a learnable embedding feature;

[0036] The sensing range is evenly divided into N squares, and the geometric center and geometric size of the square are used to initialize (x, y, w, l), and (z, h, raw) are initialized to 0;

[0037] (6.2) Using the cross-attention mechanism enhanced by the region of interest, the target feature F is extracted from the fused features in step (5) through the target query Q obj ={f0, f, ..., f N-1};

[0038] (6.3) Calculate the self-attention result of the target feature and update the target feature according to the self-attention result:

[0039]

[0040] Among them, query, key, and value represent the query vector, key vector, and value vector obtained by the target feature mapping, respectively. key Indicates the number of channels of the value vector, the superscript T indicates transpose, FFN indicates feedforward neural network, and softmax indicates the softmax function;

[0041] (6.4) Use a multi-layer perceptron to predict the detection box of the updated target features;

[0042] (6.5) Use the updated target features as new embedding features and the detection box prediction results as the new initial detection box to generate a new target query Q. Repeat steps (6.2)-(6.4), use the updated target query as the input of the next layer of network, and repeat the calculation process by stacking several layers of networks. The prediction results of the last layer are used as the final target recognition results.

[0043] Furthermore, the step (6.2) is specifically as follows:

[0044] For a single q k In terms of q, using the region of interest feature aggregation technology, k The features fused in step (5) are uniformly sampled within the corresponding grid range to obtain a fixed-length feature f k , traverse the query Q to get the target feature F obj :

[0045] F obj ={f0, f, ..., f N-1}

[0046] Extract features using region of interest feature aggregation technology k When q k The corresponding range is scaled by fixed multiples m1, m2, and m3, and the embedded feature Contfeat is passed through the multi-layer perceptron MLP to generate the attention weight α:

[0047] α=MLP(Contfeat)

[0048]

[0049] in, They represent the region of interest feature aggregation technology used to analyze q k Zoom in by m1, m2, or m3 times.

[0050] In a second aspect, the present invention proposes an autonomous driving visual multi-task perception system based on bird's-eye view features, which is used to implement the above-mentioned autonomous driving visual multi-task perception method based on bird's-eye view features.

[0051] Beneficial effects of the present invention:

[0052] The present invention designs a design method for a pure visual modality multi-task perception scheme for autonomous driving based on bird's-eye view features, covering multiple autonomous driving processes such as the generation, optimization, occupancy grid prediction, design and implementation of target detection heads, and serial multi-task collaboration of bird's-eye view features. In the generation and optimization stage of bird's-eye view features, the present invention can make full use of the dense color information of image data, and make the generated bird's-eye view features have stronger representation capabilities through semantic prediction. In the optimization stage, the redundant features introduced by monocular depth estimation are effectively reduced by predicting the weighted weights of the importance of different positions of bird's-eye view features. In the multi-task implementation stage, the present invention designs a new attention mechanism that can make full use of the intermediate results of the detection head to improve the final prediction accuracy of the model. The present invention uses a serial collaborative approach to fully utilize the synergy of occupancy grid prediction and detection, effectively reducing the training cost and the difficulty of model deployment, while achieving improved performance of different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flowchart of the multi-task perception method for autonomous driving based on bird's-eye view features proposed in this invention. DETAILED DESCRIPTION

[0054] The present invention is further described below with reference to the accompanying drawings.

[0055] The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0056] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0057] like Figure 1 , the specific implementation steps of the present invention are as follows:

[0058] Step (1). The autonomous driving system inputs data collected by multiple cameras installed at different locations on the vehicle body. The shared image feature extraction network is used to extract image features, specifically:

[0059]

[0060] in, It is a backbone convolutional neural network shared by multiple images. The input data dimension can be recorded as W×H×3, the image resolution is W×H, Represents the image features of the i-th image taken by cameras at different positions, i = 1, 2, ..., N. Use the backbone convolutional neural network to uniformly extract features from the image and downsample the image to 1 / 16, that is, F cam The dimensions are N×W / 16×H / 16×C. C is the channel dimension for feature extraction, which can be set to 256 or 512 depending on the specific situation in this embodiment. N is the number of cameras included in a set of surround view cameras, which can be set according to the specific configuration in this embodiment.

[0061] Step (2). Use the multi-task distillation module to collaboratively predict the dense depth features and dense semantic features of the downsampled resolution image, and further process the image features of step (1) to obtain the image content features, specifically:

[0062] Step (2.1). Use three prediction networks C with the same structure dep 、C sem with C cont The image feature F obtained in step (1) cam Perform preliminary processing to obtain dense depth features F dep , dense semantic features F sem and content features F cont .

[0063] F dep =C dep (F cam )

[0064] F sem =C sem (F cam )

[0065] F cont =C cont (F cam )

[0066] In this embodiment, C dep 、C sem with C cont It consists of two convolutional neural network blocks with residual connections.

[0067] Step (2.2). The dense depth feature F is fused by the multi-task distillation module Mtd dep With dense semantic features F sem , and obtain a dense depth prediction result R that is consistent with the feature resolution dep and dense semantic prediction results R sem .

[0068] R dep =Mtd(F dep , F sem )

[0069] R sem =Mtd(F sem , F dep )

[0070] Among them, the multi-task distillation module Mtd(F1, F2) generates feature weighted values ​​through the self-attention mechanism to achieve multi-task collaboration. The specific steps of Mtd(F1, F2) are as follows.

[0071] G=σ(F2)

[0072] R=Conv(F1)G

[0073] Among them, σ is a convolutional neural network with a channel number of 1, and the activation function is the sigmoid function to generate a weighted probability G in the range of [0, 1]; Conv(.) is the convolution function, and R is the prediction result R dep or R sem . Dense semantic prediction results R sem Represents the semantic classification probability of the image, and the classification category is consistent with the ground-truth category of the occupied grid. Dense depth prediction result R dep It is a discrete probability distribution, which indicates the probability value of the current pixel depth value being in the discrete depth grid.

[0074] The present invention uses an additional prediction network C sem Together with the multi-task distillation module Mtd, it can effectively avoid the single-task performance degradation caused by the imbalance between tasks due to the difference in the number of depth prediction and semantic prediction categories, while effectively utilizing the synergy between the two tasks to improve the accuracy of depth prediction tasks and semantic prediction tasks.

[0075] Step (3). The content feature F in step (2) cont and dense semantic prediction results R sem After splicing and fusion along the channel, the dense depth prediction result R is used depThe depth-weighted value of the content feature is back-projected into the camera coordinate system using the LSS (Lift, Splat, Shoot) technique. This is then transformed into the ego-vehicle coordinate system to obtain a pseudo-point cloud of the viewing cone. Finally, the pseudo-point cloud of the viewing cone projected into the ego-vehicle coordinate system is converted into voxel features using point cloud voxelization. Dimensionality reduction along the height is then performed to obtain semantically enhanced bird's-eye view features, denoted as BEV, completing the conversion of 2D image features into 3D features.

[0076] Step (4). Optimize the encoder TE using the bird’s-eye view features with deformable attention mechanism ref Suppress the depth uncertainty noise introduced by the monocular depth estimation in step (2), reduce the redundancy of the bird's-eye view features obtained in step (3), highlight the useful features, and obtain the optimized bird's-eye view features; the bird's-eye view feature optimization encoder TE ref Using the Transformer neural network structure, the specific process is as follows:

[0077] Step (4.1). Encoder TE ref First, a multi-layer convolutional neural network is used to obtain a feature pyramid with gradually decreasing resolution.

[0078] In this embodiment, the bird's-eye view feature pyramid BEV pya The bird's-eye view features include original resolution, 1 / 2 downsampled resolution, and 1 / 4 downsampled resolution, that is, K=3. In actual process, it can be flexibly configured according to specific circumstances.

[0079] BEV pya =MConv(BEV)

[0080] Among them, BEV is a semantically enhanced bird's-eye view feature, MConv is a multi-layer convolutional neural network with gradual downsampling, and the feature pyramid BEV can be obtained by collecting the results of the intermediate convolutional layers. pya .

[0081] Step (4.2). Process the image from low resolution to high resolution step by step. At each level, two encoding layers consisting of a deformable attention mechanism and a convolutional output head are used to predict the bird's-eye view occupancy probability (feature weight value) consistent with the input bird's-eye view feature resolution.

[0082]

[0083] in, is the encoding feature of the kth layer of the feature pyramid, DeformAttn is the deformable attention mechanism, is the weighted value of the feature of the kth layer, UpConv is the up-convolution layer, which increases the resolution of the feature, and UpSample is the nearest neighbor interpolation, which upsamples the low-resolution weighted value and weights it with the feature to update the feature. When k=K, That is, the features corresponding to the original resolution are not upsampled, but only weighted after the occupancy probability is predicted by the deformable attention encoding layer.

[0084] In addition, the module network can adaptively learn the bird's-eye view occupancy probability based on the dataset, and can also use the task of occupancy grid prediction to generate the bird's-eye view occupancy probability for supervised learning. By gradually predicting the bird's-eye view occupancy probability, this module can effectively reduce the redundancy along the light direction caused by the inaccuracy of depth prediction in the back projection process in step (3), thereby improving the overall performance of the model.

[0085] Step (5). Using convolutional neural network as the basic structure, design the occupancy grid prediction head C occ , using the bird's-eye view features optimized in step (4) as input, and using sub-pixel convolution technology, the optimized bird's-eye view features are upgraded to output the probability distribution results of the grid categories.

[0086] Step (5.1).C occ For the semantic output head of the convolutional neural network, the bird's-eye view feature Bev obtained by optimizing step (4) is used. ref As input, the sub-pixel convolution technique is used to upgrade the bird's-eye view features and output the category probability distribution of each grid. occ Use cross-entropy loss function (Cross-entropy loss) and set similarity metric loss Dice-loss to supervise training:

[0087] Occ=C occ (Bev ref )

[0088] In this embodiment, C occ is a convolutional neural network, and Occ is the classification probability of the occupied grid category (occupied grid prediction result).

[0089] Step (5.2). The classification probability of the occupied grid category predicted by the above step (5.1) is the final result of the occupied grid prediction task. The foreground part is extracted from Occ and the probability distribution of the Occ foreground category is Occ. front With Bev ref Fusion, generating fused feature BEV fusion It is used for subsequent target detection to achieve serial multi-task collaboration. The feature fusion here can be achieved by conventional fusion methods, such as directly splicing Occ front and Bev ref.

[0090] Step (6). Use the target detection neural network T det For the fused feature BEV obtained in step (5.2) fusion Processing is performed to generate target detection prediction results. The specific calculation process is as follows:

[0091] Step (6.1). Initialize the target query (Object Query) Q:

[0092]

[0093] q k It is a series of initial detection frames set within the predefined perception range of the present invention. N is the number of detection frames, which can be set according to specific needs. (x, y, z) is the geometric center of the detection frame in three-dimensional space. (w, l, h) is the length, width and height of the detection frame. Raw is the deflection angle of the detection frame under the bird's-eye view (top view). Contfeat is a learnable embedding feature, initialized with learnable parameters. The perception range is evenly divided into N squares, and the geometric center and geometric size of the squares are used to initialize (x, y, w, l), and (z, h, raw) are initialized to 0. The perception method based on bird's-eye view features usually has a perception range of a rectangular area centered on the vehicle, and the specific value can be flexibly adjusted according to the configured sensor. Taking the perception range of ±50m and N=100 as an example, q k It actually represents a cylindrical area with a length and width of 10m.

[0094] Step (6.2). Use the cross attention mechanism enhanced by the region of interest to obtain the fused feature BEV from step (5.2) through the target query Q fusion Extract the target feature F obj .

[0095] Specifically, for a single q k In terms of, using RoIAlign (Region of Interest Align) regional feature aggregation technology, we can k The fusion features are uniformly sampled within the corresponding grid range to obtain a fixed-length feature f k , traverse the query Q to get the target feature F obj :

[0096] F obj ={f0, f, ..., f N-1}

[0097] In one embodiment of the present invention, RoI Align is used to extract f k When q kThe corresponding range is scaled by a fixed multiple (e.g. 1.2, 1.0, 0.8), and the attention weight α is generated by the embedded feature Contfeat through a multi-layer perceptron.

[0098] α=MLP(Contfeat)

[0099] f k =α[RoI Align 1.2 (q k ), RoI Align 1.0 (q k ), RoI Align 0.8 (q k )] T

[0100] The above operation of obtaining target features using the target query Q is the proposed cross-attention mechanism MsRoIAttn for region of interest enhancement. The process of extracting target features can be expressed as:

[0101] F obj =MsRoIAttn(BEV fusion , Q)

[0102] Step (6.3). Use the Transformer standard self-attention mechanism to process the target feature F obj , through the linear layer to transform f k Mapped into query vector (query), key vector (key), value vector (value) and update the target feature. The update process is expressed as:

[0103]

[0104] Among them, C key It represents the number of channels of the three vectors, and FFN represents the feedforward neural network.

[0105] Step (6.4). Use the multi-layer perceptron to update the target feature F obj Predict the detection boxes separately, specifically:

[0106] Δx, Δy,z,w,l,h,raw,p class =MLP(f k )

[0107] x,y=x+Δx,y+Δy

[0108] Among them, Δ indicates that the parameter is predicted as an offset, p class The classification result of the target.

[0109] Step (6.5). The above process is the target detection neural network T detThe calculation process of a layer of network is to take the target feature F obj As the new embedding feature, the detection box prediction result is used as the new initial detection box to generate a new target query Q. Repeat steps (6.2)-(6.4), and use the updated target query as the input of the next layer of network. By stacking three or six layers of network, the above calculation process is continuously repeated to achieve a coarse-to-fine prediction paradigm, and the prediction result of the last layer is used as the final target recognition result.

[0110] The present invention also provides a multi-task visual perception system for autonomous driving based on bird's-eye view features, which is used to implement the above-mentioned embodiments. Terms such as "module," "unit," and the like used below may refer to a combination of software and / or hardware that implements a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible.

[0111] This embodiment provides an autonomous driving visual multi-task perception system based on bird's-eye view features, including:

[0112] An image feature extraction module, which is used to obtain a set of images from the surround view camera of the autonomous driving system and extract image features;

[0113] The dense depth, semantic, and content prediction module is used to collaboratively predict dense depth features, dense semantic features, and content features based on the image features extracted by the image feature extraction module using a multi-task distillation module, and then generate dense depth prediction results and dense semantic prediction results by combining the dense depth features and dense semantic features;

[0114] A bird's-eye view feature generation module, which is used to stitch content features and dense semantic prediction results, and use dense depth prediction results as the depth weighted value of content features to generate semantically enhanced bird's-eye view features;

[0115] A bird's-eye view feature optimization module for optimizing bird's-eye view features using a bird's-eye view feature optimization encoder with a deformable attention mechanism;

[0116] The occupancy grid prediction task module is used to predict the occupancy grid category distribution probability based on the optimized bird's-eye view features, extract the probability distribution of the foreground category from it, and fuse the probability distribution of the foreground category with the optimized bird's-eye view features;

[0117] The target detection task module is used to use the target detection neural network to process the features fused by the occupied grid prediction task module to generate target detection prediction results.

[0118] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. For example, the regression enhancement module includes:

[0119] An encoder module for encoding real process data and virtual process data;

[0120] A pre-trained multi-head attention module, which is used to extract features of the encoded real process data and virtual process data;

[0121] The linear prediction layer module is used to predict the virtual measurement data according to the features of the encoded real process data and the virtual process data.

[0122] The implementation methods of the remaining modules will not be described in detail here. The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. Those of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0123] Embodiments of the system of the present invention can be applied to any device with data processing capabilities, such as a computer or other device. System embodiments can be implemented through software, hardware, or a combination of software and hardware. For example, a software implementation, as a logical device, is implemented by a processor of any device with data processing capabilities, reading corresponding computer program instructions from non-volatile memory into internal memory and executing them.

[0124] Obviously, the above-described embodiments and drawings are only some examples of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without the need for creative work. In addition, it is understandable that although the work done in this development process may be complex and lengthy, for those of ordinary skill in the art, certain changes in design, manufacturing or production based on the technical content disclosed in this application are merely conventional technical means and should not be regarded as insufficient disclosure of the present application. Without departing from the concept of the present application, several variations and improvements can also be made, which all fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be based on the appended claims.

Claims

1. A multi-task visual perception method for autonomous driving based on bird's-eye view features, characterized by: The following steps are involved: (1) Obtain a set of images from the surround view camera of the autonomous driving system and extract image features; (2) Based on the image features obtained in step (1), a multi-task distillation module is used to collaboratively predict dense depth features, dense semantic features, and content features, and dense depth prediction results and dense semantic prediction results are generated by combining the dense depth features and dense semantic features; (3) Splicing content features and dense semantic prediction results, using dense depth prediction results as the depth weighted value of content features, to generate semantically enhanced bird's-eye view features; (4) Optimizing the bird’s-eye view features using a bird’s-eye view feature optimization encoder with a deformable attention mechanism; (5) Predict the probability distribution of the occupied grid categories based on the optimized bird's-eye view features, extract the probability distribution of the foreground category, and fuse the probability distribution of the foreground category with the optimized bird's-eye view features; (6) Using the target detection neural network to process the features fused in step (5) to generate target detection prediction results; Specifically, step (2) includes: (2.1) using three networks with the same structure to preliminarily process the image features to obtain dense depth features, dense semantic features, and content features; the parameters of the three networks with the same structure are not shared; (2.2) Using a multi-task distillation module to fuse dense depth features and dense semantic features, we can obtain dense depth prediction results and dense semantic prediction results. The calculation process of the multi-task distillation module is as follows: G=σ(F2) R=Conv(F1)G Among them, R is the output result of the multi-task distillation module, F1 and F2 are one of the dense depth features and dense semantic features respectively, F1≠F2, when F1 is a dense depth feature, the calculated R is the dense depth prediction result, when F1 is a dense semantic feature, the calculated R is the dense semantic prediction result; σ is the activation function, G is the weighted probability between [0,1], and Conv(.) represents convolution.

2. The autonomous driving visual multi-task perception method based on bird's-eye view features according to claim 1 is characterized in that: The semantically enhanced bird's-eye view feature generation method is as follows: LSS technology is used to back-project the concatenation of content features and dense semantic prediction results into the camera coordinate system according to the depth weighted value, and then the coordinate system is transformed into the vehicle coordinate system to obtain the frustum pseudo point cloud. The frustum pseudo point cloud is converted into voxel features, and the dimension is reduced along the height to obtain semantically enhanced bird's-eye view features.

3. The autonomous driving visual multi-task perception method based on bird's-eye view features according to claim 1 is characterized in that: The step (4) comprises: (4.1) Downsampling and encoding the semantically enhanced bird's-eye view features step by step, with the downsampling encoding results at each level forming a feature pyramid; the resolution of the encoded features in the feature pyramid is gradually reduced; (4.2) The encoding layer composed of a deformable attention mechanism and a convolutional output head is used to process the encoded features in the feature pyramid step by step, gradually restoring them from low resolution to high resolution, and finally obtaining the optimized bird's-eye view features with the same resolution as the semantically enhanced bird's-eye view features.

4. The autonomous driving visual multi-task perception method based on bird's-eye view features according to claim 3 is characterized in that: The step-by-step decoding process is as follows: in, is the encoding feature of the kth layer of the feature pyramid, K is the number of layers of the feature pyramid, DeformAttn is the deformable attention mechanism, is the feature weighted value of the k-th layer encoding feature in the feature pyramid, UpConv is the upconvolution layer, and UpSample is the nearest neighbor interpolation operation.

5. The autonomous driving visual multi-task perception method based on bird's-eye view features according to claim 1 is characterized in that: In the step (5), a convolutional neural network semantic output head is used to generate the probability of grid category distribution using the optimized bird's-eye view features as input.

6. The autonomous driving visual multi-task perception method based on bird's-eye view features according to claim 1 is characterized in that: The calculation process of the target detection neural network in step (6) includes: (6.1) Initialize the target query Q: Among them, q k is the kth initial detection box within the predefined perception range, N is the number of detection boxes, (x, y, z) is the geometric center of the detection box in three-dimensional space, (w, l, h) is the length, width, and height of the detection box, raw is the deflection angle of the detection box from a bird's-eye view, and Contfeat is a learnable embedding feature; The sensing range is evenly divided into N squares, and the geometric center and geometric size of the square are used to initialize (x, y, w, l), and (z, h, raw) are initialized to 0; (6.2) Using the cross-attention mechanism enhanced by the region of interest, the target feature F is extracted from the fused features in step (5) through the target query Q obj ={f0,f,…,f N-1 }; (6.3) Calculate the self-attention result of the target feature and update the target feature according to the self-attention result: Among them, query, key, and value represent the query vector, key vector, and value vector obtained by the target feature mapping, respectively. key Indicates the number of channels of the value vector, the superscript T indicates transpose, FFN indicates feedforward neural network, and softmax indicates the softmax function; (6.4) Use a multi-layer perceptron to predict the detection box of the updated target features; (6.5) Use the updated target features as new embedding features and the detection box prediction results as the new initial detection box to generate a new target query Q. Repeat steps (6.2)-(6.4), use the updated target query as the input of the next layer of network, and repeat the calculation process by stacking several layers of networks. The prediction results of the last layer are used as the final target recognition results.

7. The autonomous driving visual multi-task perception method based on bird's-eye view features according to claim 6 is characterized in that: The step (6.2) is specifically: For a single q k In terms of q, using the region of interest feature aggregation technology, k The features fused in step (5) are uniformly sampled within the corresponding grid range to obtain a fixed-length feature f k , traverse the query Q to get the target feature F obj : F obj ={f0,f,…,f N-1 } The feature f is extracted using the region of interest feature aggregation technique k When q k The corresponding range is scaled by fixed multiples m1, m2, and m3, and the embedded feature Contfeat is passed through the multi-layer perceptron MLP to generate the attention weight α: α=MLP(Contf eat) in, They represent the region of interest feature aggregation technology used to analyze q k Zoom in by m1, m2, or m3 times.

8. A multi-task autonomous driving visual perception system based on bird's-eye view features, used to implement the method of claim 1, characterized in that: The system comprises: An image feature extraction module, which is used to obtain a set of images from the surround view camera of the autonomous driving system and extract image features; The dense depth, semantic, and content prediction module is used to collaboratively predict dense depth features, dense semantic features, and content features based on the image features extracted by the image feature extraction module using a multi-task distillation module, and then generate dense depth prediction results and dense semantic prediction results by combining the dense depth features and dense semantic features; A bird's-eye view feature generation module, which is used to stitch content features and dense semantic prediction results, and use dense depth prediction results as the depth weighted value of content features to generate semantically enhanced bird's-eye view features; A bird's-eye view feature optimization module for optimizing bird's-eye view features using a bird's-eye view feature optimization encoder with a deformable attention mechanism; The occupancy grid prediction task module is used to predict the occupancy grid category distribution probability based on the optimized bird's-eye view features, extract the probability distribution of the foreground category from it, and fuse the probability distribution of the foreground category with the optimized bird's-eye view features; The target detection task module is used to use the target detection neural network to process the features fused by the occupied grid prediction task module to generate target detection prediction results.

Images