Automatic driving vision multitask sensing method and system based on aerial view features

Abstract

The invention discloses an automatic driving vision multitask sensing method and system based on aerial view features, and belongs to the technical field of automatic driving. Comprising the following steps: acquiring images from a look-around camera of the automatic driving system and extracting features; a multi-task distillation module is adopted to cooperatively predict dense depth features, semantic features and content features, and a dense depth prediction result and a dense semantic prediction result are generated; splicing the content feature and the dense semantic prediction result, and generating a semantic enhanced aerial view feature by using the dense depth prediction result as a depth weighted value of the content feature; optimizing aerial view features; predicting the distribution probability of occupied grid categories, fusing the probability distribution of foreground categories and optimized aerial view features; and generating a target detection prediction result according to the fused features. According to the method, the relevance between the target detection task and the occupied grid prediction task is fully mined, the model performance is improved, deployment redundancy caused by multiple independent tasks is reduced, and therefore the training and operation and maintenance cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to an autonomous driving visual multi-task perception method and system based on bird's-eye view features. Background Art

[0002] In the past decade, with the rapid development of computer vision and artificial intelligence technology, autonomous driving technology has also risen. In recent years, due to the emergence of large language models (LLM), end-to-end autonomous driving technology has become possible. A complete autonomous driving system usually consists of three main modules: perception, decision planning, and control execution. Among them, decision planning and control execution have tended to be replaced by large models. As the first step of autonomous driving tasks, the perception module is still difficult to be directly replaced and will exist and develop for a long time in the future. The main purpose of the perception task is to use the data obtained by a series of on-board sensors (such as point clouds, images, etc.) to understand and predict the surrounding driving environment and possible changes. Accurate and efficient perception is an important prerequisite and guarantee for the safe and stable operation of the autonomous driving system. The autonomous driving perception task can be subdivided into more than a dozen different tasks according to the functions to be achieved, but it can be roughly summarized into two categories: one is driving scene reconstruction, and the other is target recognition. The main purpose of scene reconstruction is to understand the environmental conditions of the autonomous driving vehicle, and to generate a comprehensive understanding of the current driving scene by modeling surrounding objects, roads and obstacles. Target recognition focuses on detecting and identifying surrounding dynamic and static targets, such as pedestrians, vehicles, traffic signs, etc., to ensure that the vehicle can accurately assess its relationship with surrounding objects and make safe decisions. These two types of tasks complement each other and together constitute the autonomous driving system's comprehensive perception of the environment. At present, high-performance autonomous driving perception models are often highly dependent on lidar equipment and use a series of different models to complete different perception tasks. Therefore, current autonomous driving solutions have a series of problems such as high cost, inconvenient deployment, limited flexibility and scalability, etc.

[0003] Compared to LiDAR, surround view cameras are cheaper and can obtain dense color information. However, since projection will lose depth information, a single camera cannot obtain accurate three-dimensional information, and stereo cameras require high resolution and complex calculations to indirectly obtain depth information. Despite this, due to the relatively low cost of cameras, visual solutions are still favored by some companies.

[0004] At present, mainstream autonomous driving technology usually uses multiple independent deep learning models for different tasks. This technical solution has at least the following three disadvantages: 1) Data feature extraction is a deep learning network structure common to all tasks, and the feature extraction network is one of the basic structures of modern neural network models. Training different feature extraction networks for multiple independent models will increase the training cost; 2) There is a certain correlation between different perception tasks, and it is difficult for several independent models to use this correlation to improve the accuracy of their own tasks; 3) Multiple independent models increase unnecessary redundancy of deep learning models and increase the actual deployment cost of the algorithm. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of traditional autonomous driving technology that deploys multiple independent perception models and to improve the existing perception scheme based on bird's-eye view, and to propose an autonomous driving visual multi-task perception method and system based on bird's-eye view features. The model proposed in the present invention can effectively utilize the synergy of target detection and grid occupancy prediction tasks to reduce training costs and deployment difficulties. By utilizing the rich color information of the image, the semantic features of the image are predicted and the representation ability of the bird's-eye view features is enhanced. The cross-attention mechanism designed for explicit region of interest enhancement can make full use of the results of layer-by-layer prediction to improve the final prediction results of the model.

[0006] The technical solution adopted by the present invention is as follows:

[0007] The variable subscripts "det" and "occ" are used to distinguish the detection task and the occupancy grid prediction task, and the variable superscripts are used to distinguish the intermediate variables in the submodule. "F" represents the intermediate feature of the network, and the variable subscript "prev" represents the output result of the previous layer in the Transformer structure.

[0008] In a first aspect, the present invention proposes a multi-task perception method for autonomous driving vision based on bird's-eye view features, comprising the following steps:

[0009] (1) Obtain a set of images from the surround view camera of the autonomous driving system and extract image features;

[0010] (2) Based on the image features obtained in step (1), a multi-task distillation module is used to collaboratively predict dense depth features, dense semantic features, and content features, and dense depth features and dense semantic features are combined to generate dense depth prediction results and dense semantic prediction results;

[0011] (3) Concatenate content features and dense semantic prediction results, use dense depth prediction results as the depth weighted value of content features, and generate semantically enhanced bird's-eye view features;

[0012] (4) Optimizing the bird’s-eye view features using a bird’s-eye view feature optimization encoder with a deformable attention mechanism;

[0013] (5) Predict the distribution probability of the occupied grid categories based on the optimized bird's-eye view features, extract the probability distribution of the foreground category, and fuse the probability distribution of the foreground category with the optimized bird's-eye view features;

[0014] (6) Use the target detection neural network to process the features fused in step (5) to generate target detection prediction results.

[0015] The step (2) comprises:

[0016] (2.1) using three networks with the same structure to preliminarily process the image features to obtain dense depth features, dense semantic features and content features; the parameters of the three networks with the same structure are not shared;

[0017] (2.2) A multi-task distillation module is used to fuse dense depth features and dense semantic features to obtain dense depth prediction results and dense semantic prediction results.

[0018] Furthermore, the calculation process of the multi-task distillation module is as follows:

[0019] G=σ(F2)

[0020] R=Conv(F1)G

[0021] Among them, R is the output result of the multi-task distillation module, F1 and F2 are one of the dense depth features and dense semantic features respectively, F1≠F2, when F1 is a dense depth feature, the calculated R is the dense depth prediction result, when F1 is a dense semantic feature, the calculated R is the dense semantic prediction result; σ is the activation function, G is the weighted probability between [0, 1], and Conv(.) represents convolution.

[0022] Furthermore, the semantically enhanced bird's-eye view feature generation method is:

[0023] LSS technology is used to back-project the concatenation result of content features and dense semantic prediction results to the camera coordinate system according to the depth weighted value, and then the coordinate system is transformed to the vehicle coordinate system to obtain the frustum pseudo point cloud;

[0024] The frustum pseudo point cloud is converted into voxel features and the dimension is reduced along the height to obtain the semantically enhanced bird's-eye view features.

[0025] Furthermore, the step (4) comprises:

[0026] (4.1) Downsampling and encoding the semantically enhanced bird's-eye view features step by step, and the downsampling encoding results of each level constitute a feature pyramid; the resolution of the encoded features in the feature pyramid is gradually reduced;

[0027] (4.2) The encoding layer composed of a deformable attention mechanism and a convolutional output head is used to process the encoded features in the feature pyramid step by step, and gradually restore them from low resolution to high resolution, and finally obtain the optimized bird's-eye view features with the same resolution as the semantically enhanced bird's-eye view features.

[0028] Furthermore, the step-by-step decoding process is as follows:

[0029]

[0030] in, is the encoding feature of the kth layer of the feature pyramid, K is the number of layers of the feature pyramid, DeformAttn is the deformable attention mechanism, is the feature weighted value of the k-th layer encoding feature in the feature pyramid, UpConv is the up-convolution layer, and UpSample is the nearest neighbor interpolation operation.

[0031] Furthermore, in the step (5), a convolutional neural network semantic output head is used to generate the distribution probability of the occupied grid categories with the optimized bird's-eye view features as input.

[0032] Furthermore, the calculation process of the target detection neural network in step (6) includes:

[0033] (6.1) Initialize the target query Q:

[0034]

[0035] Among them, q k is the kth initial detection box within the predefined perception range, N is the number of detection boxes, (x, y, z) is the geometric center of the detection box in three-dimensional space, (w, l, h) is the length, width and height of the detection box, raw is the deflection angle of the detection box from a bird's eye view, and Contfeat is a learnable embedding feature;

[0036] The sensing range is evenly divided into N squares, and the geometric center and geometric size of the square are used to initialize (x, y, w, l), and (z, h, raw) are initialized to 0;

[0037] (6.2) Using the cross-attention mechanism enhanced by the region of interest, the target feature F is extracted from the fused features in step (5) through the target query Q obj ={f0, f, ..., f N-1};

[0038] (6.3) Calculate the self-attention result of the target feature and update the target feature according to the self-attention result:

[0039]

[0040] Among them, query, key, and value represent the query vector, key vector, and value vector obtained by the target feature mapping, respectively. key Indicates the number of channels of the value vector, the superscript T indicates transpose, FFN indicates feedforward neural network, and softmax indicates softmax function;

[0041] (6.4) Use a multi-layer perceptron to predict the detection box of the updated target features;

[0042] (6.5) Use the updated target features as new embedded features and the detection box prediction results as the new initial detection box to generate a new target query Q. Repeat steps (6.2)-(6.4), use the updated target query as the input of the next layer of the network, and repeat the calculation process by stacking several layers of networks, and use the prediction results of the last layer as the final target recognition result.

[0043] Furthermore, the step (6.2) is specifically as follows:

[0044] For a single q k In terms of q, using the region of interest feature aggregation technology, k The features fused in step (5) are uniformly sampled within the corresponding grid range to obtain a fixed-length feature f k , traverse the query Q to get the target feature F obj :

[0045] F obj ={f0, f, ..., f N-1}

[0046] The feature f is extracted using the region of interest feature aggregation technique k When q k The corresponding range is scaled by fixed multiples m1, m2, and m3, and the embedded feature Contfeat is used to generate the attention weight α through the multi-layer perceptron MLP:

[0047] α=MLP(Contfeat)

[0048]

[0049] in, They represent the region of interest feature aggregation technology for q k Zoom in by m1, m2, or m3 times.

[0050] In a second aspect, the present invention proposes an autonomous driving visual multi-task perception system based on bird's-eye view features, which is used to implement the above-mentioned autonomous driving visual multi-task perception method based on bird's-eye view features.

[0051] Beneficial effects of the present invention:

[0052] The present invention designs a design method for a pure visual modality multi-task perception scheme for autonomous driving based on bird's-eye view features, covering multiple autonomous driving processes such as the generation, optimization, occupancy grid prediction, design and implementation of target detection heads, and serial multi-task collaboration of bird's-eye view features. In the generation and optimization stage of bird's-eye view features, the present invention can make full use of the dense color information of image data, and make the generated bird's-eye view features have stronger representation capabilities through semantic prediction. In the optimization stage, the redundant features introduced by monocular depth estimation are effectively reduced by predicting the weighted weights of the importance of different positions of bird's-eye view features. In the multi-task implementation stage, the present invention designs a new attention mechanism that can make full use of the intermediate results of the detection head to improve the final prediction accuracy of the model. The present invention uses serial collaboration to make full use of the synergy of occupancy grid prediction and detection, effectively reducing the training cost and the difficulty of model deployment, while achieving performance improvements for different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the multi-task perception method for autonomous driving based on bird's-eye view features proposed in the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings.

[0055] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0056] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0057] like Figure 1 The specific implementation steps of the present invention are as follows:

[0058] Step (1). The input of the autonomous driving system comes from data collected by multiple cameras installed at different positions on the vehicle body. The image features are extracted through a shared image feature extraction network, specifically:

[0059]

[0060] in, It is a backbone convolutional neural network shared by multiple images. The input data dimension can be recorded as W×H×3, the image resolution is W×H, Represents the image features of the i-th image taken by cameras at different positions, i = 1, 2, ..., N. The backbone convolutional neural network is used to uniformly extract features from the image and downsample the image to 1 / 16, that is, F cam The dimension is N×W / 16×H / 16×C. C is the channel dimension for extracting features, which can be set to 256 or 512 in this embodiment according to the specific situation. N is the number of cameras included in a group of surround view cameras, which can be set according to the specific configuration in this embodiment.

[0061] Step (2). Use the multi-task distillation module to collaboratively predict the dense depth features and dense semantic features of the downsampled resolution image, and further process the image features of step (1) to obtain the image content features, specifically:

[0062] Step (2.1). Use three prediction networks C with the same structure dep , C sem With C cont The image feature F obtained in step (1) cam Perform preliminary processing to obtain dense depth features F dep , dense semantic features F sem And content features F cont .

[0063] F dep =C dep (F cam )

[0064] F sem =C sem (F cam )

[0065] F cont =C cont (F cam )

[0066] In this embodiment, C dep , C sem With C cont It consists of two convolutional neural network blocks with residual connections.

[0067] Step (2.2). The dense deep features F are fused by the multi-task distillation module Mtd dep and dense semantic features F sem , and obtain a dense depth prediction result R that is consistent with the feature resolution dep And the dense semantic prediction result R sem .

[0068] R dep =Mtd(F dep , F sem )

[0069] R sem =Mtd(F sem , F dep )

[0070] Among them, the multi-task distillation module Mtd(F1, F2) generates feature weighted values ​​through the self-attention mechanism to achieve multi-task collaboration. The specific steps of Mtd(F1, F2) are as follows.

[0071] G=σ(F2)

[0072] R=Conv(F1)G

[0073] Among them, σ is a convolutional neural network with a channel number of 1, and the activation function is the sigmoid function to generate a weighted probability G in the range of [0, 1]; Conv(.) is the convolution function, and R is the prediction result R dep or R sem . Dense semantic prediction results R sem Represents the semantic classification probability of the image, and the classification category is consistent with the true value category of the occupied grid. Dense depth prediction result R dep It is a discrete probability distribution, which indicates the probability value of the current pixel depth value being in the discrete depth grid.

[0074] The present invention uses an additional prediction network C sem The multi-task distillation module Mtd can effectively avoid the single-task performance degradation caused by the imbalance between tasks due to the difference in the number of depth prediction and semantic prediction categories, and at the same time effectively utilize the synergy between the two tasks to improve the accuracy of depth prediction tasks and semantic prediction tasks.

[0075] Step (3). The content feature F in step (2) cont And the dense semantic prediction result R sem After splicing along the channel, fusion is performed using the dense depth prediction result R depAs the depth weighted value of the content feature, it is back-projected to the camera coordinate system through the LSS (Lift, Splat, Shoot) technology, and then converted to the vehicle coordinate system through the coordinate system transformation to obtain the view cone pseudo point cloud. Finally, the view cone pseudo point cloud projected in the vehicle coordinate system is converted into voxel features through the point cloud voxelization technology, and the semantically enhanced bird's-eye view feature is obtained by dimensionality reduction along the height, denoted as BEV, completing the conversion of 2D image features to 3D features.

[0076] Step (4). Optimize the encoder TE using the bird’s-eye view features with deformable attention mechanism ref Suppress the depth uncertainty noise introduced by the monocular depth estimation in step (2), reduce the redundancy of the bird's-eye view features obtained in step (3), highlight the useful features, and obtain the optimized bird's-eye view features; the bird's-eye view feature optimization encoder TE ref Using the Transformer neural network structure, the specific process is as follows:

[0077] Step (4.1). Encoder TE ref First, a multi-layer convolutional neural network is used to obtain a feature pyramid with gradually decreasing resolution.

[0078] In this embodiment, the bird's-eye view feature pyramid BEV pya The bird's-eye view features include original resolution, 1 / 2 down-sampled resolution, and 1 / 4 down-sampled resolution, that is, K=3, which can be flexibly configured according to specific circumstances in actual processes.

[0079] BEV pya =MConv(BEV)

[0080] Among them, BEV is a semantically enhanced bird's-eye view feature, MConv is a multi-layer convolutional neural network with gradual downsampling, and the feature pyramid BEV can be obtained by collecting the results of the intermediate convolutional layers. pya .

[0081] Step (4.2). Process from low resolution to high resolution step by step. Each level uses two encoding layers consisting of a deformable attention mechanism and a convolutional output head to predict the bird's-eye view occupancy probability (feature weight value) consistent with the input bird's-eye view feature resolution.

[0082]

[0083] in, is the encoding feature of the kth layer of the feature pyramid, DeformAttn is the deformable attention mechanism, is the weighted value of the feature of the kth layer, UpConv is the up-convolution layer, which increases the resolution of the feature, and UpSample is the nearest neighbor interpolation, which upsamples the low-resolution weighted value and weights it with the feature to update the feature. When k = K, That is, the features corresponding to the original resolution are not upsampled, but only weighted after the occupancy probability is predicted by the deformable attention encoding layer.

[0084] In addition, the module network can adaptively learn the bird's-eye view occupancy probability according to the data set, and can also use the task of occupying the grid to predict the occupancy probability of the bird's-eye view for supervised learning. This module can effectively reduce the redundancy caused by the inaccuracy of the depth prediction in the back projection process in step (3) by gradually predicting the bird's-eye view occupancy probability along the light direction, thereby improving the overall performance of the model.

[0085] Step (5). Using the convolutional neural network as the basic structure, design the occupancy grid prediction head C occ , using the bird's-eye view features optimized in step (4) as input, and using sub-pixel convolution technology, the optimized bird's-eye view features are upgraded to output the probability distribution results of the occupied grid categories.

[0086] Step (5.1).C occ is the semantic output head of the convolutional neural network, using the bird's-eye view feature Bev optimized in step (4) ref As input, the sub-pixel convolution technique is used to upgrade the bird's-eye view features and output the category probability distribution of each grid in the grid. occ Use the cross-entropy loss function (Cross-entropy loss) and the set similarity metric loss Dice-loss to supervise the training:

[0087] Occ=C occ (Bev ref )

[0088] In this embodiment, C occ is a convolutional neural network, Occ is the classification probability of the occupied grid category (occupied grid prediction result).

[0089] Step (5.2). The classification probability of the occupied grid category predicted by the above step (5.1) is the final result of the occupied grid prediction task. The foreground part is extracted from Occ, and the probability distribution of the foreground category of Occ is front With Bev ref Fusion, generating fused feature BEV fusion It is used for subsequent target detection to achieve serial multi-task collaboration. The feature fusion here can be achieved by conventional fusion methods, such as directly splicing Occ front and Bev ref.

[0090] Step (6). Use the target detection neural network T det For the fused feature BEV obtained in step (5.2) fusion Processing is performed to generate target detection prediction results. The specific calculation process is as follows:

[0091] Step (6.1). Initialize the target query (Object Query) Q:

[0092]

[0093] q k It is a series of initial detection frames set within the perception range predefined by the present invention. N is the number of detection frames, which can be set according to specific needs. (x, y, z) is the geometric center of the detection frame in three-dimensional space. (w, l, h) are the length, width and height of the detection frame. Raw is the deflection angle of the detection frame under a bird's-eye view (top view). Contfeat is a learnable embedding feature, initialized with learnable parameters. The perception range is evenly divided into N squares, and the geometric center and geometric size of the squares are used to initialize (x, y, w, l), and (z, h, raw) are initialized to 0. The perception method based on bird's-eye view features usually has a perception range of a rectangular area centered on the vehicle, and the specific value can be flexibly adjusted according to the configured sensors. Taking the perception range of ±50m and N=100 as an example, q k It actually represents a cylindrical area with a length and width of 10m.

[0094] Step (6.2). Use the cross-attention mechanism enhanced by the region of interest to obtain the fused feature BEV from step (5.2) through the target query Q fusion Extract the target feature F obj .

[0095] Specifically, for a single q k For example, using RoIAlign (Region of Interest Align) regional feature aggregation technology, we can k The fusion features are uniformly sampled within the corresponding grid range to obtain a fixed-length feature f k , traverse the query Q to get the target feature F obj :

[0096] F obj ={f0, f, ..., f N-1}

[0097] In a specific implementation of the present invention, RoI Align is used to extract f k When q kThe corresponding range is scaled by a fixed multiple (e.g. 1.2, 1.0, 0.8), and the attention weight α is generated by the embedded feature Contfeat through a multi-layer perceptron.

[0098] α=MLP(Contfeat)

[0099] f k =α[RoI Align 1.2 (q k ), RoI Align 1.0 (q k ), RoI Align 0.8 (q k )] T

[0100] The above operation of using the target query Q to obtain the target feature is the proposed cross-attention mechanism MsRoIAttn for region of interest enhancement. The process of extracting the target feature can be expressed as:

[0101] F obj =MsRoIAttn(BEV fusion , Q)

[0102] Step (6.3). Use the Transformer standard self-attention mechanism to process the target feature F obj , through the linear layer to transform f k Mapped into query vector (query), key vector (key), value vector (value) and update the target feature. The update process is expressed as:

[0103]

[0104] Among them, C key It represents the number of channels of the three vectors, and FFN represents the feed-forward neural network.

[0105] Step (6.4). Use a multi-layer perceptron to update the target feature F obj Predict the detection boxes separately, specifically:

[0106] Δx, Δy,z,w,l,h,raw,p class =MLP(f k )

[0107] x,y=x+Δx,y+Δy

[0108] Among them, Δ indicates that the parameter is predicted as an offset, p class is the classification result of the target.

[0109] Step (6.5). The above process is the target detection neural network T detThe calculation process of a layer of network is to transform the target feature F obj As the new embedding feature, the detection box prediction result is used as the new initial detection box to generate a new target query Q. Repeat steps (6.2)-(6.4), and use the updated target query as the input of the next layer of the network. By stacking three or six layers of networks, the above calculation process is continuously repeated to achieve a coarse-to-fine prediction paradigm, and the prediction result of the last layer is used as the final target recognition result.

[0110] The present invention also provides an autonomous driving visual multi-task perception system based on bird's-eye view features, which is used to implement the above embodiments. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible.

[0111] This embodiment provides an autonomous driving visual multi-task perception system based on bird's-eye view features, including:

[0112] An image feature extraction module, which is used to obtain a set of images from the surround view camera of the autonomous driving system and extract image features;

[0113] A dense depth, semantic, and content prediction module is used to collaboratively predict dense depth features, dense semantic features, and content features using a multi-task distillation module based on the image features extracted by the image feature extraction module, and to generate dense depth prediction results and dense semantic prediction results by combining the dense depth features and dense semantic features;

[0114] A bird's-eye view feature generation module is used to stitch content features and dense semantic prediction results, and use dense depth prediction results as depth weighted values ​​of content features to generate semantically enhanced bird's-eye view features;

[0115] A bird's-eye view feature optimization module for optimizing the bird's-eye view features using a bird's-eye view feature optimization encoder with a deformable attention mechanism;

[0116] The occupied grid prediction task module is used to predict the occupied grid category distribution probability based on the optimized bird's-eye view features, extract the probability distribution of the foreground category from it, and fuse the probability distribution of the foreground category with the optimized bird's-eye view features;

[0117] The target detection task module is used to use the target detection neural network to process the features fused by the occupied grid prediction task module to generate target detection prediction results.

[0118] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. For example, the regression enhancement module includes:

[0119] An encoder module for encoding real process data and virtual process data;

[0120] A pre-trained multi-head attention module, which is used to extract features of the encoded real process data and virtual process data;

[0121] The linear prediction layer module is used to predict the virtual measurement data according to the characteristics of the encoded real process data and the virtual process data.

[0122] The implementation methods of the remaining modules will not be described in detail here. The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Those of ordinary skill in the art may understand and implement it without creative work.

[0123] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0124] Obviously, the above-described embodiments and drawings are only some examples of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that, although the work done in this development process may be complicated and lengthy, for those of ordinary skill in the art, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application. Without departing from the concept of the present application, several variations and improvements can also be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A multi-task perception method for autonomous driving vision based on bird's-eye view features, characterized in that: The following steps are involved: (1) Obtain a set of images from the surround view camera of the autonomous driving system and extract image features; (2) Based on the image features obtained in step (1), a multi-task distillation module is used to collaboratively predict dense depth features, dense semantic features, and content features, and dense depth features and dense semantic features are combined to generate dense depth prediction results and dense semantic prediction results; (3) Concatenate content features and dense semantic prediction results, use dense depth prediction results as the depth weighted value of content features, and generate semantically enhanced bird's-eye view features; (4) Optimizing the bird’s-eye view features using a bird’s-eye view feature optimization encoder with a deformable attention mechanism; (5) Predict the distribution probability of the occupied grid categories based on the optimized bird's-eye view features, extract the probability distribution of the foreground category, and fuse the probability distribution of the foreground category with the optimized bird's-eye view features; (6) Use the target detection neural network to process the features fused in step (5) to generate target detection prediction results.

2. The multi-task perception method for autonomous driving based on bird's-eye view features according to claim 1 is characterized in that: The step (2) comprises: (2.1) using three networks with the same structure to preliminarily process the image features to obtain dense depth features, dense semantic features and content features; the parameters of the three networks with the same structure are not shared; (2.2) A multi-task distillation module is used to fuse dense depth features and dense semantic features to obtain dense depth prediction results and dense semantic prediction results.

3. The multi-task perception method for autonomous driving based on bird's-eye view features according to claim 2 is characterized in that: The calculation process of the multi-task distillation module is as follows: G=σ(F2) R=Conv(F1)G Among them, R is the output result of the multi-task distillation module, F1 and F2 are one of the dense depth features and dense semantic features respectively, F1≠F2, when F1 is a dense depth feature, the calculated R is the dense depth prediction result, when F1 is a dense semantic feature, the calculated R is the dense semantic prediction result; σ is the activation function, G is the weighted probability between [0,1], and Conv(.) represents convolution.

4. The multi-task perception method for autonomous driving based on bird's-eye view features according to claim 1 is characterized in that: The semantically enhanced bird's-eye view feature generation method is: LSS technology is used to back-project the concatenation result of content features and dense semantic prediction results to the camera coordinate system according to the depth weighted value, and then the coordinate system is transformed to the vehicle coordinate system to obtain the frustum pseudo point cloud; The frustum pseudo point cloud is converted into voxel features and the dimension is reduced along the height to obtain the semantically enhanced bird's-eye view features.

5. The multi-task perception method for autonomous driving based on bird's-eye view features in pure visual mode according to claim 1, characterized in that: The step (4) comprises: (4.1) Downsampling and encoding the semantically enhanced bird's-eye view features step by step, and the downsampling encoding results of each level constitute a feature pyramid; the resolution of the encoded features in the feature pyramid is gradually reduced; (4.2) The encoding layer composed of a deformable attention mechanism and a convolutional output head is used to process the encoded features in the feature pyramid step by step, and gradually restore them from low resolution to high resolution, and finally obtain the optimized bird's-eye view features with the same resolution as the semantically enhanced bird's-eye view features.

6. The multi-task perception method for autonomous driving based on bird's-eye view features according to claim 5 is characterized in that: The step-by-step decoding process is as follows: in, is the encoding feature of the kth layer of the feature pyramid, K is the number of layers of the feature pyramid, DeformAttn is the deformable attention mechanism, is the feature weighted value of the k-th layer encoding feature in the feature pyramid, UpConv is the up-convolution layer, and UpSample is the nearest neighbor interpolation operation.

7. The multi-task perception method for autonomous driving based on bird's-eye view features in pure visual mode according to claim 1, characterized in that: In the step (5), a convolutional neural network semantic output head is used to generate the probability of occupied grid category distribution with the optimized bird's-eye view features as input.

8. The multi-task perception method for autonomous driving based on bird's-eye view features according to claim 1, characterized in that: The calculation process of the target detection neural network in step (6) includes: (6.1) Initialize the target query Q: Among them, q k is the kth initial detection box within the predefined perception range, N is the number of detection boxes, (x, y, z) is the geometric center of the detection box in three-dimensional space, (w, l, h) is the length, width and height of the detection box, raw is the deflection angle of the detection box from a bird's eye view, and Contfeat is a learnable embedding feature; The sensing range is evenly divided into N squares, and the geometric center and geometric size of the square are used to initialize (x, y, w, l), and (z, h, raw) are initialized to 0; (6.2) Using the cross-attention mechanism enhanced by the region of interest, the target feature F is extracted from the fused features in step (5) through the target query Q obj ={f0, f, ..., f N-1 ); (6.3) Calculate the self-attention result of the target feature and update the target feature according to the self-attention result: Among them, query, key, and value represent the query vector, key vector, and value vector obtained by the target feature mapping, respectively. key Indicates the number of channels of the value vector, the superscript T indicates transpose, FFN indicates feedforward neural network, and softmax indicates softmax function; (6.4) Use a multi-layer perceptron to predict the detection box of the updated target features; (6.5) Use the updated target features as new embedded features and the detection box prediction results as the new initial detection box to generate a new target query Q. Repeat steps (6.2)-(6.4), use the updated target query as the input of the next layer of the network, and repeat the calculation process by stacking several layers of networks, and use the prediction results of the last layer as the final target recognition result.

9. The multi-task perception method for autonomous driving based on bird's-eye view features according to claim 8 is characterized in that: The step (6.2) is specifically: For a single q k In terms of q, using the region of interest feature aggregation technology, k The features fused in step (5) are uniformly sampled within the corresponding grid range to obtain a fixed-length feature f k , traverse the query Q to get the target feature F obj : F obj ={f0,f,...,f N-1 } The feature f is extracted using the region of interest feature aggregation technique k When q k The corresponding range is scaled by fixed multiples m1, m2, and m3, and the embedded feature Contfeat is used to generate the attention weight α through the multi-layer perceptron MLP: α=MLP(Contfeat) in, They represent the region of interest feature aggregation technology for q k Zoom in by m1, m2, or m3 times.

10. An autonomous driving visual multi-task perception system based on bird's-eye view features, used to implement the method of claim 1, characterized in that: The system comprises: An image feature extraction module, which is used to obtain a set of images from the surround view camera of the autonomous driving system and extract image features; A dense depth, semantic, and content prediction module is used to collaboratively predict dense depth features, dense semantic features, and content features using a multi-task distillation module based on the image features extracted by the image feature extraction module, and to generate dense depth prediction results and dense semantic prediction results by combining the dense depth features and dense semantic features; A bird's-eye view feature generation module is used to stitch content features and dense semantic prediction results, and use dense depth prediction results as depth weighted values ​​of content features to generate semantically enhanced bird's-eye view features; A bird's-eye view feature optimization module for optimizing the bird's-eye view features using a bird's-eye view feature optimization encoder with a deformable attention mechanism; The occupied grid prediction task module is used to predict the occupied grid category distribution probability based on the optimized bird's-eye view features, extract the probability distribution of the foreground category from it, and fuse the probability distribution of the foreground category with the optimized bird's-eye view features; The target detection task module is used to use the target detection neural network to process the features fused by the occupied grid prediction task module to generate target detection prediction results.

Images