An automatic driving method of fusing video images and point clouds

Abstract

The application discloses an automatic driving method for fusing video images and point clouds, and comprises the following steps: acquiring point cloud modal data collected by a millimeter wave radar and image modal data collected by a camera; inputting the point cloud modal data and the image modal data into corresponding voxel feature generation networks respectively to generate point cloud modal voxel features and image modal voxel features; and obtaining fused voxel features after preprocessing the point cloud modal voxel features and the image modal voxel features. The unified coordinate voxel features are efficiently fused in the form of generation, so that the fused features have the density of images, better small target detection capability, and accurate speed and depth information of the millimeter wave radar. Meanwhile, the correlation between perception, prediction and planning tasks is fully utilized, the training cost and the deployment difficulty of the model are reduced, and the performance of different tasks is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to an automatic driving method fusing video images and point clouds. BACKGROUND

[0002] The autonomous driving technology is a hot technology that is highly concerned in the current technology and industry fields and even the national strategic level. A complete autonomous driving system is usually composed of three major modules of perception, decision planning and control execution. Among them, the main purpose of the perception task is to use the data obtained by a series of vehicle-mounted sensors (such as radar, image, etc.) to generate positioning and classification prediction results for the entire autonomous driving scene and key targets in the scene, so as to help the autonomous driving vehicle to "feel" and "understand" the surrounding environment. Perception is the first step of autonomous driving, and accurate and efficient perception is an important prerequisite and guarantee for the safe and stable operation of the autonomous driving system. According to the specific application, the autonomous driving perception task can be further divided into sub-tasks such as 3D object tracking, 3D space occupancy prediction, online map generation, etc. A high-performance autonomous driving perception model is highly dependent on a laser radar device and uses a series of different models to complete different perception tasks, so the current autonomous driving solution has a series of problems such as high cost, inconvenience for deployment, limited flexibility and scalability, etc.

[0003] The perception sensor technology involved in the present application includes a millimeter wave radar and a camera, and the millimeter wave radar is an automatic driving sensor that has gradually emerged in recent years. Compared with a laser radar, the millimeter wave radar has a lower cost, can obtain the speed information of a target directly while obtaining the depth and shape information of a road target, and can work more stably than the laser radar under weather conditions such as rain, snow, fog, haze and smoke. However, the data obtained by the millimeter wave radar is more sparse than that obtained by the laser radar. The camera is one of the most common vehicle-mounted sensors, and the camera can obtain dense information of the surrounding environment, but it is difficult for a single camera to accurately perceive the three-dimensional spatial position information due to the loss of the depth information of a target in the projection process.

[0004] The camera and the millimeter wave radar data have advantages and disadvantages, and the dense pixel color information of the image data and the accurate depth and speed information of the millimeter wave radar can generate perception features that are dense and accurate in position in the three-dimensional space. However, the great difference between the data of different modalities also brings great difficulties to data fusion.

[0005] Currently, mainstream automatic driving technologies usually use multiple independent deep learning models for different tasks. This technical solution has at least the following three defects: 1) the data feature extraction is a deep learning network structure shared by each task, and the feature extraction network is one of the basic structures of modern neural network models. Training different feature extraction networks for multiple independent models will increase the training cost; 2) there is a certain correlation between perception, prediction and planning tasks, and it is difficult for several independent models to use this correlation to improve the accuracy of their own tasks; 3) multiple independent models increase unnecessary redundancy of deep learning models, and increase the actual deployment cost of the algorithm. SUMMARY

[0006] To solve the technical problems in the background art, the present application provides an automatic driving method fusing video images and point clouds.

[0007] The automatic driving method fusing video images and point clouds provided by the present application comprises the following steps:

[0008] S1, acquiring point cloud modal data collected by a millimeter wave radar and image modal data collected by a camera;

[0009] S2, inputting the point cloud modal data and the image modal data into corresponding voxel feature generation networks respectively to generate point cloud modal voxel features and image modal voxel features;

[0010] S3, preprocessing the point cloud modal voxel features and the image modal voxel features to obtain fused voxel features;

[0011] S4, acquiring a perception task, inputting the fused voxel features into a perception output head to complete output of the perception task, and obtaining perception-related Key and Value;

[0012] S5, inputting the perception-related Key and Value and the fused voxel features into a voxel feature sampler to obtain first-type Key and Value related to a voxel environment;

[0013] S6, inputting the first-type Key and Value and the perception-related Key and Value into a pre-established and trained Transformer model of automatic driving to complete tasks of predicting a target action of a road surface of an automatic driving vehicle and planning a driving behavior.

[0014] Preferably, the preprocessing specifically comprises:

[0015] using the image modal voxel features as Query for feature fusion, denoted as Q cam ;

[0016] generating Key and Value for feature fusion from the point cloud modal voxel features, denoted as Kradar ,V radar ;

[0017] updating the Q cam using a self-attention mechanism;

[0018] updating the Q radar ,V radar using a cross-attention mechanism to generate K cam from the voxel features of the point cloud modal, obtaining fused voxel features Q fuse , and then using a multi-layer perceptron to generate fused voxel features G from the fused voxel features Q fuse .

[0019] Preferably, “S5” specifically comprises:

[0020] inputting the perception-related Key and Value and the fused voxel features into a voxel feature sampler;

[0021] the voxel feature sampler performs voxel sampling on the fused voxel features using the perception-related Key and Value to sample out sparse and noteworthy voxels;

[0022] using the sampled voxels to construct a first type of Key and Value related to the voxel environment to provide voxel feature information for subsequent tasks.

[0023] Preferably, the Transformer model comprises a road target action prediction neural network and a driving behavior planning neural network; “S6” specifically comprises:

[0024] inputting the first type of Key and Value and the perception-related Key and Value into the road target action prediction neural network to complete the task of road target action prediction and simultaneously obtain a second type of Key and Value related to action prediction;

[0025] inputting the first type of Key and Value and the second type of Key and Value into the driving behavior planning neural network to complete the task of driving behavior planning of the autonomous vehicle.

[0026] Preferably, the task of road target action prediction and the task of driving behavior planning of the autonomous vehicle are output via one-to-one output heads.

[0027] Preferably, the perception task comprises but is not limited to three-dimensional target tracking, three-dimensional space occupation prediction, and online map generation.

[0028] Preferably, the driving behavior planning task includes but is not limited to acceleration, deceleration, parking, straight driving, turning, reversing, parking, and overtaking.

[0029] In the present application, the proposed automatic driving method fusing video images and point clouds adopts dual-modal data fusion. In the dual-modal data fusion stage, sensor data in two different data forms of camera and millimeter wave radar can be processed, and they are efficiently fused by generating unified coordinate voxel features, so that the fused features have the density of images and better small target detection capability, and can use the accurate speed and depth information of millimeter wave radar. In the multi-task implementation stage, the unified voxel features are used to complete the multi-stage tasks of perception, prediction and planning, and the correlation between perception, prediction and planning tasks is fully utilized to reduce the training cost and the difficulty of model deployment, while improving the performance of different tasks. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 A structure diagram of the workflow of the automatic driving method fusing video images and point clouds proposed in the present application;

[0031] Figure 2 A structure diagram of the system flow of the automatic driving method fusing video images and point clouds proposed in the present application. DETAILED DESCRIPTION

[0032] REFERENCE Figure 1 and Figure 2 The automatic driving method fusing video images and point clouds proposed in the present application comprises the following steps:

[0033] S1, acquiring point cloud modal data collected by a millimeter wave radar and image modal data collected by a camera.

[0034] In the present embodiment, the modal data in the automatic driving application scenario is collected by a millimeter wave radar and a camera. The point cloud modal data collected by the millimeter wave radar is denoted as P radar , and the image modal data collected by the camera is denoted as I cam .

[0035] S2, inputting the point cloud modal data and the image modal data into corresponding voxel feature generation networks to generate point cloud modal voxel features and image modal voxel features.

[0036] In the present embodiment, the input image modal data I cam is input into an image-based voxel generation network to generate image modal voxel features F cam .

[0037]

[0038] The input point cloud modal data is denoted as P. radar After voxel generation network based on millimeter-wave radar point cloud Voxel features F generated from millimeter-wave radar point clouds radar :

[0039]

[0040] Unified generation of voxel features F cam and F radar The feature dimension C and resolution D×W×H, and F cam and F radar Maintain consistent coordinates. The resolution can be adjusted by the specific sensor configuration; for example, a resolution of 400×400×32 can be mapped to 80m×80m×6.4m in the X, Y, and Z directions of real 3D space.

[0041] S3. After preprocessing the point cloud modal voxel features and the image modal voxel features, the fused voxel features are obtained.

[0042] In this embodiment, point cloud modal voxel features and image modal voxel features are input into a voxel feature fusion network to complete the preprocessing operation.

[0043] Preprocessing specifically includes:

[0044] Image modal voxel features are used as the Query for feature fusion, denoted as Q. cam ;

[0045] Key and Value, denoted as K, are generated from the voxel features of the point cloud modalities for feature fusion. radar V radar ;

[0046] Using the self-attention mechanism, Q cam Update;

[0047] K is generated from the voxel features of point cloud modalities using the cross-attention mechanism. radar V radar Update Q cam The fused voxel feature Q was obtained. fuse Then, using a multilayer perceptron The fused voxel feature Q fuse Generate fusion voxel feature G.

[0048] S4. Obtain the perception task by inputting the fused voxel features into the perception output head to complete the output of the perception task and obtain the perception-related Key and Value.

[0049] In the embodiment, the fused voxel feature G is taken as a basic processing feature of a subsequent task, and the perception output head can implement automatic driving perception tasks including but not limited to three-dimensional target tracking, three-dimensional space occupation prediction and online map generation. The multi-type perception output result is denoted as O pcep The Key and the Value are also used to construct the Key and the Value related to the multi-type perception, which are denoted as K pcep and V pcep , respectively, to provide perception information for the subsequent task.

[0050] S5, inputting the Key and the Value related to the perception and the fused voxel feature into a voxel feature sampler to obtain a first type of Key and Value related to a voxel environment.

[0051] “S5” specifically includes:

[0052] inputting the Key and the Value related to the perception and the fused voxel feature into the voxel feature sampler;

[0053] The voxel feature sampler voxel samples the fused voxel feature G using the Key and the Value related to the perception to sample out sparse and noteworthy voxels.

[0054] The sampled voxels are used to construct a first type of Key and Value related to the voxel environment to provide voxel feature information for the subsequent task.

[0055] In the embodiment, the voxel feature sampler voxel samples the fused voxel feature G using the Key and the Value related to the perception to sample out sparse and noteworthy voxels. The sampled voxels are used to construct a first type of Key and Value related to the voxel environment, which are denoted as K voxel and V voxel , respectively, to provide voxel feature information for the subsequent task.

[0056] S6, inputting the first type of Key and Value and the Key and Value related to the perception into a pre-established and trained Transformer model of an automatic driving vehicle to complete the tasks of road target action prediction and driving behavior planning of the automatic driving vehicle.

[0057] The Transformer model includes a road target action prediction neural network and a driving behavior planning neural network. “S6” specifically includes:

[0058] inputting the first type of Key and Value and the Key and Value related to the perception into the road target action prediction neural network to complete the task of road target action prediction and obtain a second type of Key and Value related to action prediction;

[0059] Input the first type of Key and Value, and the second type of Key and Value into the driving behavior planning neural network to complete the task of planning the driving behavior of autonomous vehicles.

[0060] Driving behavior planning tasks include, but are not limited to, accelerating, decelerating, stopping, going straight, turning, reversing, parking, and overtaking.

[0061] In this embodiment, the road surface target motion prediction neural network is denoted as T. forec Driving behavior planning neural network is denoted as Trans plan Road surface target motion prediction neural network T forec Including T forec1 T forec2 ; Driving behavior planning neural network Trans plan Including Trans plan1 Trans plan2 ;

[0062] Road Target Action Prediction Neural Network T forec Using K related to multi-type perception pcep With V pcep K related to voxel environment voxel With V voxel Action prediction Query (denoted as Q) was performed successively. forec The algorithm learns and updates, using the updated action prediction query to complete the task of predicting road target actions. It also serves as the key and value related to action prediction, denoted as K. forec With V forec The details are as follows:

[0063] Step 1: via T forec1 Using K related to multi-type perception pcep With V pcep Q forec The learning and updating process is based on the attention calculation method of the Transformer structure, as follows:

[0064] Q forec =T forec1 (Q forec ,K pcep V pcep )

[0065] The Transformer includes the following calculations:

[0066]

[0067] Q forec =Q forec +FFN(Q forec )

[0068] wherein, The correlation of both is calculated, and the correlation is converted into Value weighting coefficients by the sigma function; the sigma function can be implemented by the softmax function; FFN is a feedforward neural network; C forec is the feature dimension of Q forec for normalizing the correlation variance.

[0069] Step 2: Update Q forec2 by using the voxel environment-related K voxel and V voxel . forec The process is based on the calculation mode of the Transformer, as follows:

[0070] Q forec = T forec2 (Q forec , K voxel , V voxel ).

[0071] Step 3: The updated Q forec after Step 2 above is the final road target action prediction feature, and Q forec is converted into the action prediction result O forec of the road target by the action prediction output head:

[0072] O forec = H forec (Q forec ).

[0073] Step 4: The updated Q forec after Step 2 above is further used as the Key and Value related to the action prediction, denoted as K forec and V forec .

[0074] The driving behavior planning neural network Trans plan uses the K forec and V forec related to the action prediction and the K voxel and V voxel related to the voxel environment to learn and update the driving behavior planning Query (denoted as Q plan ), and the updated driving behavior planning Query will be used to complete the corresponding automatic driving vehicle driving behavior planning task, as follows:

[0075] Step 11: The driving behavior planning neural network Trans plan uses the K forec and V forec related to the action prediction and the Kpcep With V pcep Update driving behavior planning Query (denoted as Q plan ), the process is based on the calculation method of Transformer:

[0076] Q plan = Trans plan1 (Q plan , K voxel , V voxel )

[0077] Q plan = Trans plan2 (Q plan , K forec , V forec ).

[0078] Step 12: Q plan updated after step 11 is the final driving behavior planning feature, through the driving behavior planning output head H plan , Q plan is converted into the driving behavior planning prediction result O plan of the autonomous vehicle:

[0079] O plan = H plan (Q plan )

[0080] Wherein, O plan includes but is not limited to acceleration, deceleration, parking, straight, turning, reversing, parking, overtaking and other specific driving behaviors.

[0081] Specifically, as shown in Figure 2 , it also includes: the task of automatic driving vehicle road target action prediction and the task of driving behavior planning are output via one-to-one corresponding output head.

[0082] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art in the technical range disclosed by the present application, according to the technical scheme and the invention concept of the present application, equivalent replacement or change, should be covered in the protection scope of the present application.

Claims

1. An autonomous driving method that integrates video images and point clouds, characterized in that, Includes the following steps: S1. Acquire point cloud modal data collected by millimeter-wave radar and image modal data collected by camera; S2. Input the point cloud modal data and image modal data into the corresponding voxel feature generation network to generate point cloud modal voxel features and image modal voxel features respectively; S3. After preprocessing the point cloud modal voxel features and the image modal voxel features, the fused voxel features are obtained. S4. Obtain the perception task, input the fused voxel features into the perception output head to complete the perception task output, and obtain the perception-related Key and Value; S5. Input the perception-related Key and Value, and the fused voxel features into the voxel feature sampler to obtain the first type of Key and Value related to the voxel environment; Step S5 specifically includes: Input the perception-related Key and Value, and the fused voxel features into the voxel feature sampler; The voxel feature sampler uses perceptually related Key and Value to sample fused voxel features, extracting sparse and noteworthy voxels. The sampled voxels are used to construct a first-type Key and Value related to the voxel environment, providing voxel feature information for subsequent tasks; S6. Input the first type of Key and Value, and the perception-related Key and Value into the pre-established and trained autonomous driving Transformer model to complete the task of predicting the road target action of the autonomous vehicle and planning the driving behavior. The Transformer model includes: a road target action prediction neural network and a driving behavior planning neural network; step S6 specifically includes: The first type of Key and Value, and the perception-related Key and Value are input into the road target action prediction neural network to complete the task of road target action prediction, and at the same time obtain the second type of Key and Value related to action prediction. Input the first type of Key and Value, and the second type of Key and Value into the driving behavior planning neural network to complete the task of planning the driving behavior of autonomous vehicles.

2. The autonomous driving method based on the fusion of video images and point clouds according to claim 1, characterized in that, The preprocessing specifically includes: Image modal voxel features are used as the query for feature fusion, denoted as ; Key and Value are generated from point cloud modal voxel features for feature fusion, denoted as [key, value, and denot]. ; Using the self-attention mechanism, for Update; Generating voxel features from point cloud modalities using the cross-attention mechanism renew The voxel characteristics of the fusion were obtained. Then, using a multilayer perceptron The fused voxel features Generate fusion voxel features .

3. The autonomous driving method based on the fusion of video images and point clouds according to claim 1, characterized in that, Also includes: The tasks of predicting road target actions and planning driving behavior for the autonomous vehicle are output through corresponding output heads.

4. The autonomous driving method based on the fusion of video images and point clouds according to claim 1, characterized in that, The perception tasks include, but are not limited to, 3D target tracking, 3D space occupancy prediction, and online map generation.

5. The autonomous driving method based on the fusion of video images and point clouds according to claim 1, characterized in that, The driving behavior planning tasks include, but are not limited to, accelerating, decelerating, stopping, going straight, turning, reversing, parking, and overtaking.

Images