End-to-end automatic driving method and system based on autoregression synchronous prediction and planning

Abstract

The invention discloses an end-to-end automatic driving method and system based on autoregressive synchronous prediction and planning, and the method comprises the steps: S1, data input: inputting multi-modal data and an own vehicle state, the multi-modal data comprising a point cloud and a multi-view image; s2, feature coding: obtaining a point cloud feature F1 and a multi-view image feature Fi under a BEV view angle from the point cloud and the multi-view image through a point cloud encoder and an image encoder respectively, and coding a self-vehicle state through a self-vehicle encoder to obtain a query feature Qe of the self-vehicle state; s3, environment perception: obtaining a target state Sa and a target feature Qa, a map state Sm and a map feature Qm, a traffic light state St and a traffic light feature Qt through 3D obstacle detection, map detection and traffic light detection, and obtaining a prediction result Sc of an Occupancy state through Occupancy detection of a static scene; and S4, synchronous prediction and planning: disassembling the own vehicle state and the target state at N moments in the future into the state of predicting the next moment by using the states of the past and current moments and the environment information through an autoregression mode, and repeating in sequence to obtain the states of all N moments.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous driving method and system based on autoregressive synchronous prediction and planning. Background Art

[0002] In the technical fields of autonomous driving, robots and other intelligent agents, especially in the field of autonomous driving, the entire technology stack of autonomous driving is generally broken down into sub-modules such as environmental perception, tracking and post-fusion, prediction and planning for separate processing. Recently, the academic community has set off a wave of end-to-end autonomous driving technology research, and the more representative ones are UniAD and VAD. Compared with the early multi-task streaming processing method, the outstanding advantages of end-to-end autonomous driving technology are: 1. Simplify unnecessary links. For example, tracking and post-fusion, even all intermediate links can be discarded; 2. Greatly simplify the deployment process, and a single model can solve it; 3. Data-driven AI has a higher upper limit and can greatly simplify the amount of regularized code.

[0003] In recent years, representative solutions in the field of end-to-end autonomous driving include UniAD and VAD. Their proposal has broken people's doubts about end-to-end autonomous driving because they have a certain degree of explainability in terms of structural design. UniAD is the first end-to-end autonomous driving solution that has been proposed and widely used. It is characterized by complete sub-functional modules such as environment perception, tracking, prediction, occupancy, and planning. At the same time, it cleverly uses transformers to effectively connect various modules to achieve end-to-end autonomous driving. However, UniAD also has obvious disadvantages, that is, the model structure is too complex, which brings great challenges to model training and deployment. VAD uses a vectorization method to vectorize all intermediate tasks in order to increase computing efficiency and save video memory. Summary of the invention

[0004] In response to the problems in the prior art, the present invention aims to further simplify the complexity of end-to-end autonomous driving based on an autoregressive synchronous prediction and planning method while improving the performance of end-to-end autonomous driving technology.

[0005] In order to achieve the above-mentioned object, the technical solution of the present invention provides an end-to-end autonomous driving method based on autoregressive synchronous prediction and planning, which comprises the following steps: S1 data input: input multimodal data and vehicle status, wherein the multimodal data comprises point cloud and multi-view image; S2 feature encoding: the point cloud and multi-view image are respectively encoded by a point cloud encoder and an image encoder to obtain the point cloud feature F under the BEV perspective; l and multi-view image features F i , and encode the vehicle state through the vehicle encoder to obtain the query feature Q of the vehicle state e; S3 environmental perception: obtain the target state S through 3D obstacle detection, map detection and traffic light detection a and the target feature Q a , Map Status S m and map feature Q m , Traffic light status S t and traffic light feature Q t And obtain the prediction result S of the occupancy state through occupancy detection of static scenes c ; S4 Synchronous prediction and planning: Through the autoregressive method, the vehicle state and target state at the next N moments are decomposed into the state at the next moment predicted by using the state and environmental information at the past and current moments, and the state of all N moments is obtained in turn.

[0006] Furthermore, in step S3, occupancy detection specifically includes: firstly, the point cloud feature F l and a learnable global query Q g After adding, self-attention is performed first; then Deformer Attention is used to extract the image features F i The fused BEV features are obtained by querying; then the above BEV features are merged with the historical BEV features, and then the Occupancy feature Q is obtained by passing it through the BEV encoder. c ; Finally, the dimension is upgraded to obtain the final prediction result S of the occupancy state c .

[0007] Furthermore, in step S3, a query-based sparse 3D object detection paradigm is used to implement 3D obstacle detection, map detection, and traffic light detection.

[0008] Further, step S4 specifically includes: S41: the vehicle feature Q at time T e and the target feature Q a After merging, we get query Q; S42: The vehicle state S at time T e and the target state S a After merging, the position encoder is used to obtain the position encoding feature Q pos ; S43: Change the traffic light status S t , Occupancy status S c , Map Status S m After merging, the environment state features E are obtained by passing through the state encoder and position encoder respectively. se and K pos ; S45: Set the traffic light feature Q t , Occupancy feature Q c 、Map FeaturesQ mAfter merging, the environmental state feature E se After adding, we get features K and V; S46: add Q, Q pos , K, V and K pos They are sent together into the transformer to get the state at the next moment, that is, the state at moment T+1.

[0009] Furthermore, the vehicle state includes the size of the vehicle and the current position and speed, the map state includes the coordinates and categories of the vector points, and the occupancy state includes whether each voxel is occupied and the category information.

[0010] The technical solution of the present invention also provides an end-to-end autonomous driving system based on autoregressive synchronous prediction and planning, which includes the following modules: a data input module, used to input multimodal data and self-vehicle status, wherein the multimodal data includes point cloud and multi-view image; a feature encoding module, used to obtain point cloud features F under the BEV perspective from the point cloud and multi-view image through a point cloud encoder and an image encoder respectively. l and multi-view image features F i , and encode the vehicle state through the vehicle encoder to obtain the query feature Q of the vehicle state e ; Environmental perception module, used to obtain the target state S through 3D obstacle detection, map detection and traffic light detection a and the target feature Q a , Map Status S m and map feature Q m , Traffic light status S t and traffic light feature Q t And obtain the prediction result S of the occupancy state through occupancy detection of static scenes c The synchronous prediction and planning module is used to decompose the vehicle state and target state at the next N moments through autoregression, using the state and environment information at the past and current moments to predict the state at the next moment, and then repeat this process to obtain the states at all N moments.

[0011] Furthermore, the environment perception module implements occupancy detection according to the following steps: first, the point cloud feature F l and a learnable global query Q g After adding, self-attention is performed first; then Deformer Attention is used to extract the image features F i The fused BEV features are obtained by querying; then the above BEV features are merged with the historical BEV features, and then the Occupancy feature Q is obtained by passing it through the BEV encoder. c ; Finally, the dimension is upgraded to obtain the final prediction result S of the occupancy state c .

[0012] Furthermore, the environment perception module adopts a query-based sparse 3D target detection paradigm to achieve 3D obstacle detection, map detection and traffic light detection.

[0013] Furthermore, the synchronous prediction and planning module performs the following steps: the vehicle feature Q at time T e and the target feature Q a After merging, we get query Q; the state of the vehicle at time T S e and the target state S a After merging, the position encoder is used to obtain the position encoding feature Q pos ; Change the traffic light status S t , Occupancy status S c , Map Status S m After merging, the environment state features E are obtained by passing through the state encoder and position encoder respectively. se and K pos ; Set the traffic light feature Q t 、Occupancy feature Q c 、Map FeaturesQ m After merging, the environmental state feature E se After adding, we get the features K and V; pos , K, V and K pos They are sent together into the transformer to get the state at the next moment, that is, the state at moment T+1.

[0014] The technical solution of the present invention also provides a computer-readable storage medium containing a computer program, and when the computer program is executed by one or more processors, the method as described above is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 It is a schematic diagram of the overall solution architecture of the present invention;

[0017] Figure 2 It is an architectural diagram of the synchronous prediction and planning module of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The present invention focuses on the autoregressive synchronous prediction and planning technology in the proposed end-to-end autonomous driving solution. Figure 1 .like Figure 1 As shown, the present invention includes input (multimodal including but not limited to multi-view images, point clouds, etc.), a feature encoder module (Encoder), an environment perception module (Environment Perception) (including but not limited to 3D target detection, map detection, occupancy of static scenes and traffic light detection), and a synchronous prediction & planning module (Prediction & Planning).

[0020] The specific technical route of the present invention is as follows:

[0021] (1) Input: The input data of the present invention includes multi-modal input (including point cloud and multi-view image as input, respectively, l and I i The vehicle state S e (Including the size of the vehicle, current posture and speed, the past state can also be used as input. For the convenience of description, this article only takes the current state as an example).

[0022] (2) Feature encoder: including point cloud encoder E l (can be any point cloud feature extractor, such as Pointnet, sparse convolutional network, etc.), image encoder E i (generally using Resnet or Regnet series networks) and the vehicle state encoder E e (usually MLP, lightweight transformer or a hybrid structure of the two can also be used), multimodal input I l and I i After passing through the point cloud and image encoders, the point cloud features F under the BEV perspective are obtained l and multi-view image features F i . The vehicle state S e After the vehicle encoder E e Encode the query feature Q of the current state of the vehicle e .

[0023] (3) Environmental perception module: The environmental perception module includes a multi-task detection intermediate module including 3D obstacle detection, map detection, static scene occupancy, and traffic light detection. Among them, 3D obstacle detection, map detection, and traffic light detection are all implemented using the query-based sparse 3D object detection paradigm. The current state S of common objects on the road is obtained respectively. a and feature Q a , map status (including vector point coordinates and categories) S m and feature Q m , traffic light status S t and feature Q t The occupancy of static scenes is mainly to solve the problem of avoiding general static obstacles other than common targets. The specific implementation of occupancy detection is as follows: First, the point cloud feature F l and a learnable global query Q g After adding, self-attention is performed first, and then the fused BEV feature is obtained from the image feature query through the Deformer Attention (deformable attention module). Then, this BEV feature is merged with the historical BEV feature, and then the feature Q is obtained through the BEV encoder. c , and finally the dimension is upgraded to obtain the final Occupancy prediction result S c (including whether each voxel is occupied and its category information).

[0024] (4) Synchronous forecasting & planning module: Figure 2 The figure shows the architecture of the synchronous prediction & planning module. The present invention takes into account that in actual scenarios, the ego vehicle and other targets should be synchronized at every moment and have a game influence on each other, so the prediction of other targets is carried out synchronously with the planning of the ego vehicle. At the same time, drawing on the successful experience of the large language model autoregression model, the present invention also uses an autoregressive method to decompose the ego vehicle and target states at the next N moments into the state and environmental information (traffic lights, occupancy, map state and intermediate features) at the past and current moments to predict the state at the next moment, and then repeatedly obtain the state of all N moments. This model is more in line with actual objective laws, so it can greatly improve the performance of the model. The specific operation of obtaining the state at the next moment from the current moment is as follows: First, the ego vehicle feature Q at moment T is e and the target feature Q a After merging, we get query Q. Next, we transform the vehicle state S at time T into e and the target state S a After merging, the position encoder is used to obtain the position encoding feature Q pos Then, the traffic light status S t , Occupancy status S c, Map Status S m After merging, the environment state features E are obtained by passing through the state encoder and position encoder respectively. se and K pos Then, the traffic light feature Q t , Occupancy feature Q c 、Map FeaturesQ m After merging, follow E se After adding the features, we get K and V (here K is equal to V. K and V are used as the key and value in the attention module in the subsequent transformer. In order to emphasize the state information of the environment, the state encoder States Encoder can also be used as the vehicle state encoder E e A similar structure is used to encode the displayed state of the environment and then fuse it into features K and V by adding them together. Finally, Q and Q pos , K, V and K pos They are sent together into the transformer to obtain the state at the next moment (i.e., moment T+1).

[0025] In an embodiment of the present invention, an end-to-end autonomous driving method based on autoregressive synchronous prediction and planning is provided, which comprises the following steps: S1 data input: inputting multimodal data and vehicle status, wherein the multimodal data comprises point cloud and multi-view image; S2 feature encoding: respectively passing the point cloud and the multi-view image through a point cloud encoder and an image encoder to obtain a point cloud feature F under the perspective of the BEV; l and multi-view image features F i , and encode the vehicle state through the vehicle encoder to obtain the query feature Q of the vehicle state e ; S3 environmental perception: obtain the target state S through 3D obstacle detection, map detection and traffic light detection a and the target feature Q a , Map Status S m and map feature Q m , Traffic light status S t and traffic light feature Q t And obtain the prediction result S of the occupancy state through occupancy detection of static scenes c ; S4 Synchronous prediction and planning: Through the autoregressive method, the vehicle state and target state at the next N moments are decomposed into the state at the next moment predicted by using the state and environmental information at the past and current moments, and the state of all N moments is obtained in turn.

[0026] Furthermore, in step S3, occupancy detection specifically includes: firstly, the point cloud feature F l and a learnable global query Qg After adding, self-attention is performed first; then the fused BEV features are obtained from the image feature query through Deformer Attention; then the above BEV features are merged with the historical BEV features, and then the Occupancy feature Q is obtained through the BEV encoder. c ; Finally, the dimension is upgraded to obtain the final prediction result S of the occupancy state c .

[0027] Furthermore, in step S3, a query-based sparse 3D object detection paradigm is used to implement 3D obstacle detection, map detection, and traffic light detection.

[0028] Further, step S4 specifically includes: S41: the vehicle feature Q at time T e and the target feature Q a After merging, we get query Q; S42: The vehicle state S at time T e and the target state S a After merging, Q is obtained through the position encoder pos ; S43: Change the traffic light status S t , Occupancy status S c , Map Status S m After merging, the environment state features E are obtained by passing through the state encoder and position encoder respectively. se and K pos ; S45: Set the traffic light feature Q t 、Occupancy feature Q c 、Map FeaturesQ m After merging, the environmental state feature E se After adding, we get K and V; S46: add Q, Q pos , K, V and K pos They are sent together into the transformer to get the state at the next moment, that is, the state at moment T+1.

[0029] Furthermore, the vehicle state includes the size of the vehicle and the current position and speed, the map state includes the coordinates and categories of the vector points, and the occupancy state includes whether each voxel is occupied and the category information.

[0030] In another embodiment of the present invention, an end-to-end autonomous driving system based on autoregressive synchronous prediction and planning is also provided, which includes the following modules: a data input module, used to input multimodal data and self-vehicle status, wherein the multimodal data includes point cloud and multi-view image; a feature encoding module, used to obtain point cloud features F from the perspective of BEV by respectively converting the point cloud and the multi-view image through a point cloud encoder and an image encoder l and multi-view image features Fi , and encode the vehicle state through the vehicle encoder to obtain the query feature Q of the vehicle state e ; Environmental perception module, used to obtain the target state S through 3D obstacle detection, map detection and traffic light detection a and the target feature Q a , Map Status S m and map feature Q m , Traffic light status S t and traffic light feature Q t And obtain the prediction result S of the occupancy state through occupancy detection of static scenes c The synchronous prediction and planning module is used to decompose the vehicle state and target state at the next N moments through autoregression, using the state and environment information at the past and current moments to predict the state at the next moment, and then repeat this process to obtain the states at all N moments.

[0031] Furthermore, the environment perception module implements occupancy detection according to the following steps: first, the point cloud feature F l and a learnable global query Q g After adding, self-attention is performed first; then the fused BEV features are obtained from the image feature query through Deformer Attention; then the above BEV features are merged with the historical BEV features, and then the Occupancy feature Q is obtained through the BEV encoder. c ; Finally, the dimension is upgraded to obtain the final prediction result S of the occupancy state c .

[0032] Furthermore, the environment perception module adopts a query-based sparse 3D target detection paradigm to achieve 3D obstacle detection, map detection and traffic light detection.

[0033] Furthermore, the synchronous prediction and planning module performs the following steps: the vehicle feature Q at time T e and the target feature Q a After merging, we get query Q; the state of the vehicle at time T S e and the target state S a After merging, Q is obtained through the position encoder pos ; Change the traffic light status S t , Occupancy status S c , Map Status S m After merging, the environment state features E are obtained by passing through the state encoder and position encoder respectively. se and K pos ; Set the traffic light feature Q t , Occupancy feature Qc 、Map FeaturesQ m After merging, the environmental state feature E se After adding, we get K and V; pos , K, V and K pos They are sent together into the transformer to get the state at the next moment, that is, the state at moment T+1.

[0034] In other embodiments of the present invention, a computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the method described above is performed.

[0035] Beneficial technical effects brought by the technical solution of the present invention

[0036] The present invention provides an end-to-end autonomous driving solution that uses an autoregressive model for synchronous prediction and planning. It fully considers that the self-vehicle and other targets in the actual scene should be synchronized at every moment and have a game influence on each other. At the same time, drawing on the successful experience of the large language model autoregressive model, the present invention uses an autoregressive method to decompose the self-vehicle and target states at the next N moments into the state and environmental information (traffic lights, occupancy, map state and intermediate features) at the past and current moments to predict the state at the next moment, and then reciprocates to obtain the state of all the future N moments. This model is more in line with the actual objective laws of prediction and planning, so it can greatly improve the performance of the model.

[0037] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An end-to-end autonomous driving method based on autoregressive synchronous prediction and planning, characterized in that: The steps include: S1 data input: input multimodal data and vehicle status, where the multimodal data includes point cloud and multi-view images; S2 feature encoding: The point cloud and multi-view image are respectively passed through the point cloud encoder and image encoder to obtain the point cloud feature F under the BEV perspective. l and multi-view image features F i , and encode the vehicle state through the vehicle encoder to obtain the query feature Q of the vehicle state e ; S3 environment perception: obtain the target state S through 3D obstacle detection, map detection and traffic light detection a and the target feature Q a , Map Status S m and map feature Q m , Traffic light status S t and traffic light feature Q t And obtain the prediction result S of the occupancy state through occupancy detection of static scenes c ; S4 Synchronous prediction and planning: Through autoregression, the vehicle state and target state at the next N moments are decomposed into the state at the next moment predicted by using the state and environment information at the past and current moments, and the state of all N moments is obtained in turn.

2. The method according to claim 1, characterized in that: In step S3, occupancy detection specifically includes: First, the point cloud feature F l and a learnable global query Q g After adding, do self-attention first; Then use Deformer Attention to extract the image features F i Query to obtain the fused BEV features; Then the above BEV features are combined with the historical BEV features, and then the Occupancy feature Q is obtained through the BEV encoder. c ; Finally, the dimension is increased to obtain the final prediction result S of the occupancy state c .

3. The method according to claim 2, characterized in that In step S3, the Query-Based sparse 3D object detection paradigm is used to implement 3D obstacle detection, map detection, and traffic light detection.

4. The method according to claim 3, characterized in that Step S4 specifically includes: S41: The vehicle feature Q at time T e and the target feature Q a After merging, we get query Q; S42: The vehicle state S at time T e and the target state S a After merging, the position encoder is used to obtain the position encoding feature Q pos ; S43: Change the traffic light status to S t , Occupancy status S c , Map Status S m After merging, the environment state features E are obtained by passing through the state encoder and position encoder respectively. se and K pos ; S45: Set traffic light feature Q t , Occupancy feature Q c 、Map FeaturesQ m After merging, the environmental state feature E se After adding, we get the features K and V; S46: Q, Q pos , K, V and K pos They are sent together into the transformer to get the state at the next moment, that is, the state at moment T+1.

5. The method according to claim 4, characterized in that The vehicle state includes the size of the vehicle and the current position and speed, the map state includes the coordinates and categories of the vector points, and the occupancy state includes whether each voxel is occupied and the category information.

6. An end-to-end autonomous driving system based on autoregressive synchronous prediction and planning, characterized in that: Includes the following modules: A data input module, used for inputting multimodal data and vehicle status, wherein the multimodal data includes point clouds and multi-view images; The feature encoding module is used to obtain the point cloud feature F under the BEV perspective by passing the point cloud encoder and the multi-view image through the point cloud encoder and the image encoder respectively. l and multi-view image features F i , and encode the vehicle state through the vehicle encoder to obtain the query feature Q of the vehicle state e ; The environment perception module is used to obtain the target state S through 3D obstacle detection, map detection and traffic light detection respectively. a and the target feature Q a , Map Status S m and map feature Q m , Traffic light status S t and traffic light feature Q t And obtain the prediction result S of the occupancy state through occupancy detection of static scenes c ; The synchronous prediction and planning module is used to decompose the vehicle state and target state at the next N moments through autoregression, using the state and environment information at the past and current moments to predict the state at the next moment, and then repeat this process to obtain the states at all N moments.

7. The system according to claim 6, characterized in that The environment perception module implements occupancy detection according to the following steps: First, the point cloud feature F l and a learnable global query Q g After adding, do self-attention first; Then use Deformer Attention to extract the image features F i Query to obtain the fused BEV features; Then the above BEV features are combined with the historical BEV features, and then the Occupancy feature Q is obtained through the BEV encoder. c ; Finally, the dimension is increased to obtain the final prediction result S of the occupancy state c .

8. The system according to claim 7, characterized in that The environment perception module adopts the query-based sparse 3D target detection paradigm to realize 3D obstacle detection, map detection and traffic light detection.

9. The system according to claim 8, characterized in that The synchronous prediction and planning module performs the following steps: The vehicle characteristics Q at time T e and the target feature Q a After merging, we get query Q; The vehicle state S at time T e and the target state S a After merging, the position encoder is used to obtain the position encoding feature Q pos ; Set the traffic light status S t , Occupancy status S c , Map Status S m After merging, the environment state features E are obtained by passing through the state encoder and position encoder respectively. se and K pos ; Set the traffic light feature Q t , Occupancy feature Q c 、Map FeaturesQ m After merging, the environmental state feature E se After adding, we get the features K and V; Q, Q pos , K, V and K pos They are sent together into the transformer to get the state at the next moment, that is, the state at moment T+1.

10. A computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 5 is performed.

Images