End-to-end autonomous driving method and system based on autoregressive synchronous prediction and planning

Abstract

The application discloses an end-to-end automatic driving method and system based on autoregressive synchronous prediction and planning, and the method comprises the following steps: S1, data input: inputting multi-modal data and a self-vehicle state, wherein the multi-modal data comprises a point cloud and a multi-view image; S2, feature coding: obtaining point cloud features F l and multi-view image features F i in a BEV perspective by respectively coding the point cloud and the multi-view image through a point cloud encoder and an image encoder, and obtaining a query feature Q e of the self-vehicle state by coding the self-vehicle state through a self-vehicle encoder; S3, environment perception: obtaining target states S a and target features Q a , map states S m and map features Q m , and traffic light states S t and traffic light features Q t by respectively performing 3D obstacle detection, map detection and traffic light detection, and obtaining a prediction result S c of an occupancy state by performing occupancy detection on a static scene; and S4, synchronous prediction and planning: decomposing the self-vehicle states and the target states at N future time points into states at the next time point by using state and environment information at past and current time points in an autoregressive manner, and obtaining the states at all the N time points in turn.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous driving method and system based on autoregressive synchronous prediction and planning. Background Technology

[0002] In the fields of autonomous driving, robotics, and other intelligent agents, especially autonomous driving, the entire autonomous driving technology stack is generally broken down into sub-modules such as environmental perception, tracking and post-fusion, prediction, and planning for separate processing. Recently, a surge of research into end-to-end autonomous driving technology has emerged in academia, with representative examples including UniAD and VAD. Compared to earlier multi-task streaming methods, end-to-end autonomous driving technology offers the following significant advantages: 1. Simplification of unnecessary steps. For example, tracking and post-fusion, and even all intermediate steps, can be discarded; 2. Greatly simplified deployment process, requiring only a single model; 3. Data-driven AI with a higher ceiling, while significantly reducing the amount of rule-based code.

[0003] In recent years, representative solutions in the end-to-end autonomous driving field include UniAD and VAD. Their proposals have dispelled doubts about end-to-end autonomous driving because their structural design possesses a degree of interpretability. UniAD was the first proposed and widely adopted end-to-end autonomous driving solution. Its features include complete sub-modules for environmental perception, tracking, prediction, occupancy, and planning, and it cleverly utilizes transformers to effectively connect these modules, achieving end-to-end autonomous driving. However, UniAD also has significant drawbacks, namely its overly complex model structure, which poses considerable challenges to model training and deployment. VAD employs a vectorization method to vectorize intermediate tasks in an attempt to increase computational efficiency and save GPU memory. Summary of the Invention

[0004] To address the problems in existing technologies, this invention aims to further simplify the complexity of end-to-end autonomous driving while improving the performance of end-to-end autonomous driving technology based on an autoregressive synchronous prediction and planning method.

[0005] To achieve the above objectives, the technical solution of this invention provides an end-to-end autonomous driving method based on autoregressive synchronous prediction and planning, which includes the following steps: S1 Data Input: Input multimodal data and vehicle state, wherein the multimodal data includes point cloud and multi-view images; S2 Feature Encoding: Obtain point cloud features F from the BEV perspective by passing the point cloud and multi-view images through a point cloud encoder and an image encoder, respectively. l and multi-view image features F i The vehicle status is then encoded using the vehicle encoder to obtain the query feature Q of the vehicle status. eS3 Environmental Perception: The target state S is obtained through 3D obstacle detection, map detection, and traffic light detection respectively. a and target features Q a Map Status S m and map features Q m Traffic light status S t Traffic light features Q t And obtain the prediction result S of the Occupancy state through Occupancy detection in static scenes. c S4 Synchronous Prediction and Planning: The autoregressive method decomposes the vehicle state and target state at N future moments into the state at the next moment by using the state and environmental information at the past and current moments, and repeats this process to obtain the state at all N moments.

[0006] Furthermore, in step S3, the occupancy detection specifically includes: first, processing the point cloud features F... l and learnable global query Q g After addition, self-attention is performed first; then Deformer Attention is used to extract image features F. i The fused BEV features are retrieved; then, these BEV features are merged with historical BEV features, and finally, the BEV encoder is used to obtain the Occupancy feature Q. c Finally, by raising the dimension, we obtain the final prediction result S of the Occupancy state. c .

[0007] Furthermore, in step S3, a query-based sparse 3D object detection paradigm is used to achieve 3D obstacle detection, map detection, and traffic light detection.

[0008] Furthermore, step S4 specifically includes: S41: The vehicle feature Q at time T... e and target features Q a After merging, we get query Q; S42: the vehicle state S at time T. e and target state S a After merging, the position encoder produces the position encoded feature Q. pos S43: Set the traffic light status S t Occupancy status S c Map Status S m After merging, the environmental state features E are obtained by passing the state encoder and position encoder respectively. se and K pos S45: Transmit traffic light feature Q t Occupancy feature Q c Map features Q mAfter merging, the environmental state characteristic E se After adding them, we get features K and V; S46: Combine Q and Q pos K, V and K pos They are fed into the transformer to obtain the state at the next time step, i.e., time T+1.

[0009] Furthermore, the vehicle state includes the vehicle size and current pose and speed, the map state includes vector point coordinates and categories, and the occupancy state includes whether each voxel is occupied and its category information.

[0010] The technical solution of this invention also provides an end-to-end autonomous driving system based on autoregressive synchronous prediction and planning, which includes the following modules: a data input module for inputting multimodal data and vehicle status, wherein the multimodal data includes point clouds and multi-view images; and a feature encoding module for obtaining point cloud features F from the BEV perspective by passing the point cloud and multi-view images through a point cloud encoder and an image encoder, respectively. l and multi-view image features F i The vehicle status is then encoded using the vehicle encoder to obtain the query feature Q of the vehicle status. e The environmental perception module is used to obtain the target state S through 3D obstacle detection, map detection, and traffic light detection, respectively. a and target features Q a Map Status S m and map features Q m Traffic light status S t Traffic light features Q t And obtain the prediction result S of the Occupancy state through Occupancy detection in static scenes. c The synchronous prediction and planning module is used to decompose the vehicle state and target state at N future moments into predicting the state at the next moment using past and current state and environmental information, and repeating this process to obtain the state at all N moments.

[0011] Furthermore, the environment perception module implements occupancy detection according to the following steps: First, the point cloud features F l and learnable global query Q g After addition, self-attention is performed first; then Deformer Attention is used to extract image features F. i The fused BEV features are retrieved; then, these BEV features are merged with historical BEV features, and finally, the BEV encoder is used to obtain the Occupancy feature Q. c Finally, by raising the dimension, we obtain the final prediction result S of the Occupancy state. c .

[0012] Furthermore, the environment perception module adopts a query-based sparse 3D target detection paradigm to achieve 3D obstacle detection, map detection, and traffic light detection.

[0013] Furthermore, the synchronous prediction and planning module performs the following steps: It calculates the vehicle feature Q at time T... e and target features Q a After merging, we get query Q; and the vehicle state S at time T. e and target state S a After merging, the position encoder produces the position encoded feature Q. pos ; Set the traffic light status S t Occupancy status S c Map Status S m After merging, the environmental state features E are obtained by passing the state encoder and position encoder respectively. se and K pos ; the traffic light feature Q t Occupancy feature Q c Map features Q m After merging, the environmental state characteristic E se After adding them, we get features K and V; then we add Q and Q pos K, V and K pos They are fed into the transformer to obtain the state at the next time step, i.e., time T+1.

[0014] The present invention also provides a computer-readable storage medium containing a computer program, which, when executed by one or more processors, performs the method described above. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0017] Figure 2 This is an architecture diagram of the synchronous prediction and planning module of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This invention focuses on autoregressive synchronization prediction and planning techniques in the proposed end-to-end autonomous driving solution. The overall solution of this invention is as follows: Figure 1 .like Figure 1 As shown, the present invention includes an input (multimodal, including but not limited to multi-view images, point clouds, etc.), a feature encoder module, an environment perception module (including but not limited to 3D object detection, map detection, static scene occupancy and traffic light detection), and a simultaneous prediction & planning module.

[0020] The specific technical approach of this invention is as follows:

[0021] (1) Input: The input data of this invention includes multimodal input (including point cloud and multi-view image as input, respectively denoted as I) l and I i (Indicates) Self-vehicle status S e (Including vehicle dimensions, current pose and speed, and past states can also be used as input. For ease of description, this article only uses the current state as an example.)

[0022] (2) Feature encoder: including point cloud encoder E l (This can be any point cloud feature extractor, such as PointNet, sparse convolutional networks, etc.), image encoder E i (Generally using ResNet or RegNet networks) and vehicle status encoder E e (Generally an MLP, but a lightweight transformer or a hybrid of both can also be used), multimodal input I l and I i The point cloud features F from the BEV perspective are obtained after passing through the point cloud and image encoders, respectively. l and multi-view image features F i Vehicle status S e After passing through the vehicle encoder E e Encoding yields the query feature Q of the vehicle's current state. e .

[0023] (3) Environmental Perception Module: The environmental perception module includes a multi-task detection intermediate module encompassing 3D obstacle detection, map detection, static scene occupancy, and traffic light detection. Among these, 3D obstacle detection, map detection, and traffic light detection all employ a query-based sparse 3D object detection paradigm. The current state S of common targets in the road is obtained respectively. a and feature Q a Map status (including vector point coordinates and categories) S m and feature Q m Traffic light status S t and feature Q t Occupancy detection in static scenes primarily aims to solve the problem of obstacle avoidance against general static obstacles, excluding common targets. The specific implementation of occupancy detection involves first processing the point cloud features F... l and learnable global query Q g After addition, self-attention is performed first, followed by Deformer Attention to obtain the fused BEV features from image feature queries. Then, this BEV feature is merged with historical BEV features, and finally passed through a BEV encoder to obtain feature Q. c Finally, by upgrading the dimensionality, the final Occupancy prediction result S is obtained. c (Includes information on whether each voxel is occupied and its category).

[0024] (4) Synchronous Forecasting & Planning Module: such as Figure 2 The diagram shows the architecture of the synchronous prediction and planning module. This invention considers that in real-world scenarios, the vehicle and other targets should be synchronized at every moment, and there is a game-like influence between them. Therefore, it synchronizes the prediction of other targets with the planning of the vehicle. Drawing on the successful experience of the autoregressive model in large language models, this invention also uses an autoregressive approach to decompose the vehicle and target states at N future moments into predicting the state at the next moment using past and current state and environmental information (traffic lights, occupancy, map state, and intermediate features), and repeats this process to obtain the states at all N moments. This model is more consistent with actual objective laws, thus greatly improving model performance. The specific operation of obtaining the state at the next moment from the current moment is as follows: First, the vehicle features Q at time T are... e and target features Q a After merging, we obtain query Q. Next, we obtain the vehicle state S at time T. e and target state S a After merging, the position encoder produces the position encoded feature Q. pos Then, set the traffic light status S. t Occupancy status S cMap Status S m After merging, the environmental state features E are obtained by passing the state encoder and position encoder respectively. se and K pos Subsequently, the traffic light feature Q was... t Occupancy feature Q c Map features Q m After the merger with E se After adding the features, we get K and V (here K equals V). K and V are used as the key and value in the attention module of the subsequent transformer. To emphasize the state information of the environment, a State Encoder is used. Similarly, the state encoder used in the vehicle can be the same as the E-encoder. e A similar structure encodes the displayed environmental state and then fuses it into features K and V through addition. Finally, Q and Q' are... pos K, V and K pos They are fed into the transformer together to obtain the state at the next time step (i.e., time T+1).

[0025] In an embodiment of the present invention, an end-to-end autonomous driving method based on autoregressive synchronous prediction and planning is provided, comprising the following steps: S1 Data input: inputting multimodal data and vehicle state, wherein the multimodal data includes point cloud and multi-view images; S2 Feature encoding: obtaining point cloud features F from the BEV perspective by passing the point cloud and multi-view images through a point cloud encoder and an image encoder, respectively. l and multi-view image features F i The vehicle status is then encoded using the vehicle encoder to obtain the query feature Q of the vehicle status. e S3 Environmental Perception: The target state S is obtained through 3D obstacle detection, map detection, and traffic light detection respectively. a and target features Q a Map Status S m and map features Q m Traffic light status S t Traffic light features Q t And obtain the prediction result S of the Occupancy state through Occupancy detection in static scenes. c S4 Synchronous Prediction and Planning: The autoregressive method decomposes the vehicle state and target state at N future moments into the state at the next moment by using the state and environmental information at the past and current moments, and repeats this process to obtain the state at all N moments.

[0026] Furthermore, in step S3, the occupancy detection specifically includes: first, processing the point cloud features F... l and learnable global query Qg After addition, self-attention is performed first; then, the fused BEV features are obtained from image feature queries through Deformer Attention; subsequently, the above BEV features are merged with historical BEV features, and then passed through the BEV encoder to obtain the Occupancy feature Q. c Finally, by raising the dimension, we obtain the final prediction result S of the Occupancy state. c .

[0027] Furthermore, in step S3, a query-based sparse 3D object detection paradigm is used to achieve 3D obstacle detection, map detection, and traffic light detection.

[0028] Furthermore, step S4 specifically includes: S41: The vehicle feature Q at time T... e and target features Q a After merging, we get query Q; S42: the vehicle state S at time T. e and target state S a After merging, Q is obtained through a position encoder. pos S43: Set the traffic light status S t Occupancy status S c Map Status S m After merging, the environmental state features E are obtained by passing the state encoder and position encoder respectively. se and K pos S45: Transmit traffic light feature Q t Occupancy feature Q c Map features Q m After merging, the environmental state characteristic E se Add them together to get K and V; S46: Add Q and Q pos K, V and K pos They are fed into the transformer to obtain the state at the next time step, i.e., time T+1.

[0029] Furthermore, the vehicle state includes the vehicle size and current pose and speed, the map state includes vector point coordinates and categories, and the occupancy state includes whether each voxel is occupied and its category information.

[0030] In another embodiment of the present invention, an end-to-end autonomous driving system based on autoregressive synchronous prediction and planning is also provided, comprising the following modules: a data input module for inputting multimodal data and vehicle state, wherein the multimodal data includes point clouds and multi-view images; and a feature encoding module for obtaining point cloud features F from the BEV perspective by passing the point cloud and multi-view images through a point cloud encoder and an image encoder, respectively. l and multi-view image features Fi The vehicle status is then encoded using the vehicle encoder to obtain the query feature Q of the vehicle status. e The environmental perception module is used to obtain the target state S through 3D obstacle detection, map detection, and traffic light detection, respectively. a and target features Q a Map Status S m and map features Q m Traffic light status S t Traffic light features Q t And obtain the prediction result S of the Occupancy state through Occupancy detection in static scenes. c The synchronous prediction and planning module is used to decompose the vehicle state and target state at N future moments into predicting the state at the next moment using past and current state and environmental information, and repeating this process to obtain the state at all N moments.

[0031] Furthermore, the environment perception module implements occupancy detection according to the following steps: First, the point cloud features F l and learnable global query Q g After addition, self-attention is performed first; then, the fused BEV features are obtained from image feature queries through Deformer Attention; subsequently, the above BEV features are merged with historical BEV features, and then passed through the BEV encoder to obtain the Occupancy feature Q. c Finally, by raising the dimension, we obtain the final prediction result S of the Occupancy state. c .

[0032] Furthermore, the environment perception module adopts a query-based sparse 3D target detection paradigm to achieve 3D obstacle detection, map detection, and traffic light detection.

[0033] Furthermore, the synchronous prediction and planning module performs the following steps: It calculates the vehicle feature Q at time T... e and target features Q a After merging, we get query Q; and the vehicle state S at time T. e and target state S a After merging, Q is obtained through a position encoder. pos ; Set the traffic light status S t Occupancy status S c Map Status S m After merging, the environmental state features E are obtained by passing the state encoder and position encoder respectively. se and K pos ; the traffic light feature Q t Occupancy feature Qc Map features Q m After merging, the environmental state characteristic E se Add them together to get K and V; then add Q and Q' pos K, V and K pos They are fed into the transformer to obtain the state at the next time step, i.e., time T+1.

[0034] In other embodiments of the present invention, a computer-readable storage medium comprising a computer program is also provided, which, when executed by one or more processors, performs the method described above.

[0035] Beneficial technical effects brought about by the technical solution of the present invention

[0036] This invention provides an end-to-end autonomous driving solution utilizing autoregressive models for synchronous prediction and planning. It fully considers the real-world scenario where the vehicle and other targets are synchronized at every moment and interact with each other through game theory. Drawing on the successful experience of autoregressive models in large language models, this invention decomposes the vehicle and target states for the next N moments into predictive states using past and current state and environmental information (traffic lights, occupancy, map state, and intermediate features). This process is repeated to obtain the states for all N future moments. This approach better aligns with the objective laws of prediction and planning, thus significantly improving model performance.

[0037] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An end-to-end autonomous driving method based on autoregressive synchronous prediction and planning, characterized in that, Includes the following steps: S1 Data Input: Input multimodal data and vehicle status, where multimodal data includes point cloud and multi-view images; S2 Feature Encoding: Point cloud features F from the BEV perspective are obtained by passing the point cloud and multi-view images through a point cloud encoder and an image encoder, respectively. l and multi-view image features F i The vehicle status is then encoded using the vehicle encoder to obtain the query feature Q of the vehicle status. e ; S3 Environmental Perception: Obtains the target state S through 3D obstacle detection, map detection, and traffic light detection respectively. a and target features Q a Map Status S m and map features Q m Traffic light status S t Traffic light features Q t And obtain the prediction result S of the Occupancy state through Occupancy detection in static scenes. c ; S4 Synchronous Prediction and Planning: This method uses an autoregressive approach to decompose the vehicle's state and the target state at N future time points into predicting the state at the next time point using past and current state information and environmental data. This process is repeated to obtain the states at all N time points. Step S4 specifically includes: S41: The vehicle feature Q at time T e and target features Q a The merged result is query Q; S42: Set the vehicle state S at time T. e and target state S a After merging, the position encoder produces the position encoded feature Q. pos ; S43: Change the traffic light status S t Occupancy status S c Map Status S m After merging, the environmental state features E are obtained by passing the state encoder and position encoder respectively. se and K pos ; S45: Transmit traffic light feature Q t Occupancy feature Q c Map features Q m After merging, the environmental state characteristic E se After adding them together, we get the features K and V; S46: Q, Q pos K, V and K pos They are fed into the transformer to obtain the state at the next time step, i.e., time T+1.

2. The method according to claim 1, characterized in that, In step S3, the occupancy detection specifically includes: First, the point cloud features F l and learnable global query Q g After adding them together, first apply self-attention; Then, Deformer Attention is used to extract image features F i The query retrieves the fused BEV features; The aforementioned BEV features are then merged with historical BEV features, and then processed by the BEV encoder to obtain the Occupancy feature Q. c ; Finally, the dimensionality is increased to obtain the final prediction result S of the Occupancy state. c .

3. The method according to claim 2, characterized in that, In step S3, a query-based sparse 3D object detection paradigm is used to achieve 3D obstacle detection, map detection, and traffic light detection.

4. The method according to claim 3, characterized in that, The vehicle status includes the vehicle size and current pose and speed; the map status includes vector point coordinates and categories; and the occupancy status includes whether each voxel is occupied and its category information.

5. An end-to-end autonomous driving system based on autoregressive synchronous prediction and planning, characterized in that, Includes the following modules: The data input module is used to input multimodal data and vehicle status, whereby the multimodal data includes point clouds and multi-view images; The feature encoding module is used to obtain point cloud features F from the BEV viewpoint by passing the point cloud and multi-view images through the point cloud encoder and image encoder, respectively. l and multi-view image features F i The vehicle status is then encoded using the vehicle encoder to obtain the query feature Q of the vehicle status. e ; The environmental perception module is used to obtain the target state S through 3D obstacle detection, map detection, and traffic light detection, respectively. a and target features Q a Map Status S m and map features Q m Traffic light status S t Traffic light features Q t And obtain the prediction result S of the Occupancy state through Occupancy detection in static scenes. c ; The synchronous prediction and planning module is used to decompose the vehicle state and target state at N future time points into predicting the state at the next time point using past and current state and environmental information, and repeating this process to obtain the states at all N time points; The synchronous prediction and planning module performs the following steps: The vehicle's characteristics Q at time T e and target features Q a The merged result is query Q; The vehicle's state S at time T e and target state S a After merging, the position encoder produces the position encoded feature Q. pos ; Traffic light status S t Occupancy status S c Map Status S m After merging, the environmental state features E are obtained by passing the state encoder and position encoder respectively. se and K pos ; Traffic light feature Q t Occupancy feature Q c Map features Q m After merging, the environmental state characteristic E se After adding them together, we get the features K and V; Q, Q pos K, V and K pos They are fed into the transformer to obtain the state at the next time step, i.e., time T+1.

6. The system according to claim 5, characterized in that, The environmental perception module performs occupancy detection according to the following steps: First, the point cloud features F l and learnable global query Q g After adding them together, first apply self-attention; Then, Deformer Attention is used to extract image features F i The query retrieves the fused BEV features; The aforementioned BEV features are then merged with historical BEV features, and then processed by the BEV encoder to obtain the Occupancy feature Q. c ; Finally, the dimensionality is increased to obtain the final prediction result S of the Occupancy state. c .

7. The system according to claim 6, characterized in that, The environmental perception module uses a query-based sparse 3D object detection paradigm to achieve 3D obstacle detection, map detection, and traffic light detection.

8. A computer-readable storage medium containing a computer program, characterized in that, When the computer program is executed by one or more processors, the method as described in any one of claims 1-4 is performed.

Images