End-to-end autonomous driving planning method and system and storage medium

By adopting parallel network structure and memory queue technology in the autonomous driving system, information is directly obtained from multi-view image features, the problems of insufficient perceived information interaction and slow inference speed under the serialized architecture are solved, and the system's decision-making accuracy is improved Sex and reasoning speed.

CN118172618BActive Publication Date: 2024-11-26COWA TECHNOLOGY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410441618.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-12
Publication Date
2024-11-26
Estimated Expiration
2044-04-12

AI Technical Summary

Technical Problem

The serialized network architecture in the existing end-to-end system of autonomous driving leads to insufficient interaction of perceived information, slow inference speed, difficult to adapt to delay-sensitive scenarios, and insufficient utilization of bicycle status information, which reduces the system's prediction accuracy and reliability. sex.

Method used

The parallel network structure is adopted to blur the boundaries of upstream and downstream tasks, improve the information interaction rate through memory queues, directly obtain information from multi-view image features, reduce calculation costs, and improve timing consistency and inference speed.

Benefits of technology

It realizes more efficient perceived information interaction, reduces training difficulty, improves the model's timing perception ability and reasoning speed, and improves the decision-making accuracy and reliability of the autonomous driving system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BDA0004789357360000095
    Figure BDA0004789357360000095
  • Figure FHA0000006668450000021
    Figure FHA0000006668450000021
  • Figure FHA0000006668450000023
    Figure FHA0000006668450000023
Patent Text Reader

Abstract

The present invention provides an end-to-end autonomous driving planning method and system and a storage medium, the method comprising: S1: obtaining multi-view image features by preprocessing the original image obtained from the multi-view camera of the automobile; S2: initializing the memory queue to align the historical information stored in the memory queue with the coordinate system of the current time; S3: initializing the proxy query, map query and ego query of the current frame, and performing corresponding preprocessing on the proxy query, map query and ego query in steps, so as to facilitate subsequent learning from the multi-view image features; S4: after initializing the position coordinates corresponding to the three queries, splicing the three queries into a customized Transformer decoder to further enable the information between different element queries to circulate mutually; S5: for the proxy query and map query output by the decoder, firstly performing confidence prediction through the classification branch, and then sending the ego query with the highest confidence into the regression branch to predict the ego vehicle trajectory of the current frame in the next few seconds.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to autonomous driving technology, and in particular to an end-to-end autonomous driving planning method and system and a storage medium. Background Art

[0002] In the end-to-end autonomous driving system, by cleverly physically modeling the query in the Transformer network architecture, multiple modules in the traditional autonomous driving system, such as environmental perception, map construction, vehicle prediction, and control planning, can be combined to form an integrated large model, thereby improving the utilization efficiency of perception information and the reliability and stability of planning decisions.

[0003] Previous model architectures usually adopted a serialized approach to model the above links based on Transformer, where the query contains the corresponding feature codes of vehicles, pedestrians and map elements, and interacts with the BEV (bird's eye view) features, the physical state of the vehicle, and the output of other modules in different modules in turn, thereby finally outputting the future travel decision of the vehicle.

[0004] The paper "Planning-oriented Autonomous Driving" proposed the UniAD model, which uses the query in the Transformer architecture to connect multiple decoders upstream and downstream of autonomous driving. The decoders of each module simultaneously interact with BEV features, the current state of the vehicle, and the output of other modules to absorb information and output it to the downstream modules, thereby ultimately generating the vehicle's decision-making plan in the current scenario.

[0005] In the technology proposed in the paper "Planning-oriented Autonomous Driving", multiple upstream and downstream processes of autonomous driving are abstractly modeled as multiple groups of modules such as tracking decoder, map decoder, motion decoder, planning decoder, etc., which are connected through queries, and the perception information is used in a serialized manner to perform decision-making planning for the self-driving system.

[0006] However, the prior art has the following disadvantages:

[0007] (1) The serialized network architecture makes model training more difficult and there is a risk of insufficient interaction and incomplete utilization of perceptual information. Once the upstream module performs poorly, it is difficult for the downstream module to produce reliable and accurate predictions.

[0008] (2) Due to the high cost of acquiring BEV features, the overall reasoning speed of the model is relatively slow, making it difficult to make timely and effective predictions on streaming input and output, and unable to adapt to delay-sensitive autonomous driving scenarios;

[0009] (3) Insufficient use of vehicle status information reduces the accuracy and reliability of the system's overall prediction. Summary of the invention

[0010] In order to solve the above shortcomings existing in the prior art, the present invention proposes a more efficient end-to-end autonomous driving backbone network. Specifically, the previous serial network structure is modified into a parallel network structure, blurring the boundaries between upstream and downstream tasks, and improving the information interaction rate between different modules in a parallel manner, thereby reducing the difficulty of training and further reducing the loss of perceptual information. In addition, the BEV feature generation network is abandoned, and information is directly obtained from multi-view image features. At the same time, the temporal consistency of the model is improved by introducing a memory queue. This method can achieve efficient perceptual information acquisition with less computing cost.

[0011] To achieve the above-mentioned purpose, the technical solution of the present invention provides an end-to-end autonomous driving planning method, which includes the following steps: S1: obtaining multi-view image features by preprocessing the original image obtained from the multi-view camera of the car; S2: initializing the memory queue to align the historical information stored in the memory queue with the coordinate system of the current time; S3: initializing the proxy query, map query and ego query of the current frame, and preprocessing the proxy query, map query and ego query step by step, so as to facilitate subsequent learning from the multi-view image features; S4: after initializing the position coordinates corresponding to the three queries, the three queries are spliced ​​and sent to the customized Transformer decoder to further enable the information between different element queries to flow mutually; S5: for the proxy query and map query output by the decoder, firstly perform confidence prediction through the classification branch, and then send the ego query with the highest confidence to the regression branch to predict the ego vehicle trajectory in the next few seconds of the current frame.

[0012] Furthermore, in step S1, the original image is passed through the pre-trained residual neural network Resnet and feature pyramid network FPN to obtain multi-view image features.

[0013] Further, in step S1, for N×3×H×W image data, where N represents the number of images, 3 represents the RGB color channels, and H and W represent the height and width of the image, an N×(H×W / 64)×C image tag is generated by using an 8×8 patch size and a convolutional layer, where C represents the patch feature dimension.

[0014] Furthermore, in step S2, when initializing the memory queue of the current frame, the following preprocessing is performed: step S21: checking the scene mark of the current frame, if it is inconsistent with the scene mark of the previous frame, it means that the scene has changed, and the memory queue is cleared; step S22: by obtaining the speed and rotation angle information in the self-vehicle state, the agent position information in the memory queue is transformed to align it with the self-vehicle coordinate system in the current frame; step S23: for the coordinate misalignment caused by the movement of the agent itself, the historical query features are used to generate simulated deviations through the MLN network, so that the agent position information is further aligned with the current frame scene.

[0015] Furthermore, in step S2, during preprocessing, for query features, a 3×L×D all-zero queue is initialized, where L represents the queue length, D represents the feature dimension, 3 represents agent query, map query, and ego vehicle query, and for location coordinates, a 2×L×2 is initialized, where the coordinates are defined in the ego vehicle coordinate system under a bird's eye view; according to the formula Use the current frame's ego-vehicle state to align the position coordinates in the memory queue, where Refers to the vehicle change matrix from the previous frame to the current frame, represents the position coordinate at time t-1, Represents the position coordinates at time t; uses the current ego vehicle information to transform the agent position information to simulate the agent's spontaneous movement between the historical frame and the current frame Among them, LN represents inter-layer regularization, and the two parameters γ and β are derived from the encoding of the current frame's ego-vehicle transfer matrix, ego-vehicle speed, and time encoding using a multi-layer perceptron.

[0016] Furthermore, in step S3, corresponding preprocessing is performed on the proxy query, map query and self-vehicle query, including: step S31: for the proxy query and map query, multi-head attention interaction is performed with the corresponding historical query features and multi-view image features respectively, and confidence prediction is performed on the output query through the classification branch, and the k queries with the highest confidence are selected to enter the next step of interaction; step S32: for the self-vehicle query, multi-head attention interaction is performed with the self-vehicle historical query and the self-vehicle state of the current frame.

[0017] Furthermore, in step S3, when the proxy query, map query and ego vehicle query are preprocessed accordingly, in order to enhance the temporal perception of the model, a part of the historical queries in the memory queue are concatenated with the current query, thereby obtaining the temporally enhanced proxy query (A+L)×M×D and map query (I+L)×P×D, where L represents the number of historical queries read from the memory queue; the three queries are sent to the corresponding decoders for preprocessing, wherein the proxy query and the map query will perform multi-head attention interaction with the multi-view image information of size N×(H×W / 64)×C, and the ego vehicle query will interact with the current state of the ego vehicle of size 9; after the multi-head attention mechanism is completed, the output proxy query and map query are Top-K screened, wherein the generated query is sent to the classification branch, and the K queries with the highest confidence are selected according to the confidence predicted by the classification branch to enter the subsequent interaction, and the size of the screened proxy query should be K×M×D, and the size of the map query should be K×P×D.

[0018] Furthermore, in step S4, for the proxy queries, map queries and self-vehicle queries that have been confidence screened, the corresponding position coordinates are initialized, and the three queries and the corresponding historical query features in the memory queue are concatenated and sent to the decoder together with the corresponding position information. Each layer in the decoder performs the following operations: S41: Positionally encode the position coordinates of num queries×2 and concatenate them with num queries×D to the self-attention layer, where num queries=K×(M+P)+M, representing the total number of queries after the concatenation of the three queries; S42: The query is subjected to multi-head attention interaction with the multi-view image features after passing through the forward propagation network FFN and the inter-layer regularization LN sub-layer; S43: For the proxy queries and map queries output between the layers, they are sent to the regression branch to predict the offset of the position coordinates (Δx, Δy) to correct the original position coordinates of the proxy and map elements; S44: Repeat steps 4.1 to 4.3 several times so that the query can fully obtain information from the multi-view image features.

[0019] Furthermore, in step S42, the multi-view image features are first 3D position encoded and added to the learnable position embedding to model the spatial depth information in the image.

[0020] The technical solution of the present invention also provides an end-to-end autonomous driving planning system, which includes the following modules: an image preprocessing module, which is used to obtain multi-view image features by preprocessing the original image obtained from the multi-view camera of the car; a memory queue initialization module, which is used to initialize the memory queue to align the historical information stored in the memory queue with the coordinate system of the current time; a query initialization module, which is used to initialize the proxy query, map query and self-vehicle query of the current frame, and perform corresponding preprocessing on the proxy query, map query and self-vehicle query in steps to facilitate subsequent learning from the multi-view image features; a decoder module, which is used to initialize the position coordinates corresponding to the three queries, and then splice the three queries into a customized Transformer decoder to further enable the information between different element queries to flow mutually; a confidence prediction module, which is used to first perform confidence prediction on the proxy query and map query output by the decoder through the classification branch, and then send the self-vehicle query with the highest confidence to the regression branch to predict the self-vehicle trajectory of the current frame in the next few seconds.

[0021] The technical solution of the present invention also provides a computer-readable storage medium containing a computer program, and when the computer program is executed by one or more processors, the method as described above is implemented. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is further described below in conjunction with specific embodiments, but the present invention is not limited to these embodiments.

[0023] The present invention aims to solve the following technical problems:

[0024] (1) Optimize the existing serial architecture of the end-to-end autonomous driving model to reduce the difficulty of model training and enable full interaction of perception information;

[0025] (2) Remove the BEV feature generation module and directly obtain perceptual information from multi-view image features to improve reasoning speed and temporal consistency;

[0026] (3) Based on the input multi-view photos, the current driving scene environment features are implicitly constructed, and the vehicle’s route planning for a few seconds is output.

[0027] The key of the present invention is:

[0028] (1) Use a parallel network structure to reduce the difficulty of training convergence and improve the efficiency of information interaction;

[0029] (2) Using memory queues to store historical features, we can directly collect perceptual information from multi-view image features, which improves the model’s temporal perception ability and speeds up reasoning.

[0030] (3) Directly infer the vehicle’s future path from multi-view images during driving.

[0031] The end-to-end autonomous driving method proposed in the present invention mainly comprises the following steps:

[0032] 1. Read the original images and vehicle status data from the car's multi-view camera and on-board status recorder. The algorithm's task is to collect perception information from the above raw data for planning and decision-making.

[0033] 2. Since the semantic information of the original image is difficult to be directly used by the model, it is necessary to first obtain multi-view image features through the pre-trained residual neural network (Resnet) and feature pyramid network (FPN).

[0034] 3. The generation of BEV features requires expensive computational costs and large delays. We hope to learn directly from multi-view image features while maintaining good temporal consistency. Therefore, we need to set up a memory queue to store query features, agent positions, vehicle status and other information of the past few frames. At the same time, when initializing the memory queue of the current frame, the following preprocessing steps are required:

[0035] Step 3.1: Check the scene mark of the current frame. If it is inconsistent with the scene mark of the previous frame, it means that the scene has changed and the memory queue is cleared;

[0036] Step 3.2: By obtaining the speed, rotation angle and other information in the ego vehicle state, the agent position information in the memory queue is transformed to align it with the ego vehicle coordinate system in the current frame;

[0037] Step 3.3: For the coordinate misalignment caused by the agent’s own movement, the historical query features are passed through the MLN network to generate simulated deviations, so that the agent’s position information is further aligned with the current frame scene.

[0038] 4. Initialize learnable feature embeddings as queries for agents and map elements in the environment. In particular, initialize several feature embeddings as multi-modal ego vehicle feature queries, where each modality corresponds to a future path decision direction for the ego vehicle. Then perform corresponding preprocessing for agent queries, map queries, and ego vehicle queries step by step. The specific steps are listed as follows:

[0039] Step 4.1: For proxy queries and map queries, perform multi-head attention interactions with the corresponding historical query features and multi-view image features, perform confidence prediction on the output queries through the classification branch, and select the k queries with the highest confidence to enter the next step of interaction;

[0040] Step 4.2: For the ego-vehicle query, perform multi-head attention interaction with the ego-vehicle historical query and the ego-vehicle state of the current frame, so as to make full use of the beneficial decision information in the ego-vehicle state and collect time series information from the historical decision.

[0041] 5. For the proxy queries, map queries, and self-vehicle queries that have been confidence screened, initialize the corresponding location coordinates. The three queries and the corresponding historical query features in the memory queue are concatenated and sent to the decoder together with the corresponding location information. Specifically, the query will first perform a self-attention interaction in the decoder, and then perform an attention interaction with the multimodal image information. Each layer in the decoder will send the query output by the upper layer to the regression branch to generate a reference point correction. The position coordinates used by the upper layer are corrected and the above process is repeated.

[0042] 6. The ego-vehicle query after the decoder has fully interacted with the image features, agent information, and map information, and has already contained rich perception information, temporal information, and environmental features. At this time, the multimodal ego-vehicle query is sent to the classification branch for confidence prediction, and the ego-vehicle query with the highest confidence is sent to the prediction branch for final processing, and the ego-vehicle trajectory planning for the next few seconds in the current frame is output. Specific embodiment:

[0044] The end-to-end autonomous driving method proposed in the present invention solves the end-to-end autonomous driving task, and the main steps are:

[0045] Step 1: For the original image obtained from the car's multi-view camera, it is simply preprocessed to obtain perceptual features. For the N×3×H×W image data, where N represents the number of images, 3 represents the RGB color channels, and H and W represent the height and width of the image, an 8×8 patch size and a convolutional layer are used to generate N×(H×W / 64)×C image tags, where C represents the patch feature dimension.

[0046] Step 2: Initialize the memory queue to store the query features, position coordinates, and vehicle status of the historical frames to enhance the temporal consistency of the model.

[0047] Step 2.1: For query features, initialize a 3×L×D queue of all zeros, where L represents the queue length, D represents the feature dimension, 3 represents agent query, map query, and ego vehicle query, and for location coordinates, initialize 2×L×2, where the coordinates are defined in the ego vehicle coordinate system from a bird's eye view;

[0048] Step 2.2: According to the formula Use the current frame's ego-vehicle state to align the position coordinates in the memory queue, where Refers to the vehicle change matrix from the previous frame to the current frame, represents the position coordinate at time t-1, Represents the position coordinates at time t;

[0049] Step 2.3: Use the current ego vehicle information to transform the agent position information to simulate the agent's spontaneous movement between the historical frame and the current frame Among them, LN represents inter-layer regularization, and the two parameters γ and β are derived from the encoding of the current frame's ego-vehicle transfer matrix, ego-vehicle speed, and time encoding using a multi-layer perceptron.

[0050] Step 3: Initialize the agent query, map query, and ego vehicle query for the current frame: The size of the agent query is A×M×D, where A represents the maximum number of agents in the scene, M represents the possible motion modes of each agent, and D represents the feature dimension. The size of the map query is I×P×D, where I represents the maximum number of map elements, and P represents the number of position coordinates contained in each map element. The size of the ego vehicle query is M×6.

[0051] Step 3.1: In order to enhance the temporal awareness of the model, a part of the historical queries in the memory queue are concatenated with the current query to obtain the temporally enhanced proxy query (A+L)×M×D and the map query (I+L)×P×D, where L represents the number of historical queries read from the memory queue;

[0052] Step 3.2: Send the three queries to the corresponding decoders for preprocessing. The proxy query and map query will interact with the multi-view image information of size N×(H×W / 64)×C for multi-head attention, while the ego vehicle query will interact with the current state of the ego vehicle of size 9, which includes 9 driving features such as ego vehicle speed, acceleration, yaw angle, etc.

[0053] Step 3.3: After the multi-head attention mechanism ends, the output proxy query and map query are screened by Top-K. Specifically, the generated query is sent to the classification branch, and the K queries with the highest confidence are selected for subsequent interaction according to the confidence predicted by the classification branch. The size of the filtered proxy query should be K×M×D, and the size of the map query should be K×P×D.

[0054] Step 4: To further enable information exchange between different element queries, the query position coordinates are initialized and the three queries are concatenated and sent to the custom Transformer decoder. Each layer in the Transformer decoder performs the following operations:

[0055] Step 4.1: After position encoding, concatenate the position coordinates of size num queries × 2 with size num queries × D and send them to the self-attention layer, where num queries = K × (M + P) + M, which represents the total number of queries after concatenating the three queries;

[0056] Step 4.2: The query passes through the FFN (forward propagation network) and LN (inter-layer regularization) sublayers and then performs multi-head attention interaction with the multi-view image features. The multi-view image features here need to be first 3D position encoded and added to the learnable position embedding to model the spatial depth information in the image;

[0057] Step 4.3: For the proxy query and map query outputted between layers, send them to the regression branch to predict the offset (Δx, Δy) of the position coordinates to correct the original position coordinates of the proxy and map elements;

[0058] Step 4.4: Repeat steps 4.1 to 4.4 several times so that the query can fully extract information from multi-view image features.

[0059] Step 5: For the proxy and map queries output by the decoder, the classification branch first performs confidence prediction, and then the ego-vehicle query with the highest confidence is sent to the regression branch to predict the ego-vehicle trajectory in the next few seconds.

[0060] The beneficial effects brought by the technical solution of the present invention are:

[0061] (1) The present invention designs a more general and efficient backbone network for the end-to-end autonomous driving field. By inputting multi-view images collected by the vehicle during driving into this network, the optimal decision-making plan for the vehicle in the next few seconds can be reliably and accurately output;

[0062] (2) The present invention optimizes the serialized structure of the previous end-to-end autonomous driving system, reduces the difficulty of training and improves the information exchange rate;

[0063] (3) The present invention improves the drawbacks of previous end-to-end autonomous driving systems in collecting perception information from BEV features, greatly improves the inference rate, and enables the model to acquire the ability to process streaming information;

[0064] (4) The present invention designs a novel and effective way to interact with the vehicle state, thereby improving the accuracy and reliability of the model output.

[0065] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. An end-to-end autonomous driving planning method, characterized in that: The steps include: S1: The original image obtained from the multi-view camera of the car is preprocessed to obtain multi-view image features; S2: Initialize the memory queue to align the historical information stored in the memory queue with the coordinate system of the current time; S3: Initialize the proxy query, map query and ego-vehicle query of the current frame, and pre-process the proxy query, map query and ego-vehicle query step by step to facilitate subsequent learning from multi-view image features; S4: After initializing the position coordinates corresponding to the three queries, the three queries are concatenated and sent to the customized Transformer decoder to further enable the information between different element queries to flow mutually; S5: For the proxy query and map query output by the decoder, the confidence prediction is first performed through the classification branch, and the ego vehicle query output by the decoder is sent to the classification branch for confidence prediction, and then the ego vehicle query with the highest confidence is sent to the prediction branch to predict the ego vehicle trajectory in the next few seconds of the current frame; wherein, In step S2, when initializing the memory queue of the current frame, the following preprocessing is performed: Step S21: Check the scene mark of the current frame. If it is inconsistent with the scene mark of the previous frame, it means that the scene has been changed, and the memory queue is cleared; Step S22: by acquiring the speed and rotation angle information of the ego vehicle state, the proxy position information in the memory queue is transformed so as to be aligned with the ego vehicle coordinate system in the current frame; Step S23: For the coordinate misalignment caused by the movement of the agent itself, the historical query features are used to generate simulated deviations through the MLN network, so that the agent position information is further aligned with the current frame scene; And wherein, during preprocessing, For query features, initialize a 3×L×D queue of all zeros, where L represents the queue length, D represents the feature dimension, and 3 represents agent query, map query, and ego vehicle query. For location coordinates, initialize 2×L×2, where the coordinates are defined in the ego vehicle coordinate system from a bird's eye view. According to the formula Use the current frame's ego-vehicle state to align the position coordinates in the memory queue, where Refers to the vehicle change matrix from the previous frame to the current frame, represents the position coordinate at time t-1, Represents the position coordinates at time t; Use the current ego vehicle information to transform the agent position information to simulate the agent's spontaneous movement between the historical frame and the current frame Among them, LN represents inter-layer regularization, and the two parameters γ and β are derived from the encoding of the current frame's ego-vehicle transfer matrix, ego-vehicle speed, and time encoding using a multi-layer perceptron.

2. The method according to claim 1, characterized in that In step S1, the original image is passed through the pre-trained residual neural network Resnet and feature pyramid network FPN to obtain multi-view image features.

3. The method according to claim 2, characterized in that In step S1, for N×3×H×W image data, where N represents the number of images, 3 represents the RGB color channels, and H and W represent the height and width of the image, an N×(H×W / 64)×C image tag is generated by using an 8×8 patch size and a convolutional layer, where C represents the patch feature dimension.

4. The method according to claim 3, characterized in that In step S3, the corresponding preprocessing of the agent query, the map query and the vehicle query includes: Step S31: For proxy queries and map queries, multi-head attention interactions are performed with the corresponding historical query features and multi-view image features, and confidence prediction is performed on the output queries through classification branches. The k queries with the highest confidence are selected to enter the next step of interaction; Step S32: For the ego-vehicle query, perform multi-head attention interaction with the ego-vehicle history query and the ego-vehicle state of the current frame.

5. The method according to claim 4, characterized in that In step S3, when initializing the agent query, map query, and ego vehicle query of the current frame, the size of the agent query is A×M×D, where A represents the maximum number of agents in the scene, M represents the possible motion mode of each agent, and D represents the feature dimension. The size of the map query is I×P×D, where I represents the maximum number of map elements, and P represents the number of position coordinates contained in each map element. The size of the ego vehicle query is M×6. And when preprocessing the agent query, map query and self-vehicle query, In order to enhance the temporal awareness of the model, a part of the historical queries in the memory queue are concatenated with the current query, thereby obtaining the temporally enhanced proxy query (A+L)×M×D and the map query (I+L)×P×D, where L represents the number of historical queries read from the memory queue; The three queries are sent to the corresponding decoders for preprocessing. The proxy query and map query will interact with the multi-view image information of size N×(H×W / 64)×C for multi-head attention, while the ego vehicle query will interact with the current state of the ego vehicle of size 9. After the multi-head attention mechanism ends, the output proxy queries and map queries are Top-K screened. The generated queries are sent to the classification branch. According to the confidence predicted by the classification branch, the K queries with the highest confidence are selected to enter the subsequent interaction. The size of the filtered proxy query should be K×M×D, and the size of the map query should be K×P×D.

6. The method according to claim 5, characterized in that In step S4, for the proxy query, map query, and self-vehicle query that have been confidence screened, the corresponding location coordinates are initialized, and the three queries and the corresponding historical query features in the memory queue are concatenated and sent to the decoder together with the corresponding location information. Each layer in the decoder performs the following operations: S41: After position encoding, the position coordinates of size num queries × 2 are concatenated with size num queries × D and sent to the self-attention layer, where num queries = K × (M + P) + M, representing the total number of queries after concatenation of the three queries; S42: The query passes through the forward propagation network FFN and the inter-layer regularization LN sub-layer and then performs multi-head attention interaction with the multi-view image features; S43: For the proxy query and map query outputted between the layers, send them to the regression branch to predict the offset (Δx, Δy) of the position coordinates to correct the original position coordinates of the proxy and map element; S44: Repeat steps S41 to S43 several times so that the query can fully obtain information from multi-view image features.

7. The method according to claim 6, characterized in that In step S42, the multi-view image features are first 3D position encoded and added to the learnable position embedding to model the spatial depth information in the image.

8. An end-to-end autonomous driving planning system, characterized in that: For executing the method according to any one of claims 1 to 7, the system comprises the following modules: An image preprocessing module is used to obtain multi-view image features by preprocessing the original images obtained from the multi-view camera of the car; A memory queue initialization module, used to initialize the memory queue to align the historical information stored in the memory queue with the coordinate system of the current time; The query initialization module is used to initialize the proxy query, map query and ego-vehicle query of the current frame, and pre-process the proxy query, map query and ego-vehicle query step by step to facilitate subsequent learning from multi-view image features; The decoder module is used to initialize the position coordinates corresponding to the three queries, and then concatenate the three queries into the customized Transformer decoder to further enable the information between different element queries to flow mutually; The confidence prediction module is used for the proxy query and map query output by the decoder. First, the confidence prediction is performed through the classification branch, and the ego-vehicle query output by the decoder is sent to the classification branch for confidence prediction. Then, the ego-vehicle query with the highest confidence is sent to the prediction branch to predict the ego-vehicle trajectory in the next few seconds of the current frame.

9. A computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 7 is implemented.