End-to-end autonomous driving planning method based on 3D visual language pre-training

Abstract

The present invention relates to an end-to-end autonomous driving planning method based on 3D visual language pre-training. The method comprises the following steps: obtaining multi-perspective bird's-eye view features X, inputting these features into a pre-trained BEV encoder in a BEVformer to calculate global visual features; integrating a true description of the environment into a text attention block based on the BERT architecture to obtain a text representation; calculating a group alignment loss; inputting the global visual features and cue features into a language model, which outputs word vectors to train a generative end-to-end autonomous driving planning model based on 3D visual language pre-training; and inputting actual multi-perspective images into the autonomous driving planning model to output the actual planned autonomous driving path. Compared with existing technologies, this method has the advantages of improving the real-time and generalization capabilities of path planning.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to an end-to-end autonomous driving planning method based on three-dimensional visual language pre-training. Background Art

[0002] Autonomous driving is a challenging task that requires a deep understanding of the vehicle's surroundings to ensure safe and efficient real-world deployment. An excellent autonomous driving system must be able to fully perceive the driving environment and make accurate planning decisions based on road information. Recently, several end-to-end autonomous driving frameworks have been proposed, leveraging sensor data to understand driving scenarios and output planning decisions, achieving promising results. Furthermore, autonomous driving has broad real-world application prospects and commercial value, such as helping to reduce human driving errors and traffic accident rates, supporting the overall planning of smart cities, and lowering logistics costs.

[0003] Autonomous driving tasks require solving three major challenges: (1) deeply understanding the rich information contained in multi-view images, (2) efficiently and accurately performing reasoning and decision-making, and (3) maintaining stable performance in a variety of real-world scenarios. Regarding the first challenge, some early methods directly generated planning trajectories without a thorough understanding of the driving scene, resulting in poor model interpretability and difficulty in training convergence. Planning methods that lack a deep understanding of the scene cannot effectively capture key features in complex environments. Regarding the second challenge, emerging autonomous driving models that integrate large language models (LLMs) have performed well due to their powerful reasoning capabilities. However, these models only introduce two-dimensional visual features and use computationally expensive LLMs for planning and decision-making, resulting in insufficient understanding of three-dimensional visual information, poor real-time performance, and difficulty meeting efficient processing requirements. Regarding the third challenge, thanks to pre-training on large-scale datasets, large language models have demonstrated strong generalization capabilities, and new strategies that introduce LLMs have certain adaptability in a variety of scenarios. However, current advanced strategies mainly integrate two-dimensional and three-dimensional visual information, but fail to fully address the consistency problem between three-dimensional visual features and language features. This limitation weakens the generalization performance of the model to a certain extent and limits its application in a wider range of scenarios.

[0004] In summary, existing autonomous driving vehicle technology has problems with poor real-time performance and generalization ability of path planning. Summary of the Invention

[0005] The purpose of this invention is to provide an end-to-end autonomous driving planning method based on three-dimensional visual language pre-training in order to improve the real-time and generalization capabilities of path planning.

[0006] An end-to-end autonomous driving planning method based on three-dimensional visual language pre-training, the method comprising the following steps:

[0007] Obtain multi-view bird's-eye view features X, input the multi-view bird's-eye view features X into the BEV encoder in the pre-trained BEVformer to extract BEV features B, and input BEV features B into the detection, motion, and map transformers in the BEVformer to simultaneously learn 3D object boxes, agent motion, and map elements, thereby generating constrained detection features, motion features, and map features. Based on the visual embedding layer in the BEVformer, the detection features, motion features, and map features are encoded into the same channel dimension. After encoding, they are input into the Transformer structure in the BEVformer and connected in series to output global visual features.

[0008] Integrating the real description of the environment into the text attention block F based on the BERT structure tatt To obtain the detected text representation T det , motion text representation T motion , map text representation T map and concatenated text representation T global ;

[0009] Align the detection features, motion features, map features, global visual features, and the detection text representation, motion text representation, map text representation, and concatenated text representation, and calculate the group alignment loss;

[0010] The global visual features and prompt features are input into the language model, and the language model outputs word vectors. The cross entropy loss is calculated based on the word vectors. The overall loss is calculated based on the cross entropy loss, group alignment loss and downstream task loss. The text attention block F is trained based on the overall loss. tatt And language model, pre-trained BEVformer, trained text attention block F tatt Together with the language model, it forms a generative end-to-end autonomous driving planning model based on 3D visual language pre-training.

[0011] The actual multi-view images are input into a generative end-to-end autonomous driving planning model pre-trained based on 3D visual language, and the actual autonomous driving planning path is output.

[0012] Furthermore, the output of the Transformer structure in BEVformer is:

[0013]

[0014]

[0015]

[0016] Among them, the detection feature F det , motion feature F motion , map feature F map , Embed(·) represents the feature embedding layer, VisAtt(·) represents the visual attention block, d c Indicates the shared channel dimension, N d 、N o and N m represents the number of bounding boxes, future trajectories, and maps, V det , V motion , V map They are detection features F det , motion feature F motion , map feature F map Output after inputting the Transformer structure.

[0017] Furthermore, the global visual features are:

[0018]

[0019] Among them, V global is the global visual feature, and Cat(·) represents the concatenation operation.

[0020] Furthermore, the text representation T det , motion text representation T motion , map text representation T map for:

[0021]

[0022]

[0023]

[0024] Among them, L det 、L motion and L map denote the ground truth labels of foreground objects, future motions, and map elements, respectively, and P d 、P o and P m They represent the template descriptions of foreground objects, future motions, and map elements, respectively. Embed(·) represents the feature embedding layer. TxtAtt(·) represents the BERT-based text attention block. L d 、L o and L m is the sentence length.

[0025] Furthermore, the concatenated text representation T global for:

[0026]

[0027] Here, Cat(·) represents a series operation.

[0028] Furthermore, the group alignment loss is:

[0029] L ga =L align (V det ,T det )+L align (V motion ,T motion )+L align (V map ,T map )+L align (V global ,T global )

[0030] Among them, L align represents contrast loss.

[0031] Furthermore, the contrast loss is:

[0032]

[0033] where V, T denotes a set of corresponding visual and textual features, τ is a learnable temperature coefficient, and s(·,·) is a similarity function.

[0034] Furthermore, the global visual features and prompt features are input into the language model, and the language model outputs word vectors. The specific steps for calculating the cross entropy loss based on the word vectors are as follows:

[0035] Obtain two-dimensional scene description and high-level navigation, input global visual features, two-dimensional scene description and high-level navigation into the language model to generate planning features, input the planning features into the linear layer and softmax layer of the language model to generate word vectors, and calculate the cross entropy loss based on the word vectors.

[0036] Furthermore, the word vector is: p(wt|w1:t-1)=Softmax(Linear(F)), where wt represents the t-th predicted word and F represents the planning feature.

[0037] Furthermore, the overall loss is:

[0038] L gpvl =L vis +L ga +L cap,

[0039] Among them, L vis Representation loss is relevant to downstream visual tasks, including 3D object detection, map construction, and motion prediction.ga represents the group alignment loss, L cap, represents the cross entropy loss.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention uses the BEV encoder in the pre-trained BEVformer to extract BEV features, and inputs the basic BEV features into the detection, motion and map transformer (Transformer) to simultaneously learn three-dimensional object boxes, agent motion and map elements, thereby generating constrained detection features, motion features and map features. A text attention module is also proposed based on the BERT structure. This module processes perception and planning labels such as bounding boxes, future trajectories and map elements, and converts them into driving-specific language prompts using predefined templates. The descriptions related to the detection, motion and map features are then input into the text attention block to generate the corresponding language representation. The generated detection, motion and map language text representations and the constrained visual detection features, motion features and map features are input into the group alignment module to project these visual and language features into a shared semantic space. Finally, the aligned visual features, generated scene-level descriptions and navigation prompts, and the cross-modal language model generate future planning decisions in an autoregressive manner by aligning the visual features and text attention block F generated by the pre-trained BEVformer. tatt The output text features can combine three-dimensional visual features with text features of two-dimensional scene descriptions to produce reliable decisions and trajectories, reduce displacement errors and collision rates in decisions, improve the real-time performance of the output trajectory, and improve the generalization of the model, enabling stable planning and decision-making even in extreme scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic flow chart of the main steps of the present invention;

[0043] Figure 2 It is a generative end-to-end autonomous driving planning method framework based on 3D visual language pre-training, Figure 2 (a) is Figure 4 , Figure 2 (b) is Figure 5 ;

[0044] Figure 3 This is a framework diagram comparison of three common methods for end-to-end autonomous driving. Figure 3 (a) is the end-to-end autonomous driving flowchart. Figure 3 (b) is a flowchart of autonomous driving based on a large language model. Figure 3 (c) is a flow chart of the present invention;

[0045] Figure 4 Navigation instruction text prompt templates used for vision-language alignment pre-training and full model training;

[0046] Figure 5 Examples of ways to align visual language;

[0047] Figure 6 Examples of trajectories generated for the model. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0049] A generative end-to-end autonomous driving planning method based on three-dimensional visual language pre-training, which extracts the bird's-eye view (BEV) feature map of multi-view images for processing, and autonomously generates scene-level descriptions. By inputting the aligned three-dimensional perception features, two-dimensional visual descriptions and navigation instructions into the language model, the overall driving decision and fine-grained trajectory are generated in an autoregressive manner. The end-to-end model includes a three-dimensional visual backbone network, a two-dimensional scene description module, a three-dimensional visual language pre-training module, and a cross-modal language model. The three-dimensional visual backbone network generates supervised detection, maps and motion features based on the extracted BEV features, the two-dimensional scene description module generates scene-level descriptions for multi-view images, the three-dimensional visual language pre-training module focuses on aligning visual and language features in a shared semantic space, and the cross-modal language model outputs reliable decisions and trajectories in a generative manner. Compared with the existing state-of-the-art methods, the present invention has the advantages of more accurate trajectory generation, stronger generalization ability, better real-time performance and higher system security.

[0050] The method of the present invention comprises the following steps:

[0051] Obtain multi-view bird's-eye view features X, input the multi-view bird's-eye view features X into the BEV encoder in the pre-trained BEVformer to extract BEV features B, and input BEV features B into the detection, motion, and map transformers in the BEVformer to simultaneously learn 3D object boxes, agent motion, and map elements, thereby generating constrained detection features, motion features, and map features. Based on the visual embedding layer in the BEVformer, the detection features, motion features, and map features are encoded into the same channel dimension. After encoding, they are input into the Transformer structure in the BEVformer and connected in series to output global visual features.

[0052] Integrating the real description of the environment into the text attention block F based on the BERT structure tatt To obtain the detected text representation Tdet , motion text representation T motion , map text representation T map and concatenated text representation T global ;

[0053] Align the detection features, motion features, map features, global visual features, and the detection text representation, motion text representation, map text representation, and concatenated text representation, and calculate the group alignment loss;

[0054] The global visual features and prompt features are input into the language model, and the language model outputs word vectors. The cross entropy loss is calculated based on the word vectors. The overall loss is calculated based on the cross entropy loss, group alignment loss and downstream task loss. The text attention block F is trained based on the overall loss. tatt And language model, pre-trained BEVformer, trained text attention block F tatt Together with the language model, it forms a generative end-to-end autonomous driving planning model based on 3D visual language pre-training.

[0055] The actual multi-view images are input into a generative end-to-end autonomous driving planning model pre-trained based on 3D visual language, and the actual autonomous driving planning path is output.

[0056] The flowchart of each of the above steps is as follows Figure 1 As shown, specifically:

[0057] 1) Feature extraction and decoding: First, the basic bird’s-eye view (BEV) features are extracted and decoded into a multimodal data representation containing constraint detection, motion information, and map features.

[0058] 2) Design a 2D scene description model: Design and train a 2D scene description generation model, train and test it on the OmnidrivenuScenes dataset to generate scene-level semantic descriptions.

[0059] 3) Alignment of 3D visual and language features: Based on the 3D visual language pre-training module, a group alignment mechanism is used to establish associations between visual and language features, achieving deep fusion of multimodal features.

[0060] 4) Generate future planning decisions: Using the aligned visual features, generated scene-level descriptions and navigation prompts, the cross-modal language model generates future planning decisions in an autoregressive manner, and completes training and testing on the nuScene dataset and Bench2drive dataset to verify its performance and effectiveness. Figure 2 shown.

[0061] The framework diagram of three common methods for end-to-end autonomous driving is as follows: Figure 3 shown. Figure 4 Navigation instruction text prompt templates used for vision-language alignment pre-training and full model training; Figure 5 Examples of ways to align visual language.

[0062] 1), the extraction of BEV feature maps includes the extraction of basic bird's-eye view (BEV) features and further processing of visual features. Specifically, assuming Represents the sampled N multi-view images. For the input multi-view image X, we first use the BEV encoder in the pre-trained BEVformer to extract the BEV feature B = F bev (X). B is then fed into the detection, motion, and map transformers to simultaneously learn 3D object boxes, agent motion, and map elements, resulting in constrained detection features F det , motion characteristics F motion and map feature F map Then, the visual embedding layer is used to transform and Encoded to the same channel dimension d c In which N d 、N o and N m represents the number of bounding boxes, future trajectories, and maps, and d d d o and d m Represent the feature dimensions of box, motions and map respectively. Then, the transformer structure is introduced to establish a compact interaction through long-range attention to enhance visual features. The whole function can be expressed as

[0063]

[0064]

[0065]

[0066] Where Embed(·) represents the feature embedding layer, VisAtt(·) represents the visual attention block, and d c represents the shared channel dimension. Then, attention detection, motion, and map features are integrated along the spatial dimension to obtain the global visual feature, which can be defined as

[0067]

[0068] Where Cat(·) represents a series operation.

[0069] The generation of scene-level descriptions is achieved with the help of a pre-trained BLIP model.

[0070] The extraction and decoding of bird's-eye view (BEV) features are as follows:

[0071] First, the BEV encoder in the pre-trained BEVformer is used to extract BEV features, and the basic BEV features are input into the detection, motion and map transformers (Transformer) to simultaneously learn 3D object boxes, agent motion and map elements, thereby producing constrained detection features, motion features and map features.

[0072] The design of the two-dimensional scene description model is specifically as follows:

[0073] A 2D scene description model is designed based on the pre-trained BLIP, which is trained and tested on the OminidrivenuScenes dataset to generate scene-level descriptions.

[0074] The three-dimensional visual features are aligned with the language features, specifically:

[0075] A text attention module is proposed based on the BERT architecture. This module processes perception and planning labels, such as bounding boxes, future trajectories, and map elements, and converts them into driving-specific language cues using predefined templates. Descriptions associated with detection, motion, and map features are then fed into the text attention block to generate corresponding language representations. The generated detection, motion, and map language textual representations and constrained visual detection features, motion features, and map features are fed into a group alignment module to project these visual and language features into a shared semantic space.

[0076] The steps of 2) include:

[0077] 21) First, the real description of the environment is integrated into the text attention block F based on the BERT structure tatt To obtain the text representation T det 、T motion 、T map and its series T global Specifically, a text attention module is proposed using the BERT architecture. This module processes perception and planning labels, such as bounding boxes, future trajectories, and map elements, and converts them into driving-specific language cues using predefined templates. Descriptions related to detection, motion, and map features are then input into the text attention block to generate corresponding language representations. Now, the entire process can be expressed as

[0078]

[0079]

[0080]

[0081] Among them L det 、L motion and L map P denotes the ground truth labels for foreground objects, future motions, and map elements, respectively. d 、P o and P m They represent the template descriptions of foreground objects, future motions, and map elements, respectively. TxtAtt(·) represents the BERT-based text attention block, L d 、L o and L m is the sentence length. Subsequently, these descriptive features are concatenated to form a global text representation, which is defined as

[0082]

[0083] 22) A group alignment module is further designed to project the visual features obtained in S1 and these language features into a shared semantic space. Specifically, four types of 3D visual-language (VL) alignment groups are considered, including detection-related VL groups, motion-related VL groups, map-related VL groups, and global VL groups. For example, in a batch of K training samples, the goal of the alignment function is to regard matching VL samples as positive samples and mismatching VL samples as negative samples. Given visual features X and text features T as input, the contrast loss is defined as

[0084]

[0085] Where τ is a learnable temperature coefficient and s(·,·) is a similarity function. The function s(·,·) is obtained by calculating the similarity score between visual features and text features after global averaging. In this module, learnable weights are introduced to explore fine-grained relationships between different representations. For example, the similarity function for detecting related VL groups can be expressed as

[0086]

[0087] in and Therefore, the total group alignment loss is defined as

[0088] L ga =L align (V det ,T det )+L align (V motion ,T motion )+L align (V map ,T map )+L align(V global ,T global )

[0089] Leveraging the aligned 3D visual features, generated scene-level descriptions, and navigation cues, a cross-modal language model generates future planning decisions in an autoregressive manner.

[0090] The proposed 3D visual feature extraction backbone network, 2D scene description module, visual language alignment module, and cross-modal language large model are trained and optimized end-to-end. The training loss function is expressed as:

[0091] L gpvl =L vis +L ga +L cap

[0092] Among them L vis represents the loss associated with downstream vision tasks, including 3D object detection, map building, and motion prediction, L ga is the loss caused by the alignment of 3D visual features and text features, L cap It is the loss in accuracy of planning decisions output by a large model describing cross-modal language.

[0093] The steps of 3) include:

[0094] 31) First, we design a self-agent cross-modal decoder. This module generates language prompts for the current state of the autonomous vehicle, including high-level driving commands and location information. These prompts are sent to the text embedding layer to produce the initial language representation of the autonomous vehicle. Figure 2 As shown in Figure 2, the designed two-dimensional scene description model generates visual descriptions (such as traffic lights, signs, important objects and weather). Subsequently, the environmental visual features V global The and prompt features are input into the language model to learn planning features with rich driving scene and navigation information. The process can be expressed as (8) where LM(·) represents the pre-trained language model, Cap is the two-dimensional scene description, and Nav represents high-level navigation. Its detailed template is “The box of the ego-car is [cx, cy, cz, cw, ch, cl]. The current one is {command}, which cannot collide with other vehicles or the boundary of the BEV map. Please predict the trajectory at the next p timestamps.” Finally, F is sent to the linear layer and the softmax layer to generate the word vector p(wt|w1:t-1)=Softmax(Linear(F)), where p represents the probability prediction of the vocabulary.

[0095] 32) Then, the cross entropy loss is introduced to output the trajectory in a language modeling manner. During the training process, each sample X is equipped with a prompt description Gt As a reference, which includes high-level commands and the future trajectory of the ego vehicle, the loss function is expressed as

[0096]

[0097] Where θ represents all trainable parameters during training, w t represents the t-th predicted word.

[0098] 33) The overall loss L of the proposed GPVL gpvl It consists of three training parts, including visual perception loss, 3D visual language alignment loss and trajectory description loss, which can be defined as

[0099] L gpvl =L vis +L ga +L cap,

[0100] Among them L vis The representation loss is relevant to downstream vision tasks, including 3D object detection, map building, and motion prediction.

[0101] The trajectory example generated by the present invention (GPVL) is as follows Figure 6 shown.

[0102] The method also includes: comprehensively evaluating the performance of the cross-modal language large model based on automatic evaluation indicators and manual evaluation indicators. The automatic evaluation indicators used include two widely used objective indicators: displacement error and collision rate. In addition, latency and FPS indicators are introduced to evaluate the real-time performance of the model. The manual evaluation indicators used are mainly evaluated from the two aspects of generalization and robustness. The details are as follows:

[0103] 51) For automatic evaluation metrics, displacement error and collision rate are referred to as L2 and Collision, respectively. L2 measures the distance between the predicted trajectory and the ground truth trajectory, while Collision evaluates the frequency of collisions in real-world driving scenarios. Latency refers to the time delay in the entire model reasoning process, including data transmission, computational reasoning, and result transmission. Lower latency means the model has better real-time performance. FPS refers to the number of frames transmitted per second, that is, how many frames of images the network can process per second. A high FPS means the model has better real-time performance.

[0104] 52) For manual evaluation indicators, generalization requires that the model can output correct trajectories in a variety of different actual scenarios. This example uses the zero-shot capability of the detection model to test the generalization of the model; robustness requires that the model can make stable plans and decisions even in extreme scenarios.

[0105] Compared with the prior art, the present invention has the following beneficial effects:

[0106] 1. This paper proposes a generative end-to-end autonomous driving planning method based on 3D visual language pre-training, which can combine 3D visual features and text features of 2D scene descriptions to generate overall driving decisions and fine-grained trajectories in an autoregressive manner.

[0107] 2. The present invention uses the BEV encoder in the pre-trained BEVformer to extract BEV features, and inputs the basic BEV features into the detection, motion and map transformers (Transformer) to simultaneously learn the 3D object box, agent motion and map elements, thereby generating constrained detection features, motion features and map features.

[0108] 3. This paper designs a two-dimensional scene description model based on pre-trained BLIP, which is trained and tested on the OminidrivenuScenes dataset to generate scene-level descriptions.

[0109] Fourth, based on the BERT architecture, this paper proposes a text attention module. This module processes perception and planning labels, such as bounding boxes, future trajectories, and map elements, and converts them into driving-specific language cues using predefined templates. Descriptions associated with detection, motion, and map features are then input into the text attention block to generate corresponding language representations. The generated detection, motion, and map language text representations and constrained visual detection features, motion features, and map features are input into a group alignment module to project these visual and language features into a shared semantic space.

[0110] In order to verify the performance of the method of this application, the following experiments were designed.

[0111] This example compares eight other advanced non-autoregressive models and five autoregressive models. The details are as follows:

[0112] Non-autoregressive models: (1) NMP, an initial story description model based on RNN structure; (2) SA-NMP, a relational attention model using the designed GRU; (3) FF, a hierarchical attention recurrent network; (4) ST-P3, an LSTM-based encoding and decoding model; (5) Ego-MLP, an adversarial reward optimization framework; (6) UniAD, a hierarchical image encoding and decoding model; (7) VAD, a hierarchical reinforcement learning framework; (8) BEV-Planner, a semantic perception mining network;

[0113] Autoregressive models: (9) LLaVA, a relevance-based text reinforcement learning method; (10) Vicuna, an imaginative content reasoning network; (11) Merlin, a scene graph knowledge enhancement model; (12) Ominidrive, a multi-graph knowledge reasoning framework; (13) Atlas, a 3D visual language trajectory generation model.

[0114] The specific experimental results are shown in Table 1, Table 2 and Table 3.

[0115] Table 1 Performance (%) of different autonomous driving models on the nuScenes dataset.

[0116]

[0117] Table 2 Generalization results of the present invention and other autonomous driving models

[0118] The first set of experimental models were trained in Boston and tested in Singapore, and the second set of experimental models were trained in Singapore and tested in Boston.

[0119]

[0120]

[0121] Table 3 Robustness results of the present invention and other autonomous driving models after adding noise.

[0122]

[0123] It can be seen from Tables 1, 2, and 3 that the method of the present invention achieves better results in terms of accuracy, real-time performance, generalization, and robustness compared to other autonomous driving models.

[0124] The present invention extracts BEV feature maps from multi-view images and autonomously generates scene-level descriptions. By inputting aligned 3D visual perception features, 2D visual descriptions, and navigation instructions into a language model, it generates overall driving decisions and fine-grained trajectories in an autoregressive manner. The end-to-end model includes a 3D visual backbone network, a 2D scene description module, a 3D visual language pre-training module, and a cross-modal language model. The 3D visual backbone network generates supervised detection, maps, and motion features based on the extracted BEV features. The 3D visual language pre-training module focuses on aligning visual and language features in a shared semantic space. The 2D scene description module generates scene-level descriptions for multi-view images. The cross-modal language model generates reliable decisions and trajectories in a generative manner.

[0125] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. An end-to-end autonomous driving planning method based on 3D visual language pre-training, characterized by: The method comprises the following steps: Obtain multi-view bird's-eye view features X, input the multi-view bird's-eye view features X into the BEV encoder in the pre-trained BEVformer to extract BEV features B, and input BEV features B into the detection, motion, and map transformers in the BEVformer to simultaneously learn 3D object boxes, agent motion, and map elements, thereby generating constrained detection features, motion features, and map features. Based on the visual embedding layer in the BEVformer, the detection features, motion features, and map features are encoded into the same channel dimension. After encoding, they are input into the Transformer structure in the BEVformer and connected in series to output global visual features. Integrating the real description of the environment into the text attention block F based on the BERT structure tatt To obtain the detected text representation T det , motion text representation T motion , map text representation T map and concatenated text representation T global ; Align the detection features, motion features, map features, global visual features, and the detection text representation, motion text representation, map text representation, and concatenated text representation, and calculate the group alignment loss; The global visual features and prompt features are input into the language model, and the language model outputs word vectors. The cross entropy loss is calculated based on the word vectors. The overall loss is calculated based on the cross entropy loss, group alignment loss and downstream task loss. The text attention block F is trained based on the overall loss. tatt And language model, pre-trained BEVformer, trained text attention block F tatt Together with the language model, it forms a generative end-to-end autonomous driving planning model based on 3D visual language pre-training. The actual multi-view images are input into a generative end-to-end autonomous driving planning model pre-trained based on 3D visual language, and the actual autonomous driving planning path is output.

2. The end-to-end autonomous driving planning method based on 3D visual language pre-training according to claim 1, characterized in that: The output of the Transformer structure in BEVformer is: Among them, the detection feature F det , motion feature F motion , map feature F map , Embed(·) represents the feature embedding layer, VisAtt(·) represents the visual attention block, d c Indicates the shared channel dimension, N d 、N o and N m represents the number of bounding boxes, future trajectories, and maps, V det , V motion , V map They are detection features F det , motion feature F motion , map feature F map Output after inputting the Transformer structure.

3. The end-to-end autonomous driving planning method based on 3D visual language pre-training according to claim 2, characterized in that: The global visual features are: Among them, V global is the global visual feature, and Cat(·) represents the concatenation operation.

4. The end-to-end autonomous driving planning method based on 3D visual language pre-training according to claim 3, characterized in that: Detect text representation T det , motion text representation T motion , map text representation T map for: Among them, L det 、L motion and L map denote the ground truth labels of foreground objects, future motions, and map elements, respectively, and P d 、P o and P m They represent the template descriptions of foreground objects, future motions, and map elements, respectively. Embed(·) represents the feature embedding layer. TxtAtt(·) represents the BERT-based text attention block. L d 、L o and L m is the sentence length.

5. The end-to-end autonomous driving planning method based on 3D visual language pre-training according to claim 4, characterized in that: Concatenated text representation T global for: Here, Cat(·) represents a series operation.

6. The end-to-end autonomous driving planning method based on 3D visual language pre-training according to claim 5, characterized in that: The group alignment loss is: L ga =K align (V det ,T det )+L align (V motion ,T motion )+L align (V map ,T map ) +L align (V global ,T global ) Among them, L align represents contrast loss.

7. The end-to-end autonomous driving planning method based on 3D visual language pre-training according to claim 6, characterized in that: The contrast loss is: where V, T denotes a set of corresponding visual and textual features, τ is a learnable temperature coefficient, and s(·,·) is a similarity function.

8. The end-to-end autonomous driving planning method based on 3D visual language pre-training according to claim 1, characterized in that: The global visual features and prompt features are input into the language model, and the language model outputs word vectors. The specific steps for calculating the cross entropy loss based on the word vectors are: Obtain two-dimensional scene description and high-level navigation, input global visual features, two-dimensional scene description and high-level navigation into the language model to generate planning features, input the planning features into the linear layer and softmax layer of the language model to generate word vectors, and calculate the cross entropy loss based on the word vectors.

9. The end-to-end autonomous driving planning method based on 3D visual language pre-training according to claim 8, characterized in that: The word vector is: p(wt|w1:t-1)=Softmax(Linear(F)), wt represents the t-th predicted word, and F represents the planning feature.

10. The end-to-end autonomous driving planning method based on 3D visual language pre-training according to claim 9, characterized in that: The overall loss is: L gpvl =L vis +L ga +L cap, Among them, L vis Representation loss is relevant to downstream visual tasks, including 3D object detection, map construction, and motion prediction. ga represents the group alignment loss, L cap, represents the cross entropy loss.

Images