End-to-end automatic driving planning method based on three-dimensional visual language pre-training

Abstract

The invention relates to an end-to-end automatic driving planning method based on three-dimensional visual language pre-training, and the method comprises the following steps: obtaining a multi-view bird's-eye view feature X, and inputting the multi-view bird's-eye view feature X into a BEV encoder in a pre-trained BEV former to calculate a global visual feature; integrating the real description of the environment into a BERT structure-based text attention block to obtain a text representation; calculating grouping alignment loss; inputting the global visual features and the prompt features into a language model, outputting word vectors by the language model, and training a generative end-to-end automatic driving planning model pre-trained based on a three-dimensional visual language; and inputting the actual multi-view image into the automatic driving planning model, and outputting an actual automatic driving planning path. Compared with the prior art, the method has the advantages of improving the real-time performance and generalization ability of path planning and the like.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to an end-to-end autonomous driving planning method based on three-dimensional visual language pre-training. Background Art

[0002] Autonomous driving is a challenging task that requires a deep understanding of the surrounding environment of the autonomous vehicle to ensure safe and efficient real-world deployment. An excellent autonomous driving system must have the ability to fully perceive the driving environment and make accurate planning decisions based on road information. Recently, several end-to-end autonomous driving frameworks have been proposed and achieved promising results by leveraging sensor data to understand driving scenarios and output planning decisions. In addition, autonomous driving tasks have broad application prospects and commercial value in the real world, such as helping to reduce human driving errors and traffic accident rates, supporting the overall planning of smart cities, and reducing logistics costs.

[0003] The autonomous driving task needs to solve three major difficulties: (1) deeply understand the rich information contained in multi-view images, (2) efficiently and accurately perform reasoning and decision-making, and (3) maintain stable performance in a variety of real-world scenarios. Regarding the first difficulty, some early methods directly generate planning trajectories without a thorough understanding of the driving scene, resulting in poor model interpretability and difficulty in training convergence. Planning methods that lack a deep understanding of the scene cannot effectively capture key features in complex environments. Regarding the second difficulty, emerging autonomous driving models that integrate large language models (LLMs) have excellent performance due to their powerful reasoning capabilities. However, these models only introduce two-dimensional visual features and use computationally expensive LLMs for planning and decision-making, resulting in insufficient understanding of three-dimensional visual information, poor real-time performance, and difficulty in meeting efficient processing requirements. Regarding the third difficulty, thanks to pre-training on large-scale datasets, large language models have shown strong generalization capabilities, and new strategies that introduce LLMs have certain adaptability in a variety of scenarios. However, current advanced strategies mainly integrate two-dimensional and three-dimensional visual information, but fail to fully address the consistency problem between three-dimensional visual features and language features. This limitation weakens the generalization performance of the model to a certain extent and limits its application in a wider range of scenarios.

[0004] In summary, the existing autonomous driving vehicle technology has problems with poor real-time performance and generalization ability of path planning. Summary of the invention

[0005] The purpose of the present invention is to provide an end-to-end autonomous driving planning method based on three-dimensional visual language pre-training in order to improve the real-time and generalization ability of path planning.

[0006] An end-to-end autonomous driving planning method based on three-dimensional visual language pre-training, the method comprising the following steps:

[0007] Obtain multi-view bird's-eye view feature X, input the multi-view bird's-eye view feature X into the BEV encoder in the pre-trained BEVformer to extract BEV feature B, input BEV feature B into the detection, motion and map transformer in BEVformer to simultaneously learn the three-dimensional object box, agent motion and map elements, thereby generating constrained detection features, motion features and map features, and encode the detection features, motion features and map features into the same channel dimension based on the visual embedding layer in BEVformer, input the encoded Transformer structure in BEVformer and connect them in series, and output the global visual feature;

[0008] Integrate the real description of the environment into the text attention block F based on the BERT structure tatt To obtain the detected text representation T det , sports text representation T motion , map text representation T map and concatenated text representation T global ;

[0009] Align the detection features, motion features, map features, global visual features, detection text representation, motion text representation, map text representation, and concatenated text representation, and calculate the group alignment loss;

[0010] The global visual features and prompt features are input into the language model. The language model outputs word vectors. The cross entropy loss is calculated based on the word vectors. The overall loss is calculated based on the cross entropy loss, group alignment loss and downstream task loss. The text attention block F is trained based on the overall loss. tatt and language model, pre-trained BEVformer, trained text attention block F tatt The language model forms a generative end-to-end autonomous driving planning model based on 3D visual language pre-training;

[0011] The actual multi-view images are input into the generative end-to-end autonomous driving planning model pre-trained based on 3D visual language, and the actual autonomous driving planning path is output.

[0012] Furthermore, the output of the Transformer structure in BEVformer is:

[0013]

[0014]

[0015]

[0016] Among them, the detection feature F det , motion feature F motion , map feature F map , Embed(·) represents the feature embedding layer, VisAtt(·) represents the visual attention block, d c represents the shared channel dimension, N d 、N o and N m represents the number of bounding boxes, future trajectories, and maps, V det , V motion , V map They are the detection features F det , motion feature F motion , map feature F map Output after inputting the Transformer structure.

[0017] Furthermore, the global visual features are:

[0018]

[0019] Among them, V global is the global visual feature, and Cat(·) represents the concatenation operation.

[0020] Furthermore, the detected text representation T det , sports text representation T motion , map text representation T map for:

[0021]

[0022]

[0023]

[0024] Among them, L det , L motion and L map denote the ground truth labels of foreground objects, future motions, and map elements, respectively, and P d , P o and P m denote the template descriptions of foreground objects, future motions, and map elements, respectively. Embed(·) denotes the feature embedding layer. TxtAtt(·) denotes the BERT-based text attention block. L d , L o and L m is the sentence length.

[0025] Furthermore, the concatenated text representation T global for:

[0026]

[0027] Here, Cat(·) represents a series operation.

[0028] Furthermore, the group alignment loss is:

[0029] L ga =L align (V det ,T det )+L align (V motion ,T motion )+L align (V map ,T map )+L align (V global ,T global )

[0030] Among them, L align represents contrast loss.

[0031] Furthermore, the contrast loss is:

[0032]

[0033] where V, T denotes a set of corresponding visual and textual features, τ is a learnable temperature coefficient, and s(·,·) is a similarity function.

[0034] Furthermore, the global visual features and prompt features are input into the language model, and the language model outputs word vectors. The specific steps of calculating the cross entropy loss based on the word vectors are:

[0035] Obtain two-dimensional scene description and high-level navigation, input global visual features, two-dimensional scene description and high-level navigation into the language model, generate planning features, input the planning features into the linear layer and softmax layer of the language model, generate word vectors, and calculate the cross entropy loss based on the word vectors.

[0036] Furthermore, the word vector is: p(wt|w1:t-1)=Softmax(Linear(F)), where wt represents the tth predicted word and F represents the planning feature.

[0037] Furthermore, the overall loss is:

[0038] L gpvl =L vis +L ga +L cap,

[0039] Among them, L vis Representation loss is relevant to downstream visual tasks, including 3D object detection, map construction, and motion prediction.ga represents the group alignment loss, L cap, represents the cross entropy loss.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention uses the BEV encoder in the pre-trained BEVformer to extract BEV features, and inputs the basic BEV features into the detection, motion and map transformers (Transformer) to simultaneously learn three-dimensional object boxes, agent motion and map elements, thereby generating constrained detection features, motion features and map features. A text attention module is also proposed based on the BERT structure. This module processes perception and planning labels, such as bounding boxes, future trajectories and map elements, and converts them into driving-specific language prompts using predefined templates. The descriptions related to the detection, motion and map features are then input into the text attention block to generate the corresponding language representation. The generated detection, motion and map language text representations and the constrained visual detection features, motion features and map features are input into the group alignment module to project these visual and language features into a shared semantic space. Finally, the aligned visual features, generated scene-level descriptions and navigation prompts, and the cross-modal language model generate future planning decisions in an autoregressive manner by aligning the visual features and text attention blocks generated by the pre-trained BEVformer. tatt The output text features can combine three-dimensional visual features with text features of two-dimensional scene descriptions to produce reliable decisions and trajectories, reduce displacement errors and collision rates of decisions, improve the real-time performance of the output trajectory while improving the generalization of the model, and be able to make stable plans and decisions even in extreme scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the main steps of the present invention;

[0043] Figure 2 It is a generative end-to-end autonomous driving planning method framework based on 3D visual language pre-training, where Figure 2 (a) is Figure 4 , Figure 2 (b) is Figure 5 ;

[0044] Figure 3 The following is a comparison of the framework diagrams of three common methods for end-to-end autonomous driving. Figure 3 (a) is the end-to-end autonomous driving flow chart. Figure 3 (b) is a flowchart of autonomous driving based on a large language model. Figure 3 (c) is a flow chart of the present invention;

[0045] Figure 4 Navigation instruction text prompt templates used for visual-language alignment pre-training and full model training;

[0046] Figure 5 Examples of ways to align visual language;

[0047] Figure 6 Examples of trajectories generated for the model. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0049] A generative end-to-end autonomous driving planning method based on three-dimensional visual language pre-training, which extracts the bird's-eye view (BEV) feature map of multi-view images for processing, and autonomously generates scene-level descriptions. By inputting the aligned three-dimensional perception features, two-dimensional visual descriptions and navigation instructions into the language model, the overall driving decision and fine-grained trajectory are generated in an autoregressive manner. The end-to-end model includes a three-dimensional visual backbone network, a two-dimensional scene description module, a three-dimensional visual language pre-training module, and a cross-modal language model. The three-dimensional visual backbone network generates supervised detection, maps and motion features based on the extracted BEV features, the two-dimensional scene description module generates scene-level descriptions for multi-view images, the three-dimensional visual language pre-training module focuses on aligning visual and language features in a shared semantic space, and the cross-modal language model outputs reliable decisions and trajectories in a generative manner. Compared with the most advanced existing methods, the present invention has the advantages of more accurate trajectory generation, stronger generalization ability, better real-time performance, and higher system security.

[0050] The method of the present invention comprises the following steps:

[0051] Obtain multi-view bird's-eye view feature X, input the multi-view bird's-eye view feature X into the BEV encoder in the pre-trained BEVformer to extract BEV feature B, input BEV feature B into the detection, motion and map transformer in BEVformer to simultaneously learn the three-dimensional object box, agent motion and map elements, thereby generating constrained detection features, motion features and map features, and encode the detection features, motion features and map features into the same channel dimension based on the visual embedding layer in BEVformer, input the encoded Transformer structure in BEVformer and connect them in series, and output the global visual feature;

[0052] Integrate the real description of the environment into the text attention block F based on the BERT structure tatt To obtain the detected text representation Tdet , sports text representation T motion , map text representation T map and concatenated text representation T global ;

[0053] Align the detection features, motion features, map features, global visual features, detection text representation, motion text representation, map text representation, and concatenated text representation, and calculate the group alignment loss;

[0054] The global visual features and prompt features are input into the language model. The language model outputs word vectors. The cross entropy loss is calculated based on the word vectors. The overall loss is calculated based on the cross entropy loss, group alignment loss and downstream task loss. The text attention block F is trained based on the overall loss. tatt and language model, pre-trained BEVformer, trained text attention block F tatt The language model forms a generative end-to-end autonomous driving planning model based on 3D visual language pre-training;

[0055] The actual multi-view images are input into the generative end-to-end autonomous driving planning model pre-trained based on 3D visual language, and the actual autonomous driving planning path is output.

[0056] The flowchart of each of the above steps is as follows Figure 1 As shown, specifically:

[0057] 1) Extract and decode features: First, the basic bird’s-eye view (BEV) features are extracted and decoded into a multimodal data representation containing constraint detection, motion information, and map features.

[0058] 2) Design a 2D scene description model: Design and train a 2D scene description generation model, which is trained and tested on the OmnidrivenuScenes dataset to generate scene-level semantic descriptions.

[0059] 3) Alignment of 3D vision and language features: Based on the 3D vision and language pre-training module, a group alignment mechanism is used to establish associations between visual features and language features, thereby achieving deep fusion of multimodal features.

[0060] 4) Generate future planning decisions: Using the aligned visual features, generated scene-level descriptions and navigation prompts, the cross-modal language model generates future planning decisions in an autoregressive manner, and completes training and testing on the nuScene dataset and Bench2drive dataset to verify its performance and effect. Figure 2 shown.

[0061] The framework diagram of three common methods for end-to-end autonomous driving is as follows: Figure 3 shown. Figure 4 Navigation instruction text prompt templates used for visual-language alignment pre-training and full model training; Figure 5 Examples of ways to align visual language.

[0062] 1), the extraction of BEV feature map includes the extraction of basic bird's-eye view (BEV) features and further processing of visual features. Specifically, assuming Represents the sampled N multi-view images. For the input multi-view image X, we first use the BEV encoder in the pre-trained BEVformer to extract the BEV feature B = F bev (X). B is then fed into the detection, motion, and map transformers to simultaneously learn 3D object boxes, agent motion, and map elements, resulting in constrained detection features F. det , motion characteristics F motion and map feature F map Then, the visual embedding layer is used to transform and Encoded to the same channel dimension d c In which N d 、N o and N m represents the number of bounding boxes, future trajectories, and maps, and d d ,d o and d m Represent the feature dimensions of box, motions and map respectively. Then, the Transformer structure is introduced to establish compact interactions through long-range attention to enhance visual features. The whole function can be expressed as

[0063]

[0064]

[0065]

[0066] where Embed(·) denotes the feature embedding layer, VisAtt(·) denotes the visual attention block, and d c represents the shared channel dimension. Then, attention detection, motion, and map features are integrated along the spatial dimension to obtain the global visual feature, which can be defined as

[0067]

[0068] Where Cat(·) represents a series operation.

[0069] The generation of scene-level descriptions is achieved with the help of a pre-trained BLIP model.

[0070] The extraction and decoding of the bird's-eye view (BEV) features are as follows:

[0071] First, the BEV encoder in the pre-trained BEVformer is used to extract BEV features, and the basic BEV features are input into the detection, motion, and map transformers (Transformer) to simultaneously learn 3D object boxes, agent motion, and map elements, thereby producing constrained detection features, motion features, and map features.

[0072] The design of the two-dimensional scene description model is specifically as follows:

[0073] A 2D scene description model is designed based on the pre-trained BLIP, which is trained and tested on the OminidrivenuScenes dataset to generate scene-level descriptions.

[0074] The three-dimensional visual features are aligned with the language features, specifically:

[0075] A text attention module is proposed based on the BERT structure. This module processes perception and planning labels, such as bounding boxes, future trajectories, and map elements, and converts them into driving-specific language cues using predefined templates. Descriptions related to detection, motion, and map features are then input into the text attention block to generate corresponding language representations. The generated detection, motion, and map language text representations and constrained visual detection features, motion features, and map features are input into the group alignment module to project these visual and language features into a shared semantic space.

[0076] The steps of 2) include:

[0077] 21) First, integrate the true description of the environment into the text attention block F based on the BERT structure tatt to obtain the text representation T det , T motion , T map And its series T global Specifically, a text attention module is proposed using the BERT structure. This module processes perception and planning labels, such as bounding boxes, future trajectories, and map elements, and converts them into driving-specific language cues using predefined templates. Descriptions related to detection, motion, and map features are then input into the text attention block to generate the corresponding language representations. Now, the whole process can be expressed as

[0078]

[0079]

[0080]

[0081] Where L det , L motion and L map P denotes the ground truth labels of foreground objects, future motions, and map elements, respectively. d , P o and P m They represent the template descriptions of foreground objects, future motion, and map elements, respectively. TxtAtt(·) represents the BERT-based text attention block, L d , L o and L m is the sentence length. These descriptive features are then concatenated to form a global text representation, which is defined as

[0082]

[0083] 22) A group alignment module is further designed to project the visual features obtained in S1 and these language features into a shared semantic space. Specifically, four types of 3D visual-language (VL) alignment groups are considered, including detection-related VL groups, motion-related VL groups, map-related VL groups, and global VL groups. For example, in a batch of K training samples, the goal of the alignment function is to regard matching VL samples as positive samples and mismatching VL samples as negative samples. Given visual features X and text features T as input, the contrast loss is defined as

[0084]

[0085] where τ is a learnable temperature coefficient and s(·,·) is a similarity function. The function s(·,·) is obtained by calculating the similarity score between visual features and text features after global averaging. In this module, learnable weights are introduced to explore fine-grained relationships between different representations. For example, the similarity function for detecting related VL groups can be expressed as

[0086]

[0087] in and Therefore, the total group alignment loss is defined as

[0088] L ga =L align (V det ,T det )+L align (V motion ,T motion )+L align (V map ,T map )+L align(V global ,T global )

[0089] Leveraging the aligned 3D visual features, generated scene-level descriptions, and navigation cues, a cross-modal language model generates future planning decisions in an autoregressive manner.

[0090] The proposed 3D visual feature extraction backbone network, 2D scene description module, visual language alignment module, and cross-modal language large model are trained and optimized end-to-end. The training loss function is expressed as:

[0091] L gpvl =L vis +L ga +L cap

[0092] Where L vis represents the loss associated with downstream visual tasks, including 3D object detection, map building, and motion prediction, L ga is the loss caused by aligning 3D visual features with text features, L cap It is the loss in accuracy of planning decisions output by large models describing cross-modal languages.

[0093] The steps of 3) include:

[0094] 31) First, we design the self-agent cross-modal decoder. This module formulates language prompts for the current state of the self-driving car, which includes high-level driving commands and location information. These information prompts are sent to the text embedding layer to produce the initial language representation of the self-driving car. At the same time, Figure 2 As shown in Figure 1, the designed two-dimensional scene description model generates visual descriptions (such as traffic lights, signs, important objects, and weather). Subsequently, the environmental visual features V global The prompt features are input into the language model to learn planning features with rich driving scene and navigation information. The process can be described as (8) where LM(·) represents the pre-trained language model, Cap is the two-dimensional scene description, and Nav represents high-level navigation, with the detailed template being “The box of the ego-car is [cx, cy, cz, cw, ch, cl]. The current one is {command}, which cannot collide with other vehicles or the boundary of the BEV map. Please predict the trajectory at the next p timestamps”. Finally, F is sent to the linear layer and the softmax layer to generate the word vector p(wt|w1:t-1)=Softmax(Linear(F)), where p represents the probability prediction of the vocabulary.

[0095] 32) Then, the cross entropy loss is introduced to output the trajectory in a language modeling manner. During the training process, each sample X is equipped with a prompt describing Gt As a reference, which includes high-level commands and the future trajectory of the ego vehicle, the loss function is expressed as

[0096]

[0097] where θ represents all trainable parameters during training, and w t represents the tth predicted word.

[0098] 33) The overall loss L of the proposed GPVL gpvl It consists of three training parts, including visual perception loss, 3D visual language alignment loss and trajectory description loss, which can be defined as

[0099] L gpvl =L vis +L ga +L cap,

[0100] Where L vis The representation loss is relevant to downstream vision tasks, including 3D object detection, map building, and motion prediction.

[0101] The trajectory example generated by the present invention (GPVL) is as follows Figure 6 shown.

[0102] The method also includes: comprehensively evaluating the performance of the cross-modal language large model based on automatic evaluation indicators and manual evaluation indicators. The automatic evaluation indicators used include two widely used objective indicators, displacement error and collision rate. In addition, Latency and FPS indicators are introduced to evaluate the real-time performance of the model. The manual evaluation indicators used are mainly evaluated from two aspects: generalization and robustness. The details are as follows:

[0103] 51) For automatic evaluation indicators, displacement error and collision rate are referred to as L2 and Collision respectively. L2 measures the distance between the predicted trajectory and the ground truth trajectory, while collision evaluates the frequency of collisions in real driving scenarios. Latency refers to the time delay in the entire model reasoning process, including data transmission, computational reasoning, and result transmission. Lower latency means that the model has better real-time performance. FPS refers to the number of frames transmitted per second, that is, how many frames of images the network can process per second. High FPS means that the model has better real-time performance.

[0104] 52) For manual evaluation indicators, generalization requires that the model can output correct trajectories in a variety of different actual scenarios. This example uses the zero-shot capability of the detection model to detect the generalization of the model; robustness requires that the model can make stable plans and decisions even in extreme scenarios.

[0105] Compared with the prior art, the present invention has the following beneficial effects:

[0106] 1. The present invention proposes a generative end-to-end autonomous driving planning method based on three-dimensional visual language pre-training, which can combine three-dimensional visual features and text features of two-dimensional scene descriptions to generate overall driving decisions and fine-grained trajectories in an autoregressive manner.

[0107] 2. The present invention uses the BEV encoder in the pre-trained BEVformer to extract BEV features, and inputs the basic BEV features into the detection, motion and map transformers (Transformer) to simultaneously learn the three-dimensional object box, agent motion and map elements, thereby generating constrained detection features, motion features and map features.

[0108] 3. This paper designs a two-dimensional scene description model based on pre-trained BLIP, which is trained and tested on the OminidrivenuScenes dataset to generate scene-level descriptions.

[0109] 4. The present invention proposes a text attention module based on the BERT structure. This module processes perception and planning labels, such as bounding boxes, future trajectories, and map elements, and converts them into driving-specific language prompts using predefined templates. Descriptions related to detection, motion, and map features are then input into the text attention block to generate corresponding language representations. The generated detection, motion, and map language text representations and constrained visual detection features, motion features, and map features are input into the group alignment module to project these visual and language features into a shared semantic space.

[0110] In order to verify the performance of the method of this application, the following experiments were designed.

[0111] This example is compared with 8 other advanced non-autoregressive models and 5 autoregressive models. The details are as follows:

[0112] Non-autoregressive models: (1) NMP, an initial story description model based on RNN structure; (2) SA-NMP, a relational attention model using the designed GRU; (3) FF, a hierarchical attention recurrent network; (4) ST-P3, an encoding and decoding model based on LSTM; (5) Ego-MLP, an adversarial reward optimization framework; (6) UniAD, a hierarchical image encoding and decoding model; (7) VAD, a hierarchical reinforcement learning framework; (8) BEV-Planner, a semantic perception mining network;

[0113] Autoregressive models: (9) LLaVA, a relevance text reinforcement learning method; (10) Vicuna, an imaginative content reasoning network; (11) Merlin, a scene graph knowledge enhancement model; (12) Ominidrive, a multi-graph knowledge reasoning framework; (13) Atlas, a 3D visual language trajectory generation model.

[0114] The specific experimental results are shown in Table 1, Table 2 and Table 3.

[0115] Table 1 Performance of different autonomous driving models on the nuScenes dataset (%).

[0116]

[0117] Table 2 Generalization results of the present invention and other autonomous driving models

[0118] The first set of experimental models were trained in Boston and tested in Singapore, and the second set of experimental models were trained in Singapore and tested in Boston.

[0119]

[0120]

[0121] Table 3 Robustness results of the present invention and other autonomous driving models after adding noise.

[0122]

[0123] It can be seen from Tables 1, 2 and 3 that the method of the present invention achieves better results in terms of accuracy, real-time performance, generalization and robustness compared with other autonomous driving models.

[0124] The present invention extracts BEV feature maps of multi-view images and autonomously generates scene-level descriptions. By inputting aligned three-dimensional visual perception features, two-dimensional visual descriptions and navigation instructions into the language model, overall driving decisions and fine-grained trajectories are generated in an autoregressive manner. The end-to-end model includes a three-dimensional visual backbone network, a two-dimensional scene description module, a three-dimensional visual language pre-training module and a cross-modal language model. The three-dimensional visual backbone network generates supervised detection, maps and motion features based on the extracted BEV features. The three-dimensional visual language pre-training module focuses on aligning visual and language features in a shared semantic space. The two-dimensional scene description module generates scene-level descriptions for multi-view images, and the cross-modal language model generates reliable decisions and trajectories in a generative manner.

[0125] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. An end-to-end autonomous driving planning method based on 3D visual language pre-training, characterized in that: The method comprises the following steps: Obtain multi-view bird's-eye view feature X, input the multi-view bird's-eye view feature X into the BEV encoder in the pre-trained BEVformer to extract BEV feature B, input BEV feature B into the detection, motion and map transformer in BEVformer to simultaneously learn the three-dimensional object box, agent motion and map elements, thereby generating constrained detection features, motion features and map features, and encode the detection features, motion features and map features into the same channel dimension based on the visual embedding layer in BEVformer, input the encoded Transformer structure in BEVformer and connect them in series, and output the global visual feature; Integrate the real description of the environment into the text attention block F based on the BERT structure tatt To obtain the detected text representation T det , sports text representation T motion , map text representation T map and concatenated text representation T global ; Align the detection features, motion features, map features, global visual features, detection text representation, motion text representation, map text representation, and concatenated text representation, and calculate the group alignment loss; The global visual features and prompt features are input into the language model. The language model outputs word vectors. The cross entropy loss is calculated based on the word vectors. The overall loss is calculated based on the cross entropy loss, group alignment loss and downstream task loss. The text attention block F is trained based on the overall loss. tatt and language model, pre-trained BEVformer, trained text attention block F tatt The language model forms a generative end-to-end autonomous driving planning model based on 3D visual language pre-training; The actual multi-view images are input into the generative end-to-end autonomous driving planning model pre-trained based on 3D visual language, and the actual autonomous driving planning path is output.

2. The end-to-end autonomous driving planning method based on three-dimensional visual language pre-training according to claim 1, characterized in that: The output of the Transformer structure in BEVformer is: Among them, the detection feature F det , motion feature F motion , map feature F map , Embed(·) represents the feature embedding layer, VisAtt(·) represents the visual attention block, d c represents the shared channel dimension, N d 、N o and N m represents the number of bounding boxes, future trajectories, and maps, V det , V motion , V map They are the detection features F det , motion feature F motion , map feature F map Output after inputting the Transformer structure.

3. The end-to-end autonomous driving planning method based on three-dimensional visual language pre-training according to claim 2 is characterized in that: The global visual features are: Among them, V global is the global visual feature, and Cat(·) represents the concatenation operation.

4. The end-to-end autonomous driving planning method based on three-dimensional visual language pre-training according to claim 3 is characterized in that: Detect text representation T det , sports text representation T motion , map text representation T map for: Among them, L det , L motion and L map denote the ground truth labels of foreground objects, future motions, and map elements, respectively, and P d , P o and P m denote the template descriptions of foreground objects, future motions, and map elements, respectively. Embed(·) denotes the feature embedding layer. TxtAtt(·) denotes the BERT-based text attention block. L d , L o and L m is the sentence length.

5. The end-to-end autonomous driving planning method based on three-dimensional visual language pre-training according to claim 4, characterized in that: Concatenation text representation T global for: Here, Cat(·) represents a series operation.

6. The end-to-end autonomous driving planning method based on three-dimensional visual language pre-training according to claim 5, characterized in that: The group alignment loss is: L ga =K align (V det ,T det )+L align (V motion ,T motion )+L align (V map ,T map )+L align (V global ,T global ) Among them, L align represents contrast loss.

7. The end-to-end autonomous driving planning method based on three-dimensional visual language pre-training according to claim 6, characterized in that: The contrast loss is: where V, T denotes a set of corresponding visual and textual features, τ is a learnable temperature coefficient, and s(·,·) is a similarity function.

8. The end-to-end autonomous driving planning method based on three-dimensional visual language pre-training according to claim 1, characterized in that: The global visual features and prompt features are input into the language model, and the language model outputs word vectors. The specific steps for calculating the cross entropy loss based on the word vectors are: Obtain two-dimensional scene description and high-level navigation, input global visual features, two-dimensional scene description and high-level navigation into the language model, generate planning features, input the planning features into the linear layer and softmax layer of the language model, generate word vectors, and calculate the cross entropy loss based on the word vectors.

9. The end-to-end autonomous driving planning method based on three-dimensional visual language pre-training according to claim 8, characterized in that: The word vector is: p(wt|w1:t-1)=Softmax(Linear(F)), wt represents the tth predicted word, and F represents the planning feature.

10. The end-to-end autonomous driving planning method based on three-dimensional visual language pre-training according to claim 9, characterized in that: The overall loss is: L gpvl =L vis +L ga +L cap, Among them, L vis Representation loss is relevant to downstream visual tasks, including 3D object detection, map construction, and motion prediction. ga represents the group alignment loss, L cap, represents the cross entropy loss.

Images