Driving behavior labeling method and device based on end-to-end autonomous driving large model

Abstract

The application provides a driving behavior labeling method and device based on an end-to-end automatic driving large model, wherein the method comprises: acquiring multi-view images of a current driving scene; acquiring multiple groups of query vectors corresponding to the multi-view images based on a pre-trained end-to-end automatic driving large model; acquiring a prompt word text, and acquiring a labeling result of a current driving behavior according to the prompt word text and the multiple groups of query vectors. The method can realize accurate and effective labeling of automatic driving behavior by taking the multiple groups of query vectors output by multiple functional modules in the end-to-end automatic driving large model as one of the triggering conditions for driving behavior labeling, instead of only taking images as the triggering condition for driving behavior labeling, and the labeling result and the intermediate output result of the automatic driving large model have high consistency, thereby improving the labeling accuracy and interpretability of the automatic driving behavior.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and apparatus for labeling driving behavior based on an end-to-end autonomous driving large model. Background Technology

[0002] Traditional end-to-end autonomous driving models exhibit a "black box" characteristic when making driving behavior decisions, meaning passengers cannot understand the reasons behind the decisions and only receive the direct result. To achieve interpretability of these decisions, natural language explanations are needed. Deep learning-based image-language multimodal models have been used for autonomous driving behavior annotation, but most rely directly on images as input for inference. This results in the loss of intermediate information such as reasoning and decision-making, making it impossible to analyze the underlying reasons for the model's decisions and thus failing to truly achieve interpretability for autonomous driving models. Furthermore, due to their large number of parameters, a significant amount of data is often required for training to obtain a high-quality model.

[0003] Therefore, it is essential to address the problems of lack of interpretability, low accuracy, and poor effectiveness in existing driving behavior labeling methods. Summary of the Invention

[0004] This invention provides a driving behavior annotation method and apparatus based on an end-to-end autonomous driving large model, which overcomes the shortcomings of existing driving behavior annotation methods such as lack of interpretability, low annotation accuracy, and poor effect, and achieves accurate and effective annotation of autonomous driving behavior.

[0005] On one hand, the present invention provides a driving behavior annotation method based on an end-to-end autonomous driving big model, including: acquiring multi-view images of the current driving scene; acquiring multiple sets of query vectors corresponding to the multi-view images based on a pre-trained end-to-end autonomous driving big model; acquiring prompt text, and obtaining the annotation result of the current driving behavior according to the prompt text and the multiple sets of query vectors.

[0006] Furthermore, the step of obtaining the annotation result of the current driving behavior based on the prompt text and the multiple sets of query vectors includes: encoding the prompt text to obtain a text encoding vector; preprocessing and transforming the multiple sets of query vectors to obtain a query encoding vector; and obtaining the annotation result of the current driving behavior based on the text encoding vector and the query encoding vector.

[0007] Further, the preprocessing and transformation of the multiple sets of query vectors to obtain query encoding vectors includes: adjusting the dimensions of the multiple sets of query vectors and concatenating the adjusted multiple sets of query vectors to obtain an initial query encoding vector; performing a multi-head cross-attention mechanism on the initial query encoding vector based on a pre-trained query vector converter to obtain an intermediate query encoding vector; and transforming the intermediate query encoding vector to obtain the query encoding vector based on a pre-trained adaptor.

[0008] Further, obtaining the annotation result of the current driving behavior based on the text encoding vector and the query encoding vector includes: concatenating the text encoding vector and the query encoding vector to obtain a model input vector; and inputting the model input vector into a pre-trained multimodal large language model to obtain the annotation result of the current driving behavior.

[0009] Furthermore, it also includes: training and optimizing the end-to-end autonomous driving big model separately; freezing the model parameters of the end-to-end autonomous driving big model and training and optimizing the query vector converter and adaptor separately; and jointly training the end-to-end autonomous driving big model, query vector converter and adaptor to obtain a pre-trained end-to-end autonomous driving big model, query vector converter and adaptor.

[0010] Furthermore, the end-to-end autonomous driving big data model includes a feature extraction module, a tracking module, a trajectory prediction module, and a driving decision module. Correspondingly, the process of obtaining multiple sets of query vectors corresponding to the multi-view images based on the pre-trained end-to-end autonomous driving big data model includes: extracting bird's-eye view query vectors from the multi-view images based on the feature extraction module; obtaining tracking query vectors of objects in the multi-view images based on the tracking module and the bird's-eye view query vectors; obtaining trajectory prediction query vectors of objects in the multi-view images based on the trajectory prediction module and the bird's-eye view query vectors; and obtaining driving decision query vectors for a specific number of steps the vehicle will take in the future based on the trajectory prediction query vectors. The multiple sets of query vectors include the bird's-eye view query vector, the tracking query vector, the trajectory prediction query vector, and the driving decision query vector.

[0011] Furthermore, the step of obtaining the annotation result of the current driving behavior includes: adjusting the driving trajectory prediction output by the driving decision module in the end-to-end autonomous driving big model in reverse based on the annotation result of the current driving behavior.

[0012] Secondly, the present invention also provides a driving behavior annotation device based on an end-to-end autonomous driving big model, comprising: a multi-view image acquisition module for acquiring multi-view images of the current driving scene; a query vector acquisition module for acquiring multiple sets of query vectors corresponding to the multi-view images based on a pre-trained end-to-end autonomous driving big model; and a driving behavior annotation module for acquiring prompt text and, based on the prompt text and the multiple sets of query vectors, acquiring the annotation result of the current driving behavior.

[0013] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the driving behavior annotation method based on the end-to-end autonomous driving large model as described above.

[0014] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the driving behavior annotation method based on an end-to-end autonomous driving large model as described above.

[0015] This invention provides a driving behavior annotation method based on an end-to-end autonomous driving large-scale model. It acquires multi-view images of the current driving scene and, based on a pre-trained end-to-end autonomous driving large-scale model, obtains multiple sets of query vectors corresponding to the multi-view images. This leads to the acquisition of prompt text, and based on the prompt text and multiple sets of query vectors, the annotation result of the current driving behavior is obtained. This method uses multiple sets of query vectors output by multiple functional modules in the end-to-end autonomous driving large-scale model as one of the trigger conditions for driving behavior annotation, rather than solely using images. This enables accurate and effective annotation of autonomous driving behavior, and the annotation results show a high degree of consistency with the intermediate output results of the autonomous driving large-scale model, thereby improving the annotation accuracy and interpretability of autonomous driving behavior. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the driving behavior annotation method based on an end-to-end autonomous driving large model provided in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the model structure of the end-to-end autonomous driving large model provided in the embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the first stage of the three-stage joint training provided in this embodiment of the invention.

[0020] Figure 4 This is a schematic diagram of the second stage of the three-stage joint training provided in this embodiment of the invention.

[0021] Figure 5 This is a schematic diagram of the third stage of the three-stage joint training provided in this embodiment of the invention.

[0022] Figure 6 This is a schematic diagram of vehicle trajectory prediction optimization provided in an embodiment of the present invention.

[0023] Figure 7 This is a schematic diagram of the overall process of the driving behavior annotation method based on the end-to-end autonomous driving large model provided in the embodiments of the present invention.

[0024] Figure 8 This is a schematic diagram of the driving behavior annotation device based on an end-to-end autonomous driving large model provided in an embodiment of the present invention.

[0025] Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] It's important to note that most existing autonomous driving behavior annotation methods directly use images as model input for inference, which results in the loss of intermediate information such as reasoning and decision-making. Furthermore, the annotation process does not employ a multimodal large language model. These two factors contribute to the poor annotation performance of existing autonomous driving behavior annotation methods.

[0028] In view of this, the present invention proposes a driving behavior annotation method based on an end-to-end autonomous driving large model, specifically, Figure 1 The diagram illustrates a flowchart of the driving behavior annotation method based on an end-to-end autonomous driving large model provided in an embodiment of the present invention.

[0029] like Figure 1 As shown, the method includes steps S11-S130, which will be described in detail below.

[0030] S110: Acquire multi-view images of the current driving scene.

[0031] Understandably, during autonomous driving, multiple surround-view cameras can be used to collect multi-view images of the current driving scene, which can then be used as input for subsequent end-to-end autonomous driving models.

[0032] Among them, the multi-view images are RGB images from multiple perspectives.

[0033] The number of surround-view cameras corresponds to the number of multi-view images and can be set according to actual needs; no specific limit is set here.

[0034] The current driving scenario refers to the scenario in which the vehicle makes decisions during autonomous driving, including pedestrians, other vehicles, and static environmental objects.

[0035] S120: Based on a pre-trained end-to-end autonomous driving large model, obtain multiple sets of query vectors corresponding to the multi-view images.

[0036] Understandably, based on obtaining the multi-view images of the current driving scene in step S110, step S120 will further utilize the pre-trained end-to-end autonomous driving big model to obtain multiple sets of query vectors corresponding to the multi-view images.

[0037] The end-to-end autonomous driving big model, or UniAD for short, consists of four perception and prediction modules based on Transformer decoders and one planning module. It integrates three main tasks (perception, prediction, and planning) and six sub-tasks (object detection, object tracking, scene mapping, trajectory prediction, grid prediction, and path planning) into a unified end-to-end network framework.

[0038] Specifically, a series of multi-view images are input into the feature extractor and converted into a unified bird's-eye view (BEV) using BEVFormer. This part can be quickly replaced with other BEV models, demonstrating good scalability.

[0039] In the perception phase, the target detection and tracking module of the end-to-end autonomous driving big model can extract features from dynamic elements and track objects between frames; the online mapping module can extract features from static objects and predict instance-level maps.

[0040] In the prediction module, the end-to-end autonomous driving big model can realize the interaction of dynamic and static elements and long-term trajectory prediction; the occupancy grid prediction module can realize short-term full-scenario BEV and instance-level prediction.

[0041] In the planning module, the end-to-end autonomous driving big model realizes trajectory prediction based on vehicle query vectors and collision optimization based on occupied grids.

[0042] In other words, the end-to-end autonomous driving big model includes multiple functional modules. When the model input (multi-view images) is obtained, multiple sets of different query vectors can be obtained through the multiple functional modules in the end-to-end autonomous driving big model for subsequent annotation.

[0043] It should be noted that the overall structure and training process of the end-to-end autonomous driving big model will be described in the embodiments below, and will not be detailed here.

[0044] S130, obtain the prompt text, and obtain the annotation result of the current driving behavior based on the prompt text and the multiple sets of query vectors.

[0045] Understandably, in step S120, based on the pre-trained end-to-end autonomous driving large model, and after obtaining multiple sets of query vectors corresponding to the multi-view images, step S130 will further obtain the annotation results of the current driving behavior based on the obtained prompt text and multiple sets of query vectors.

[0046] Specifically, first, obtain the prompt text used to generate driving behavior annotations. The prompt text can be customized according to the actual situation, and no specific limitations are made here.

[0047] For example, in one specific embodiment, the prompt text includes a labeling request and a corresponding labeling example. For instance, the labeling request could be: "Based on the given bird's-eye view, object recognition, trajectory prediction, and driving decision information, please provide a description of the driving behavior and the reasons for it."

[0048] Then, based on the obtained prompt text and multiple sets of query vectors, the annotation results of the current driving behavior are obtained. Specifically, this can be achieved through a multimodal large language model. That is, the prompt text and multiple sets of query vectors are used as input to the multimodal large language model to output the annotation results of the current driving behavior.

[0049] It is worth mentioning that since multiple query vectors are not data types that the multimodal large language model can directly process, a series of preprocessing and transformation processes are required before they are input into the multimodal large language model. The specific steps will be described in the following examples.

[0050] In one specific embodiment, the multimodal large language model used is the LLaMA multimodal large language model.

[0051] The labeling result of the current driving behavior includes a description of the driving behavior and the corresponding reason. For example, in a specific embodiment, the driving behavior is described as "the car is moving forward", and the corresponding reason is "because there are no cars on the road ahead".

[0052] In this embodiment, multi-view images of the current driving scene are acquired, and multiple sets of query vectors corresponding to the multi-view images are obtained based on a pre-trained end-to-end autonomous driving model. Then, prompt text is obtained, and the annotation result of the current driving behavior is obtained based on the prompt text and the multiple sets of query vectors. This method uses multiple sets of query vectors output by multiple functional modules in the end-to-end autonomous driving model as one of the trigger conditions for driving behavior annotation, rather than using only images as the trigger condition. This enables accurate and effective annotation of autonomous driving behavior, and the annotation results are highly consistent with the intermediate output results of the autonomous driving model, thereby improving the annotation accuracy and interpretability of autonomous driving behavior.

[0053] Based on the above embodiments, the following will further describe in detail the process of obtaining multiple sets of query vectors corresponding to multi-view images using an end-to-end autonomous driving big model.

[0054] The end-to-end autonomous driving big model includes a feature extraction module, a tracking module, a trajectory prediction module, and a driving decision module.

[0055] Accordingly, based on the pre-trained end-to-end autonomous driving large model, multiple sets of query vectors corresponding to multi-view images are obtained, including: extracting bird's-eye view query vectors from multi-view images based on the feature extraction module; obtaining tracking query vectors of objects in multi-view images based on the tracking module according to the bird's-eye view query vectors; obtaining trajectory prediction query vectors of objects in multi-view images based on the trajectory prediction module according to the bird's-eye view query vectors and tracking query vectors; and obtaining driving decision query vectors for a specific number of steps in the future based on the driving decision module. Among these, the multiple sets of query vectors include bird's-eye view query vectors, tracking query vectors, trajectory prediction query vectors, and driving decision query vectors.

[0056] Specifically, Figure 2 The diagram shows a schematic representation of the model structure of the end-to-end autonomous driving large-scale model provided in an embodiment of the present invention.

[0057] like Figure 2As shown, firstly, the bird's-eye view query vector is extracted from the multi-view images based on the feature extraction module. Specifically, the multi-view images (Multi-view Vision-only Input) captured by multiple surround-view cameras are first converted into image features by the feature extractor, and then the BEV (Bird's-Eye-View) encoder converts the image features into BEV features, i.e., the bird's-eye view query vector. The bird's-eye view query vector includes information from the bird's-eye view images.

[0058] Then, based on the tracking module, tracking query vectors for objects in the multi-view images are obtained from the bird's-eye view query vector. Specifically, in TrackFormer (the tracking module), a set of tracking query vectors is used to detect newly emerging objects (agents) from the BEV features and to continuously track already detected objects. The tracking query vectors include the detected object's location, trajectory, and type information.

[0059] MapFormer (the map segmentation module) segments map elements of different categories, such as lane lines and sidewalks, using map query vectors.

[0060] MotionFormer (the trajectory prediction module) predicts the future trajectory of each object by modeling the interaction between the object and its environment, thus obtaining a trajectory prediction query vector. Simultaneously, a vehicle query vector (Sdc query) is introduced to model the vehicle's motion; this vehicle query vector will be used in subsequent vehicle planning.

[0061] OccFormer (Occupancy Grid Prediction Module) is used to predict occupancy grids. It uses BEV features as the query vector and object features as the key and value, continuously updates the future BEV features, and then decodes them into occupancy grids.

[0062] The Planner module (i.e., the driving decision module) decodes the vehicle query vector, generates a planned path, and optimizes the path using occupancy grid prediction to avoid obstacles, thereby obtaining the driving decision query vector.

[0063] The above describes the process of obtaining multiple sets of query vectors corresponding to multi-view images using an end-to-end autonomous driving big model.

[0064] In this embodiment, multiple sets of query vectors corresponding to multi-view images are obtained based on a pre-trained end-to-end autonomous driving model, and then prompt text is obtained. Based on the prompt text and multiple sets of query vectors, the annotation result of the current driving behavior is obtained. This method uses multiple sets of query vectors output by multiple functional modules in the end-to-end autonomous driving model as one of the trigger conditions for driving behavior annotation, rather than using only images as the trigger condition. This enables accurate and effective annotation of autonomous driving behavior, and the annotation results are highly consistent with the intermediate output results of the autonomous driving model, thereby improving the annotation accuracy and interpretability of autonomous driving behavior.

[0065] Based on the above embodiments, the following will further describe in detail the process of obtaining the annotation results of the current driving behavior based on the prompt text and multiple sets of query vectors.

[0066] Based on the prompt text and multiple sets of query vectors, the annotation results of the current driving behavior are obtained, including: encoding the prompt text to obtain a text encoding vector; preprocessing and transforming multiple sets of query vectors to obtain a query encoding vector; and obtaining the annotation results of the current driving behavior based on the text encoding vector and the query encoding vector.

[0067] Understandably, after obtaining the prompt text, a text encoder is used to encode the prompt text to obtain the corresponding text encoding vector, which is then used as input to the model along with the query encoding vector.

[0068] Simultaneously, the acquired multiple sets of query vectors undergo preprocessing and transformation to obtain query encoding vectors. Specifically, firstly, the dimensions of the multiple sets of query vectors are adjusted, and the adjusted sets of query vectors are concatenated to obtain initial query encoding vectors; then, based on a pre-trained query vector transformer, a multi-head cross-attention mechanism is applied to the initial query encoding vectors to obtain intermediate query encoding vectors; finally, based on a pre-trained adapter, the intermediate query encoding vectors are transformed to obtain the final query encoding vector.

[0069] In one specific embodiment, the adapter is LLaMA-adapter-v2.

[0070] To elaborate, the multiple query vectors include bird's-eye view query vectors, tracking query vectors, trajectory prediction query vectors, and driving decision query vectors.

[0071] The bird's-eye view query vector has a dimension of (B, bev_w*bev_h, query_dim) and contains information from the bird's-eye view. The tracking query vector has a dimension of (B, num_agents, query_dim) and contains information about the detected object's location, trajectory, and type. The trajectory prediction query vector has a dimension of (num_decoder, B, num_agents, num_mode, query_dim) and contains multi-step trajectory prediction information obtained from decoders at different layers. The driving decision query vector has a dimension of (B, planning_steps, 2) and contains a prediction of the vehicle's trajectory for a specific number of steps in the future.

[0072] Then, the multiple sets of query vectors are preprocessed.

[0073] Specifically, regarding the bird's-eye view query vector In this case, it is downsampled to make its dimension (B, bev_w / 5*bev_h / 5, query_dim).

[0074] Regarding trajectory prediction query vector In this case, the output of the last layer decoder of the trajectory prediction module is selected, and it is transformed into a vector of dimension (B, num_agents, query_dim) by performing average pooling on its step dimension.

[0075] Then, the average pooled trajectory prediction query vector and tracking query vector are... The vectors are passed through a multilayer perceptron (MLP) network to obtain a vector of dimension (B, num_agents, query_dim). .

[0076] .

[0077] in, Represents the tracking query vector. This represents the trajectory prediction query vector after average pooling. This indicates vector concatenation.

[0078] Regarding driving decision query vector In other words, it is made through position encoding A multilayer perceptron network is used to transform the query into a vector of dimension (B, query_dim). The position encoding uses triangular encoding.

[0079] .

[0080] Then, the dimensions of the preprocessed query vectors are adjusted to accommodate the subsequent large language model input.

[0081] Specifically, the query vector obtained through preprocessing , , Each vector passes through a multi-layer perceptron network, resulting in a vector with the last dimension being `embed_dim`. These three vectors are then concatenated to obtain the initial query encoding vector. This is used as input for large language models.

[0082] .

[0083] Following this, based on a pre-trained query vector converter, and a trainable... We perform a multi-head cross-attention mechanism (WHCA) to extract features and obtain the intermediate query encoding vector. .

[0084] .

[0085] Finally, based on the pre-trained adaptor, the intermediate query encoding vector is combined with the trainable... Add them together to obtain the final query encoding vector. .

[0086] .

[0087] Based on the above, we can obtain the text encoding vector corresponding to the prompt word text, as well as the query encoding vector corresponding to multiple sets of query vectors.

[0088] Furthermore, based on the text encoding vector and the query encoding vector, the annotation results of the current driving behavior are obtained.

[0089] Specifically, first, the text encoding vector and the query encoding vector are concatenated to obtain the model input vector; then, the model input vector is input into a pre-trained multimodal large language model to obtain the annotation result of the current driving behavior.

[0090] In one specific embodiment, the multimodal large language model is the LLaMA-7B multimodal large language model.

[0091] In this embodiment, the prompt text is encoded to obtain a text encoding vector, and multiple sets of query vectors are preprocessed and transformed to obtain a query encoding vector. Then, based on the text encoding vector and the query encoding vector, the annotation result of the current driving behavior is obtained. This method uses multiple sets of query vectors output by multiple functional modules in the end-to-end autonomous driving model as one of the trigger conditions for driving behavior annotation, instead of using only images as the trigger condition. This enables accurate and effective annotation of autonomous driving behavior, and the annotation results are highly consistent with the intermediate output results of the autonomous driving model, thereby improving the annotation accuracy and interpretability of autonomous driving behavior.

[0092] Based on the above embodiments, the training process of the end-to-end autonomous driving large model, multi-layer perceptron network and adaptor involved in the above embodiments will be described in detail below.

[0093] Specifically, first, the end-to-end autonomous driving large model is trained and optimized separately; then, the model parameters of the end-to-end autonomous driving large model are frozen, and the query vector converter and adaptor are trained and optimized separately; finally, the end-to-end autonomous driving large model, query vector converter and adaptor are trained together to obtain the pre-trained end-to-end autonomous driving large model, query vector converter and adaptor.

[0094] It is understood that, for the driving behavior annotation method based on the end-to-end autonomous driving large model provided by the present invention, this embodiment specifically proposes a three-stage joint training method.

[0095] Specifically, this training process covers two stages: 1) training the end-to-end autonomous driving large model separately; 2) freezing the end-to-end autonomous driving large model and training the query vector converter and adaptor; 3) freezing the multimodal large language model and jointly training the end-to-end autonomous driving large model, query vector converter and adaptor.

[0096] Figure 3 This diagram illustrates the first stage of the three-stage joint training provided in an embodiment of the present invention. Figure 3 As shown, this stage trains the end-to-end autonomous driving large model (UniAD) separately. The parameters in the training include the feature extraction backbone, the bird's-eye view processing framework BevFormer, the object tracking head TackFormer, the map drawing framework MapFormer, the trajectory prediction framework MotionFormer, the occupancy prediction framework OccFormer, and the decision head PlanningHead.

[0097] Figure 4 This diagram illustrates the second stage of the three-stage joint training provided in an embodiment of the present invention. Figure 4 As shown, the model parameters of the end-to-end autonomous driving big model are frozen at this stage, and the bird's-eye view query vectors generated during its inference process are... Tracking query vectors Trajectory prediction query vector and driving decision query vector The data is fed into the query vector converter and adaptor, and the text output is supervised using the calibrated Nuscenes-X dataset. The parameters used in training include those of the query vector converter, adaptor, multiple multilayer perceptron networks, and embedding network provided in the examples above. The parameters of the pre-trained multimodal large language model LLaMA-V2 are frozen.

[0098] Figure 4 The Caption header includes the query vector converter, adaptor, multiple multilayer perceptron networks, and embedding network provided in the above embodiments.

[0099] Figure 5 This diagram illustrates the third stage of the three-stage joint training provided in an embodiment of the present invention. Figure 5 As shown, this stage trains the parameters of all modules except the feature extraction framework backbone in the end-to-end autonomous driving big model. Furthermore, the annotation results output by the multimodal big language model are passed to the driving decision module in the end-to-end autonomous driving big model. The driving decision module further optimizes the driving trajectory prediction to achieve the effect of mutual promotion between decision-making and annotation.

[0100] The loss function applicable to the three-stage joint training process can be selected according to the actual situation. For example, in a specific embodiment, the cross-entropy loss function is used.

[0101] Based on the above embodiments, after obtaining the annotation results of the current driving behavior, the method further includes: adjusting the driving trajectory prediction output by the driving decision module in the end-to-end autonomous driving big model in reverse according to the annotation results of the current driving behavior.

[0102] Understandably, after obtaining the labeling results of the current driving behavior through the multimodal large language model, the labeling results of the current driving behavior will be transmitted to the driving decision module of the end-to-end autonomous driving large model. Then, the driving decision module will optimize the driving trajectory prediction based on the labeling results, so as to achieve the effect of mutual promotion between decision-making and labeling.

[0103] For example, Figure 6 A schematic diagram illustrating the vehicle trajectory prediction optimization provided by an embodiment of the present invention is shown. Figure 6As shown, facing an open road ahead, the end-to-end autonomous driving big model (UniAD) provides a curved trajectory (see the second image from the left in the third row). If the multiple sets of query vectors output by the end-to-end autonomous driving big model (including the aforementioned trajectory prediction) are transmitted to the query vector converter and the adaptor, after a series of processing steps, the annotation result "maintain current speed" output by the multimodal big language model is obtained.

[0104] With the help of the output of the multimodal large language model, the end-to-end autonomous driving large model will give a smoother driving prediction trajectory (see the second one from the left in the first row).

[0105] In this embodiment, by adjusting the driving trajectory prediction output by the driving decision module in the end-to-end autonomous driving big model in reverse according to the annotation results of the current driving behavior after obtaining the annotation results of the current driving behavior, the accuracy of autonomous driving decision can be effectively improved.

[0106] in addition, Figure 7 This diagram illustrates the overall process of the driving behavior annotation method based on an end-to-end autonomous driving large model provided in an embodiment of the present invention.

[0107] like Figure 7 As shown, firstly, based on the end-to-end autonomous driving big model (UniAD), multiple sets of query vectors corresponding to multi-view images are obtained, which respectively convey perception, reasoning (prediction), and decision-making information.

[0108] Then, based on the query vector converter and the adaptor, a series of preprocessing and transformation processes are performed on these multiple sets of query vectors to obtain the query encoded vector.

[0109] At the same time, the input text, i.e. the prompt text, is obtained and encoded to obtain the text encoding vector.

[0110] Furthermore, the text encoding vector and the query encoding vector are used together as model inputs to the multimodal large language model to obtain the annotation results corresponding to the current driving behavior.

[0111] Corresponding to the driving behavior annotation method based on the end-to-end autonomous driving large model described in the above embodiments, the present invention also provides a driving behavior annotation device based on the end-to-end autonomous driving large model.

[0112] Specifically, Figure 8 A schematic diagram of the driving behavior annotation device based on an end-to-end autonomous driving large model provided in an embodiment of the present invention is shown.

[0113] like Figure 8As shown, the device includes: a multi-view image acquisition module 810, used to acquire multi-view images of the current driving scene; a query vector acquisition module 820, used to acquire multiple sets of query vectors corresponding to the multi-view images based on a pre-trained end-to-end autonomous driving large model; and a driving behavior annotation module 830, used to acquire prompt word text and, based on the prompt word text and the multiple sets of query vectors, acquire annotation results of the current driving behavior.

[0114] In this embodiment, the multi-view image acquisition module 810 acquires multi-view images of the current driving scene, the query vector acquisition module 820 acquires multiple sets of query vectors corresponding to the multi-view images based on a pre-trained end-to-end autonomous driving model, and then the driving behavior annotation module 830 acquires the prompt text and obtains the annotation result of the current driving behavior based on the prompt text and the multiple sets of query vectors. This device uses multiple sets of query vectors output by multiple functional modules in the end-to-end autonomous driving model as one of the trigger conditions for driving behavior annotation, rather than solely using images as the trigger condition. This enables accurate and effective annotation of autonomous driving behavior, and the annotation results are highly consistent with the intermediate output results of the autonomous driving model, thereby improving the annotation accuracy and interpretability of autonomous driving behavior.

[0115] It should be noted that the driving behavior annotation device based on the end-to-end autonomous driving large model provided in this embodiment can be referred to in correspondence with the driving behavior annotation method based on the end-to-end autonomous driving large model described in the above embodiments, and will not be repeated here.

[0116] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, communications interface 920, and memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a driving behavior annotation method based on an end-to-end autonomous driving model. This method includes: acquiring multi-view images of the current driving scene; acquiring multiple sets of query vectors corresponding to the multi-view images based on a pre-trained end-to-end autonomous driving model; acquiring prompt text; and obtaining the annotation result of the current driving behavior based on the prompt text and the multiple sets of query vectors.

[0117] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the driving behavior annotation method based on the end-to-end autonomous driving big model provided by the above methods. The method includes: acquiring multi-view images of the current driving scene; acquiring multiple sets of query vectors corresponding to the multi-view images based on a pre-trained end-to-end autonomous driving big model; acquiring prompt word text, and acquiring the annotation result of the current driving behavior according to the prompt word text and the multiple sets of query vectors.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driving behavior labeling method based on an end-to-end autonomous driving large model, characterized in that, include: Acquire multi-view images of the current driving scene; Based on a pre-trained end-to-end autonomous driving big model, multiple sets of query vectors corresponding to the multi-view images are obtained. Obtaining the prompt text and, based on the prompt text and the multiple sets of query vectors, obtaining the annotation result of the current driving behavior, includes: encoding the prompt text to obtain a text encoding vector; preprocessing and transforming the multiple sets of query vectors to obtain a query encoding vector; and obtaining the annotation result of the current driving behavior based on the text encoding vector and the query encoding vector. The multiple sets of query vectors include bird's-eye view query vectors, tracking query vectors, trajectory prediction query vectors, and driving decision query vectors.

2. The driving behavior labeling method based on an end-to-end autonomous driving large model according to claim 1, wherein The preprocessing and transformation of the multiple sets of query vectors to obtain query encoding vectors includes: Adjust the dimensions of the multiple sets of query vectors, and concatenate the adjusted multiple sets of query vectors to obtain the initial query encoding vector; Based on a pre-trained query vector converter, a multi-head cross-attention mechanism is applied to the initial query encoding vector to obtain an intermediate query encoding vector; Based on the pre-trained adaptor, the query encoding vector is obtained by transforming the intermediate query encoding vector.

3. The driving behavior labeling method based on the end-to-end autonomous driving large model according to claim 1, wherein, The step of obtaining the annotation result of the current driving behavior based on the text encoding vector and the query encoding vector includes: The text encoding vector and the query encoding vector are concatenated to obtain the model input vector; The model input vector is fed into a pre-trained multimodal large language model to obtain the annotation results of the current driving behavior.

4. The driving behavior labeling method based on the end-to-end autonomous driving large model according to claim 2, characterized in that, Also includes: The end-to-end autonomous driving large model is trained and optimized separately. Freeze the model parameters of the end-to-end autonomous driving big model and train and optimize the query vector converter and adaptor separately; The end-to-end autonomous driving big model, query vector converter, and adaptor are jointly trained to obtain a pre-trained end-to-end autonomous driving big model, query vector converter, and adaptor.

5. The driving behavior labeling method based on the end-to-end autonomous driving large model according to claim 1, wherein The end-to-end autonomous driving big model includes a feature extraction module, a tracking module, a trajectory prediction module, and a driving decision module; Accordingly, the pre-trained end-to-end autonomous driving big model obtains multiple sets of query vectors corresponding to the multi-view images, including: Based on the feature extraction module, the bird's-eye view query vector is extracted from the multi-view image; Based on the tracking module, the tracking query vector of the object in the multi-view image is obtained according to the bird's-eye view query vector; Based on the trajectory prediction module, the trajectory prediction query vector of the object in the multi-view image is obtained according to the bird's-eye view query vector and the tracking query vector; Based on the driving decision module, the driving decision query vector for a specific number of steps in the future is obtained according to the trajectory prediction query vector.

6. The driving behavior labeling method based on the end-to-end autonomous driving large model according to any one of claims 1-5, characterized in that, The process of obtaining the annotation results of the current driving behavior includes: Based on the annotation results of the current driving behavior, the driving trajectory prediction output by the driving decision module in the end-to-end autonomous driving big model is adjusted in reverse.

7. An apparatus for labeling driving behavior based on an end-to-end autonomous driving large model, characterized in that, include: The multi-view image acquisition module is used to acquire multi-view images of the current driving scene; The query vector acquisition module is used to acquire multiple sets of query vectors corresponding to the multi-view images based on a pre-trained end-to-end autonomous driving big model. The driving behavior annotation module is used to obtain prompt word text and, based on the prompt word text and the multiple sets of query vectors, obtain the annotation result of the current driving behavior, including: encoding the prompt word text to obtain a text encoding vector; preprocessing and transforming the multiple sets of query vectors to obtain a query encoding vector; and obtaining the annotation result of the current driving behavior based on the text encoding vector and the query encoding vector. The multiple sets of query vectors include bird's-eye view query vectors, tracking query vectors, trajectory prediction query vectors, and driving decision query vectors.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the driving behavior annotation method based on the end-to-end autonomous driving large model as described in any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the driving behavior annotation method based on the end-to-end autonomous driving large model as described in any one of claims 1 to 6.

Images