Automatic driving interpretation text determination method based on large visual language model

Abstract

The invention relates to the technical field of automatic driving, and particularly discloses an automatic driving interpretation text determination method based on a large visual language model, and the method comprises the steps: obtaining a multi-view camera image, a BEV map image and a text query template; video features in the multi-view camera image are extracted, and image features in the BEV map image are extracted; aligning the image features with the video features based on a modal encoder; encoding the text query template, generating a text mark, mapping the aligned image features and video features to a text embedding space based on a shared projector, generating a visual mark, and inputting the visual mark and the text mark into an LLM backbone model; comprehensive three-dimensional dynamic / static situation perception and visual representation unification of a local scene and a global map are realized, the 3D LVLM architecture has excellent three-dimensional space understanding and dynamic time reasoning capabilities, and the accuracy and safety of end-to-end automatic driving are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for determining autonomous driving interpretation text based on a large visual language model. Background Art

[0002] Recently, the rapid development of Large Vision-Language Models (LVLMs) and their excellent commonsense reasoning and generalization capabilities have triggered a series of applications in end-to-end autonomous driving. End-to-end autonomous driving aims to directly predict the vehicle's control signals and / or planned paths by using on-board sensor data as input. However, the primary challenge in achieving end-to-end autonomous driving is scene understanding, which involves navigating in complex, changing, and globally coordinated 3D scenes and directly affects the vehicle's future state prediction, driving behavior decision-making, and safe interaction with the environment. Therefore, LVLM needs to expand its capabilities from two-dimensional understanding to comprehensive three-dimensional dynamic / static situational awareness, unifying the local scene with the visual representation of the global map to fully unleash its potential in real-world applications. Although previous studies have demonstrated examples of successful applications of LLMs in end-to-end autonomous driving, a holistic approach is still needed to extend the application of LVLMs to complex real-world driving scenarios.

[0003] Another urgent challenge to be solved in end-to-end autonomous driving is the interpretability of the decision-making process. End-to-end autonomous driving integrates full-stack driving components into a comprehensive framework, eliminating the cumulative errors, insufficient coordination and suboptimal resource utilization of discontinuous intermediate steps. However, it operates in a manner similar to a "black box", which means that the decision-making process lacks intuitive interpretability and may lead to widespread public distrust and legal concerns. Some methods rely on visual maps as a means to explain the system's decisions, or make decisions by providing meaningful intermediate representations, but visual maps or intermediate representations are often difficult to understand for drivers and passengers. In addition, some methods also convert complex decision-making processes into easy-to-understand natural language texts through LVLM, thereby providing a new level of explanation for traditional systems, but they generally only consider the potential factors of the scene and ignore the textual behavior description of the vehicle's future control command sequence. Summary of the invention

[0004] The purpose of the present invention is to provide a method for determining autonomous driving interpretation text based on a large visual language model to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for determining an explanation text for autonomous driving based on a large visual language model, the method comprising: Obtain multi-view camera images, BEV map images, and text query templates; Extract video features from multi-view camera images based on a video encoder, and extract image features from BEV map images based on an image encoder; Align image features with video features based on a modality encoder; Encode the text query template to generate text tags, and map the aligned image features and video features to the text embedding space based on the shared projector to generate visual tags; The visual tags and text tags are input into the LLM backbone model, and the autonomous driving explanation text is output.

[0006] As a further solution of the present invention, the text query template includes a historical control signal template and a task instruction template.

[0007] As a further solution of the present invention, the shared projector is a two-layer sensor.

[0008] As a further solution of the present invention, it also includes: constructing a video image question and answer benchmark based on multi-view camera images and BEV map images, and the video image question and answer benchmark includes a 3D scene understanding benchmark and an explainable end-to-end driving benchmark.

[0009] As a further solution of the present invention, the steps of constructing the 3D scene understanding benchmark specifically include: Determine the 3D scene layering system based on multi-view camera images and BEV map images; Determine question options and answer templates based on the 3D scene hierarchical system; Generate scene understanding question and answer pairs based on GPT-4V; As a further solution of the present invention, the steps of constructing the explainable end-to-end driving benchmark include: Acquire control signal sequences based on multi-view camera images and BEV map images; determining a threshold vector, and determining an ego-vehicle action based on the threshold vector; ChatGPT is used to generate explanation text based on the vehicle's meta-actions, control signal sequences, and scene understanding question-answer pairs.

[0010] As a further solution of the present invention, it also includes: Align image features and video features in the 2D spatial domain with the embedding space of a large language model; 3D fine-tuning of 3D scene understanding benchmarks; End-to-end fine-tuning on the explainable end-to-end driving benchmark.

[0011] As a further solution of the present invention, the step of aligning the image features and video features in the 2D spatial domain with the embedding space of the large language model includes: The weights of the video encoder, image encoder, and LLM backbone are kept frozen, and only the weights of the shared projector are updated.

[0012] As a further solution of the present invention, the step of 3D fine-tuning the 3D scene understanding benchmark and the step of end-to-end fine-tuning the explainable end-to-end driving benchmark include: The weights of the video encoder and image encoder are kept frozen, and the weights of the shared projector and LLM backbone are updated by minimizing the cross entropy loss.

[0013] Compared with the prior art, the beneficial effects of the present invention are: by jointly encoding dynamic 3D multi-view scene videos and static BEV map images through a modal encoder, comprehensive three-dimensional dynamic / static situational perception and unified visual representation of local scenes and global maps are achieved. Compared with other end-to-end models, the 3D LVLM architecture proposed in the present invention has excellent three-dimensional spatial understanding and dynamic time reasoning capabilities, effectively improving the accuracy and safety of end-to-end autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0015] Figure 1 A flowchart of a method for determining text interpretation for autonomous driving based on a large visual language model is provided in an embodiment of the present invention.

[0016] Figure 2 A flowchart of the overall large-scale visual language model framework provided by an embodiment of the present invention.

[0017] Figure 3 A model architecture diagram of the Drive3D-LVLM provided in an embodiment of the present invention.

[0018] Figure 4 A flowchart for constructing Driver3D-nuScenes provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] See also Figures 1 to 3 In an embodiment of the present invention, a method for determining an explanation text for autonomous driving based on a large visual language model is characterized in that the method comprises: Step S100, acquiring multi-view camera images, BEV map images and text query templates; Step S200, extracting video features from the multi-view camera image based on a video encoder, and extracting image features from the BEV map image based on an image encoder; Step S300, aligning image features with video features based on a modal encoder; Step S400, encoding the text query template to generate a text tag, mapping the aligned image features and video features to a text embedding space based on a shared projector, and generating a visual tag; Step S500: input the visual labels and text labels into the LLM backbone model, and output the autonomous driving explanation text.

[0021] In this embodiment, Drive3D-LVLM is proposed, which uses multi-view scene videos to provide dynamic local scene information; BEV map images to provide static global map information; and multi-type text queries as input, aiming to handle the highly challenging 3D scene understanding and explainable end-to-end driving tasks in autonomous driving.

[0022] like Figure 3 As shown, Drive3D-LVLM includes a video encoder and image encoder , used to extract visual features from raw video and image signals, in addition, it also includes the LLM backbone (such as Vicuna-7b), shared projector , Text Encoder and text decoder The text response generation process of Drive3D-LVLM is similar to LLaVA. Given a text query , video signal and image signal , Drive3D-LVLM first uses a text encoder Text query Convert to a series of text tokens : ; Then, use the video encoder Extract video signal Video Features , using the image encoder Extract image signal Image features : ; It is worth noting that we chose the modality encoder from LanguageBind to spontaneously align video and image features into a unified visual representation, thus solving the problem of decreased multimodal interactive learning effectiveness that may be caused by insufficient alignment before projection.

[0023] The present invention uses a two-layer perceptron (MLP) as a shared projector , used to transform video features and image features Mapping to text embedding space: ; in, Indicates a unified visual mark, with text mark have the same embedding space dimensions.

[0024] Finally, the unified visual mark With text mark Combine and input to the LLM backbone To generate the corresponding response, and use the text decoder Decode it back into text form: ; in, Represents a splicing operation, Represents the response output in text form.

[0025] The working principle of Drive3D-LVLM is as follows: First, Drive3D-LVLM uses a text encoder to convert text queries into a series of text tags. Secondly, Drive3D-LVLM selects the modality encoder from LanguageBind to extract video features of multi-view scene videos and image features of BEV map images, and spontaneously aligns the video and image features into a unified visual representation, thereby achieving joint encoding of dynamic multi-view scene videos and static BEV map images. Then, the video features and image features are mapped to the text embedding space through a shared projector, so that they have the same embedding space dimension as the text tags. Finally, the unified visual tags are combined with the text tags and input into the LLM backbone to generate the corresponding response, and decoded back to text form using the text decoder. This spontaneous alignment of video and image allows joint encoding of dynamic multi-view scene videos and static BEV map images, thereby achieving the extension of two-dimensional understanding to comprehensive three-dimensional dynamic / static situational awareness and the unification of visual representations of local scenes and global maps.

[0026] The whole process enables LVLM to jointly encode dynamic multi-view scene videos with static BEV map images with minimal efforts and changes to obtain excellent 3D spatial understanding and dynamic temporal reasoning capabilities.

[0027] See also Figure 3 As a preferred embodiment of the present invention, the text query template includes a historical control signal template and a task instruction template.

[0028] As a preferred embodiment of the present invention, the shared projector is a two-layer sensor.

[0029] In this embodiment, a two-layer perceptron (MLP) is used as a shared projector , used to transform video features and image features Mapping to text embedding space.

[0030] See also Figure 4 As a preferred embodiment of the present invention, it also includes: constructing a video image question-answering benchmark based on multi-view camera images and BEV map images, wherein the video image question-answering benchmark includes a 3D scene understanding benchmark and an explainable end-to-end driving benchmark.

[0031] In this embodiment, to support model training and testing, we further proposed Drive3D-nuScenes, the first new 3D video-image question-answering benchmark in the field of autonomous driving, focusing on 3D scene understanding and end-to-end driving tasks in outdoor, multi-modal and multi-view contexts, challenging the true 3D spatial understanding and dynamic temporal reasoning capabilities of LVLM.

[0032] There are currently various visual question answering (VQA) benchmarks, including the image question answering (ImageQA) benchmark and the video question answering (VideoQA) benchmark, but models trained on existing VQA benchmarks still have limitations in coping with the complexity of autonomous driving scenarios, mainly due to the difference in visual data between real driving scenarios and VQA benchmarks. This requires the model to effectively utilize the complementary information of videos and images to understand complex scenes and capture the 3D properties of objects from dynamic data streams.

[0033] To address this limitation, we built Drive3D-nuScenes, the first novel 3D Video-Image QA benchmark in the field of autonomous driving, focusing on 3D scene understanding and explainable end-to-end driving tasks in outdoor, multi-modal, and multi-view contexts. Different from other existing VQA benchmarks, Drive3D-nuScenes provides richer visual information, i.e., multi-view driving videos and BEV map images, and multi-type text queries, i.e., natural language, control signals, and meta-actions, bringing new challenges to the VQA and autonomous driving communities.

[0034] In order to train and test Drive3D-LVLM, the present invention contributes Driver3D-nuScenes, a new 3D Video-ImageQA benchmark built on the general dataset nuScenes, which contains large-scale, high-quality video-image-question-answer pair data, involving scene understanding and end-to-end driving in the field of autonomous driving, challenging the true 3D spatial understanding and dynamic temporal reasoning capabilities of LVLM. Drive3D-nuScenes contains two task benchmarks: 3D scene understanding benchmark; explainable end-to-end driving benchmark.

[0035] See also Figure 4 As a preferred embodiment of the present invention, the steps of constructing the 3D scene understanding benchmark specifically include: Determine the 3D scene layering system based on multi-view camera images and BEV map images; Determine question options and answer templates based on the 3D scene hierarchical system; Generate scene understanding question and answer pairs based on GPT-4V; In this embodiment, 1. 3D scene understanding benchmark The 3D scene understanding benchmark uses GPT-4V's fully automatic programmatic QA generation pipeline. The pipeline requires two types of structured data: multi-view scene videos and BEV map images; and question options and answer templates. By combining these structured data, GPT-4V can automatically generate multiple rounds of question-answer pairs. This paper will introduce in detail the construction process of the 3D scene understanding benchmark, including scene layering system development, question option and answer template design, multiple rounds of question-answer pair generation and manual post-processing.

[0036] The present invention proposes a scene stratification system, which divides the driving scene into four layers according to different scene elements: weather conditions , road conditions , transportation facilities and traffic participants The present invention also constructs a scene graph to abstractly represent the driving scene, wherein the first layer of nodes represents scene elements and the second layer of nodes represents various attributes of the scene elements.

[0037] Weather conditions are basic elements in driving scenarios and determine driving difficulty. The present invention defines weather conditions from three dimensions: type, lighting, and visibility. Types include sunny, cloudy, rainy, foggy, and snowy, which help reflect the impact of different weather types on driving styles. Lighting is divided into daytime and nighttime, which is directly related to the choice of driving strategy. Visibility is divided into high, medium, and low levels to quantify potential driving risks.

[0038] Road conditions are an important factor in driving scenarios and affect driving styles. We define road conditions from four aspects: type, structure, direction, and function. Road types include urban, suburban, rural, and highways, presenting unique driving challenges. Road structures include single lanes, dual lanes, multi-lanes, intersections, and unstructured lanes, with specific traffic flow patterns. Road directions are divided into one-way and two-way, which are related to lane selection and safe lane changes. Road functions include straight ahead, left turn, right turn, and omnidirectional, which guide the vehicle's expected driving strategy.

[0039] Traffic facilities are key elements in driving scenarios, and two common types of traffic facilities are traffic signs and road markings. By analyzing their existence and specific types, guidance can be provided for driving behavior, such as stop signs and sidewalk markings.

[0040] Traffic participants are a wide range of elements in the driving scene, and they directly affect the driving state. Different from traditional methods, the present invention only focuses on the key traffic participants that are most likely to affect the current scene. Each key traffic participant contains two types of attributes: 3D static attributes and 2D motion attributes.

[0041] 3D static attributes characterize the inherent characteristics of traffic participants, including type, size (length, width, height), state, position and orientation. Type, size, state and position are directly obtained from the original nuScenes annotation. For orientation, the original annotation does not provide the relative spatial relationship between the traffic participant and the ego vehicle, so this method develops a rule to determine the orientation. First, six orientations are defined, namely front, left front, right front, rear, left rear and right rear. Then, the angle between the traffic participant and the ego vehicle's forward direction is calculated. Based on the angle range, the orientation is determined as follows: ; Among them, the forward direction of the vehicle is defined as , and clockwise angles are positive.

[0042] The 2D motion attribute characterizes the dynamic characteristics of traffic participants within a period of time. The present invention designs fixed sentences to describe them, for example, "the car moves quickly from (0.15, 2.17) to (0.21, 10.36)." or "the adult is stationary at (2.38, 5.42).".

[0043] For question options and answer template design, the present invention manually creates four groups of question options around each layer of scene elements: , , and In order to prevent Drive3D-LVLM from overfitting fixed question patterns, each set of question options should contain multiple synonymous expressions of a question. The present invention also manually designed four sets of answer templates: , , and , which are matched with the question options respectively.

[0044] For the generation and post-processing of multi-round question-answer pairs, this embodiment uses GPT-4V's fully automatic programmatic QA generation pipeline: First, the general viewing direction is annotated for each frame of the multi-view scene video (for example, the upper left is the left front view) to prevent GPT-4V from ignoring details in the video. The multi-view scene video is used as visual input, the question options and answer templates are used as contextual information, and the BEV map image is further used as additional information to prompt GPT-4V to automatically generate multi-round question-answer pairs. Finally, it is manually post-processed to delete inappropriate questions and answers and modify incorrect expressions.

[0045] See also Figure 4 As a preferred embodiment of the present invention, the steps of constructing the explainable end-to-end driving benchmark include: Acquire control signal sequences based on multi-view camera images and BEV map images; determining a threshold vector, and determining an ego-vehicle action based on the threshold vector; ChatGPT is used to generate explanation text based on the vehicle's meta-actions, control signal sequences, and scene understanding question-answer pairs.

[0046] In this embodiment, the explainable end-to-end driving benchmark records the future driving strategy of the ego vehicle, which includes two parts: control signal prediction and meta-action and explanation text generation.

[0047] The ego vehicle control signal includes speed , Corner ,Location , acceleration and the turning rate The original nuScenes annotation provides a corresponding control signal sequence for each scene video. The present invention extracts historical control signals as context information to assist model prediction and extracts the current or next frame control signal as the true label. The present invention generates two rounds of questions and answers, involving the prediction of the acceleration and turning angle of the vehicle in the current frame, and the prediction of the speed, turning angle and position of the future frame.

[0048] Self-vehicle action Including direction judgment , longitudinal velocity estimation and lateral velocity estimation Three dimensions. First, define a threshold vector ,in and represents the longitudinal acceleration threshold, and represents the lateral acceleration threshold, represents the speed threshold, represents the lateral displacement threshold, Represents the yaw angle gain threshold. Based on the threshold vector , respectively, to explain the determination rules. Specifically, for the lateral velocity estimation , according to the lateral acceleration of the vehicle Determine its type: ; Similarly, for the longitudinal velocity estimate , according to the longitudinal acceleration of the vehicle Determine its type: ; For direction judgment First, according to the vehicle speed Make a judgment. hour, To stop; when When the lateral displacement of the vehicle Make a judgment. hour, For straight line; When the combined yaw angle gain Determine the type of turn and lane change. When, according to Determines the steering type. If , is a left turn; if , is a right turn. Then according to Determine the lane change type. If , is offset to the left; if , Shift to the right.

[0049] Finally, the self-car action Direction judgment , longitudinal velocity estimation and lateral velocity estimation Combination of: ; A total of 64 different meta-actions are generated. These structured meta-actions can relatively comprehensively simulate the behavior patterns of vehicles in different scenarios.

[0050] The explanation text provides a reason analysis of the future short-term driving strategy of the ego vehicle, expressed as a fixed sentence structure. The present invention inputs the complete scene understanding question and answer pair, predicted control signal and meta-action information as prompts to ChatGPT, requiring it to automatically generate the corresponding explanation text. This approach enables the explanation text to more accurately reflect various potential factors such as the scene and the ego vehicle.

[0051] See also Figure 1 , as a preferred embodiment of the present invention, further comprising: Align image features and video features in the 2D spatial domain with the embedding space of a large language model; 3D fine-tuning of 3D scene understanding benchmarks; End-to-end fine-tuning on the explainable end-to-end driving benchmark.

[0052] In this embodiment, 2D pre-training is included, which aims to achieve alignment between video / image features in the 2D spatial domain and the embedding space of the LLM backbone. Drive3D-LVLM is pre-trained on 703K video-text pairs from WedVid and 558K image-text pairs from CC3M. The pre-trained images and videos cover a variety of topics and are not specific to the field of autonomous driving. At this stage, the video encoder , Image Encoder and LLM backbone The weights of remain frozen and only the shared projector is updated The weight of .

[0053] Also included: 3D fine-tuning, 3D fine-tuning aims to train the model to perform 3D scene understanding tasks to enhance the model's 3D spatial understanding capabilities while retaining 2D semantic understanding as much as possible. Drive3D-LVLM performs 3D fine-tuning on 34K video-image-question-answer pairs built around scene understanding. Considering that the response output of Drive3D-LVLM is text type, cross entropy loss is used To supervise the reasoning process of the model: ; in, represents the trainable parameters, Represents a text response At this stage, the video encoder and image encoder The weights of are kept frozen and the shared projector is updated by minimizing the cross entropy loss and LLM backbone The weight of .

[0054] It also includes: end-to-end fine-tuning, the goal of end-to-end fine-tuning is to train the model to perform interpretable end-to-end driving tasks based on 3D scene understanding capabilities. Drive3D-LVLM performs end-to-end fine-tuning on 34K video-image-question-answer pairs built around end-to-end driving. At this stage, the same strategy is used to update the model weights.

[0055] A new three-stage training strategy is adopted to update the weights of Drive3D-LVLM to achieve the multi-task challenge of autonomous driving. 2D pre-training aims to achieve the alignment between video / image features in the two-dimensional spatial domain and the embedding space of the LLM backbone. 3D fine-tuning aims to train the model to perform 3D scene understanding tasks to enhance the model's three-dimensional spatial understanding ability while retaining two-dimensional semantic understanding as much as possible. The goal of end-to-end fine-tuning is to train the model to perform interpretable end-to-end driving tasks based on the three-dimensional spatial understanding ability. During the fine-tuning process, the cross-entropy loss is introduced to supervise the output of the model to ensure the consistency of the model's perception and reasoning results with the true labels. Compared with traditional end-to-end autonomous driving methods, the present invention performs well in both 3D scene understanding and interpretable end-to-end driving tasks, and has excellent three-dimensional spatial understanding and dynamic temporal reasoning capabilities.

[0056] This patent conducts extensive experiments on the Drive3D-nuScenes benchmark to evaluate the performance of Drive3D-LVLM, focusing on two tasks: 3D scene understanding; explainable end-to-end driving.

[0057] Image and video processing: We resize each image to 224×224, uniformly sample 8 frames from each video, and perform image preprocessing on each frame. The data in each batch is a combination of images and videos.

[0058] LVLM training: In the 2D pre-training stage, the model was trained for 1 epoch with a total batch size of 128, distributed on 8 A100 (80GB) GPUs. In the 3D fine-tuning and task fine-tuning stages, the AdamW optimizer with cosine learning rate scheduling was used, the initial learning rate was set to 2e-5, the warm-up ratio was 0.03, and the gradient accumulation step size was 2. The 3D fine-tuning model was trained for 1 epoch, and the task fine-tuning model was trained for 3 epochs. The whole process was completed on 8 L20 (48GB) GPUs, and the batch size of each GPU was 4.

[0059] For 3D scene understanding and meta-action and explanation text generation, this paper uses four common indicators in the NLP field for evaluation, including BLEU4, ROUGE, METEOR, and CIDEr. These indicators are designed to evaluate the quality of text generation, among which BLEU4 focuses on the accuracy of n-grams, ROUGE and METEOR consider the subtle differences in semantics and syntax, and CIDEr emphasizes the consensus and relevance of image descriptions. For control signal prediction, this paper pays special attention to the speed (m / s) and angle ( ), using the root mean square error (RMSE) and the "tolerance accuracy" ( ) for evaluation. Quantifies the prediction error below the tolerance threshold The proportion of test samples.

[0060] Table 1 shows the performance of Drive3D-LVLM in the 3D scene understanding task. Drive3D-LVLM achieves excellent results, with scores of 54.9, 61.5, 78.4, and 577.9 on the four indicators of BLEU4, ROUGE, METEOR, and CIDEr, respectively. In summary, thanks to the joint encoding of local scenes and global maps and the high-quality 3D scene understanding benchmark, Drive3D-LVLM is able to have excellent spatial understanding and temporal reasoning capabilities, achieving excellent performance in 3D scene understanding.

[0061] Table 1 Quantitative results of 3D scene understanding

[0062] This experiment is compared with other baseline methods, including RNN S2VT based on video language sequence to sequence and CNN WAA based on visual attention. In addition, this method is also compared with the latest interpretable autonomous driving methods, including ADAPT based on video transformer, DriveGPT4 based on instruction fine-tuning LVLM, and RAG-Driver, the latter of which has the ability to predict control signals.

[0063] The present invention evaluates the quality and accuracy of Drive3D-LVLM in the generation of meta-actions and explanation texts. The comparison results are shown in Table 2. Drive3D-LVLM is much higher than other methods in the four indicators of BLEU4, ROUGE, METEOR and CIDEr, fully demonstrating its effectiveness and superiority in the interpretability of autonomous driving. This method also evaluates the accuracy of Drive3D-LVLM in control signal prediction, including speed and angle. As shown in Table 3, the RMSE of Drive3D-LVLM in speed and angle prediction is 0.08 and 0.29, respectively. Drive3D-LVLM also surpasses other methods in tolerance accuracy within various ranges. In summary, the joint encoding of local scenes and global maps enhances the reasoning performance of the model, enabling Drive3D-LVLM to perform well in explainable end-to-end driving.

[0064] Table 2 Quantitative results of meta-action and explanation text generation

[0065] Table 3 Quantitative results of control signal prediction (speed and angle)

[0066] In summary, the experimental results show that the Drive3D-LVLM proposed in this paper has excellent 3D spatial understanding and dynamic temporal reasoning capabilities, and performs well in both scene understanding and end-to-end driving.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining interpretation text for autonomous driving based on a large visual language model, characterized in that: The method comprises: Obtain multi-view camera images, BEV map images, and text query templates; Extract video features from multi-view camera images based on a video encoder, and extract image features from BEV map images based on an image encoder; Align image features with video features based on a modality encoder; Encode the text query template to generate text tags, and map the aligned image features and video features to the text embedding space based on the shared projector to generate visual tags; The visual tags and text tags are input into the LLM backbone model, and the autonomous driving explanation text is output.

2. The method for determining the interpretation text of autonomous driving based on a large visual language model according to claim 1, characterized in that: The text query template includes a historical control signal template and a task instruction template.

3. The method for determining the interpretation text of autonomous driving based on a large visual language model according to claim 1, characterized in that: The shared projector is a two-layer perceptron.

4. The method for determining the interpretation text of autonomous driving based on a large visual language model according to claim 1, characterized in that: Also includes: A video image question-answering benchmark is constructed based on multi-view camera images and BEV map images. The video image question-answering benchmark includes a 3D scene understanding benchmark and an explainable end-to-end driving benchmark.

5. The method for determining the interpretation text of autonomous driving based on a large visual language model according to claim 4, characterized in that: The steps for constructing the 3D scene understanding benchmark specifically include: Determine the 3D scene layering system based on multi-view camera images and BEV map images; Determine question options and answer templates based on the 3D scene hierarchical system; Generate scene understanding question and answer pairs based on GPT-4V.

6. The method for determining the interpretation text of autonomous driving based on a large visual language model according to claim 5, characterized in that: The steps for constructing the explainable end-to-end driving benchmark include: Acquire control signal sequences based on multi-view camera images and BEV map images; determining a threshold vector, and determining an ego-vehicle action based on the threshold vector; ChatGPT is used to generate explanation text based on the vehicle's meta-actions, control signal sequences, and scene understanding question-answer pairs.

7. The method for determining the interpretation text of autonomous driving based on a large visual language model according to claim 6, characterized in that: Also includes: Align image features and video features in the 2D spatial domain with the embedding space of a large language model; 3D fine-tuning of 3D scene understanding benchmarks; End-to-end fine-tuning on the explainable end-to-end driving benchmark.

8. The method for determining the interpretation text of autonomous driving based on a large visual language model according to claim 7, characterized in that: The step of aligning the image features and video features in the 2D spatial domain with the embedding space of the large language model includes: The weights of the video encoder, image encoder, and LLM backbone are kept frozen, and only the weights of the shared projector are updated.

9. The method for determining the interpretation text of autonomous driving based on a large visual language model according to claim 7, characterized in that: The step of 3D fine-tuning the 3D scene understanding benchmark and the step of end-to-end fine-tuning the explainable end-to-end driving benchmark include: The weights of the video encoder and image encoder are kept frozen, and the weights of the shared projector and LLM backbone are updated by minimizing the cross entropy loss.

Images