Driving scene understanding method based on multi-modal large language model

Abstract

The invention belongs to a driving scene understanding method based on a multi-modal large language model, belongs to the technical field of automatic driving, and aims to improve the understanding ability of an automatic driving system in a complex and variable driving environment by introducing an LoRA fine tuning technology and performing efficient instruction fine tuning on an InternVL2-26B model. And particularly, the adaptability and the accuracy are improved when facing different driving conditions and diversified scene elements. In the wide evaluation of the driving scene, the method shows the significant potential of the method in the aspect of improving the model performance, facilitates the improvement of the scene understanding ability of an automatic driving system in a complex environment, and is of great significance in promoting the development of the automatic driving technology. According to the method, the image understanding capability in the automatic driving scene is improved by finely adjusting the multi-modal large language model, so that the natural language result of the driving scene understanding task can be output, and the limitation of the prior art on the long-tail problem of understanding the driving scene in the real world is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a driving scene understanding method based on a multimodal large language model. Background Art

[0002] The multimodal large language model is an advanced artificial intelligence model that combines deep learning, natural language processing, computer vision and speech recognition technologies. It is designed to comprehensively process and understand data in multiple modalities, including text, images, sound and video. The core feature of the model is to use deep neural networks, such as Transformer, to process sequence data and learn complex features and patterns from large amounts of multimodal data. The multimodal large language model extracts deep semantic information from text data through advanced natural language processing technology and can generate fluent and relevant text content. A major innovation of the model is its cross-modal fusion capability, which effectively integrates data information from different modalities to achieve complementarity and enhancement between data, such as being able to understand image content and generate descriptive text, or generate corresponding images based on text content. The multimodal large language model is pre-trained on large-scale multimodal datasets, and has mastered a wide range of knowledge and information, which can then be fine-tuned to adapt to specific application scenarios and needs. In summary, the multimodal large language model represents a major advancement in the field of natural language processing and artificial intelligence in recent years. Its unique ability to understand and generate multimodal data not only improves the analysis depth within each modality, but also realizes the organic combination and complementarity of information between modalities, opening up new paths for research and application in related fields.

[0003] OpenAI launched the CLIP model in 2021. The core idea is to use contrastive learning to encode images and text in a common embedding space. In this way, the CLIP model can bring semantically related images and texts closer together, and move irrelevant images and texts away. Although the CLIP model performs well in tasks such as image classification, it cannot be directly applied to text generation tasks. To solve this problem, Salesforce launched the BLIP model for unified visual language understanding and generation in 2022. As the scale of visual and language pre-training models continues to increase, the computational and economic costs of end-to-end training are also increasing. To solve this problem, Salesforce launched the BLIP-2 model in 2023.

[0004] The rapid development of multimodal large language models also provides new application possibilities for other technical fields. With the continuous development of autonomous driving technology, the automotive industry is rapidly moving towards automation and intelligence. Autonomous driving vehicles need to be able to accurately understand driving scenes, including identifying traffic signs on the road, detecting other vehicles and pedestrians, and predicting traffic conditions, so that vehicles can make correct decisions and behaviors. At present, there are many driving scene understanding algorithms based on traditional computer vision technology. These algorithms may achieve good results in certain specific scenarios, but their scene understanding capabilities are still not ideal in the face of complex and changing driving environments. Therefore, how to improve the scene understanding ability of autonomous driving systems to improve the safety of autonomous driving is still one of the current hot research issues, and multimodal large language models provide a new way to solve this problem. Compared with traditional machine learning algorithms, multimodal large language models have stronger generalization capabilities and can better handle rare situations that rarely or never appear in training data. Therefore, it has a high research value in solving the long-tail problem of autonomous driving scene understanding. Based on this purpose, the present invention proposes a driving scene understanding method based on a multimodal large language model. The method realizes driving scene understanding tasks, such as risk object perception, driving scene classification, and vehicle action recommendations, by fine-tuning the multimodal large language model. Summary of the invention

[0005] (I) Purpose of the invention

[0006] The purpose of the present invention is to provide a driving scene understanding method based on a multimodal large language model, which aims to improve the image understanding ability in autonomous driving scenarios by fine-tuning the multimodal large language model, so as to output natural language results of driving scene understanding tasks and solve the limitations of existing technologies in the long-tail problem of real-world driving scene understanding.

[0007] (II) Technical solution

[0008] In order to achieve the above objectives and solve the above technical problems, the technical solutions of the present invention are as follows:

[0009] A driving scene understanding method based on a multimodal large language model specifically comprises the following steps:

[0010] Step 1: Design of a set of driving scene understanding questions, including: designing a series of specific natural language questions to guide the multimodal large language model to make accurate analysis and decisions in complex driving scenes. These questions can be divided into three different levels: perception, prediction, and planning, aiming to comprehensively evaluate important factors in driving scenes. Covering prompt word design, model return content and its format design, etc., combined with prompt word engineering technology, design system prompt words used in the instruction fine-tuning process, and guide the multimodal large language model to output accurate and high-quality results through carefully designed prompts.

[0011] Step 2: Construction of a multimodal driving scene dataset, including: collecting image data in autonomous driving scenes. The collected data needs to cover as many driving conditions and environments as possible, such as sunny days, rainy days, nights, etc., and should also include a variety of driving scene elements, such as vehicles, pedestrians, road signs, traffic accidents, etc., to ensure the comprehensiveness and accuracy of model training and improve the generalization ability of the model. Afterwards, the collected driving scene image data is annotated with natural language text to ensure that the model can learn the association information between the image content and the corresponding text.

[0012] Step 3: Fine-tuning of multimodal large language model instructions, including: fine-tuning the multimodal large language model using the data processed in steps 1 and 2. First, the autonomous driving scene image is input into the model, and the model extracts visual features through its built-in visual processing component. Next, the text description corresponding to the image is input into the language processing component of the model, and the fine-tuning parameter settings such as learning rate, batch size, optimizer selection, etc. are adjusted. Through joint training, the model learns how to align visual information with language descriptions, improves the multimodal large language model's ability to understand driving scenes, and thus achieves driving scene understanding tasks such as risk object perception, driving scene classification, and vehicle action recommendations.

[0013] (III) Effective income

[0014] 1. The present invention enhances the generalization ability and environmental adaptability of the multimodal large language model. By collecting and processing multimodal data containing various driving conditions and environments, the present invention significantly improves the generalization ability of the multimodal large language model in the autonomous driving scenario. This diverse data enables the model to effectively cope with different driving conditions such as sunny and rainy days, as well as various driving scene elements such as vehicles, pedestrians, and road signs, thereby improving the accuracy and reliability of the multimodal large language model in complex actual driving environments.

[0015] 2. This invention improves the quality and stability of the output results of the multimodal large language model. Through carefully designed fine-tuning content, such as fine-tuning the prompt word design and the model return content design, the model is effectively guided to generate more accurate and relevant outputs. It achieves precise control of the model output content and improves the quality and stability of the model output results.

[0016] 3. The present invention improves the comprehensive driving scene understanding ability of the multimodal large language model: After effective fine-tuning, the multimodal large language model can better understand complex driving scenes and achieve driving scene understanding tasks such as risk object perception, driving scene classification, and vehicle action suggestions. The model can not only identify key elements in the driving scene, but also accurately understand the semantic relationship and potential risks of these elements in a specific driving environment, thereby providing effective decision support for drivers and autonomous driving systems, and improving the decision-making efficiency and safety of autonomous driving systems in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The overall framework diagram of the driving scene understanding method of the present invention;

[0018] Figure 2 A schematic diagram of the logical relationship between the problem sets designed by the present invention;

[0019] Figure 3 The driving scene data set construction process of the present invention;

[0020] Figure 4 Examples of driving scene image data and corresponding natural language annotations;

[0021] Figure 5 Schematic diagram of the experimental scene of the present invention;

[0022] Figure 6 Schematic diagram of comparative test results. Specific implementation methods

[0023] A key task for autonomous vehicles is to accurately understand driving scenarios, which involves identifying road conditions, monitoring other vehicles and pedestrians, and predicting traffic flow. Although many traditional computer vision-based algorithms currently perform well in specific environments, their scene understanding capabilities are often limited when faced with more complex and changing driving environments. Therefore, in current autonomous driving safety research, how to effectively improve the scene understanding capabilities of these systems remains a key issue.

[0024] The present invention proposes a driving scene understanding method based on a multimodal large language model. With the powerful understanding and reasoning capabilities of the multimodal large language model and the efficient fine-tuning technology of LoRA, the driving scene understanding tasks such as risk object perception and driving scene classification are realized. The overall framework of the present invention is as follows: Figure 1 shown.

[0025] Step 1: Design of driving scenario understanding problem set

[0026] In order to guide the multimodal large language model to perform accurate analysis and decision-making in complex driving scenarios, the present invention first designs a series of specific natural language questions, which can be divided into three different levels: perception, prediction, and planning, aiming to comprehensively evaluate the important factors in driving scenarios.

[0027] Specifically, these questions include description of the current driving environment, judgment of weather and road conditions, identification of the most important objects in the scene, detection and interpretation of traffic signs, judgment of the driving state of the vehicle, identification of the state of other vehicles, analysis of potential risks, and driving recommendations based on the current scenario. Figure 2 The figure shows a schematic diagram of the logical relationship between questions. Through these questions, the present invention aims to make full use of the thinking chain mechanism of the large language model, and gradually guide the model to think and answer in a reasonable logical order, so as to ensure that the model can have a comprehensive understanding and accurate analysis when facing complex driving scenarios.

[0028] During the design process, the present invention pays attention to the clarity and consistency of the questions. For example, when describing the current driving environment, the question is clearly and accurately stated to avoid misleading the model. When judging weather and road conditions, the present invention adopts unified terminology and standards to ensure the consistency of annotations between different data. In particular, for questions about the vehicle's motion status and action suggestions, the present invention limits the model to output only limited answers, such as moving, turning left, turning right or stationary, etc., to ensure that the model answers are concise and clear, avoiding misunderstandings that may be caused by complex descriptions. Based on the above considerations, the present invention proposes a set of questions to guide the model to fully understand the driving scene. These questions include:

[0029] 1.Describe the current driving scenario.

[0030] 2.What is the current weather condition?

[0031] 3.What is the current road condition?

[0032] 4.What is the single most important object in the scene?

[0033] 5.Are there any traffic signs in the image?If so,identify and explainthem

[0034] and their impact on the ego vehicle.

[0035] 6.What is the current driving status of the ego vehicle?(Answerwithone word only:moving,turning_left,turning_right,stationary.Provide noother explanation.)

[0036] 7.Are there any other vehicles in the image?If so,identify anddescribe

[0037] their state and direction.

[0038] 8.Are there any potential risks in the current driving scenario?Ifso,

[0039] identify and describe these risks.

[0040] 9.Based on the current driving scene,provide an action recommendation

[0041] for the vehicle.(Answer with one word only:accelerate,decelerate,stop,change_lane_left,change_lane_right,continue_straight.Provide no otherexplanation.)

[0042] Step 2: Construction of multimodal driving scenario dataset

[0043] The present invention is based on a total of 34,149 images taken by the CAM_FRONT sensor in the nuScenes open source dataset, which covers a variety of driving scenarios, including urban streets, highways, and rural roads. The closed-source multimodal large model GPT-4O's API interface is used in combination with the set of questions introduced in step one to automatically annotate the image data. In order to ensure the accuracy and consistency of the annotations, a manual review process is introduced, and a sampling inspection method is used to extract 15% of the annotated data for detailed review. For each sample extracted, carefully check the matching degree of the natural language annotation data generated by the API with the actual image content to evaluate the accuracy and consistency of the annotations. If an incorrect or inaccurate annotation is found, the sample content is manually corrected. Figure 3 The present invention proposes a process for constructing a multimodal driving scene dataset. Based on the above process, the present invention constructs a high-quality, multi-level multimodal driving scene dataset. Figure 4 The figure shows an annotation example. This dataset not only covers a variety of driving scenarios, but also meets high standards in annotation quality and consistency, effectively solving the shortcomings of existing open source natural language annotation datasets for driving scenarios.

[0044] Step 3: Fine-tune the multimodal large language model instructions

[0045] The present invention comprehensively considers the scene understanding ability and reasoning performance of the model, and finally selects the InternVL2-26B model as the basic model and uses the multimodal driving scene dataset proposed in step 2 to perform visual command fine-tuning based on the model.

[0046] Specifically, the format of the multimodal dataset was first converted according to the training format template used by InternVL2-26B in the pre-training stage, and then the InternVL2-26B model was trained on 8 Nvidia A40 GPUs using the LoRA (Low-Rank Adaptation) method. The single GPU training batch size was set to 8, the gradient accumulation was 2, the global batch size was 128, and 1 epoch was trained on the complete dataset. Finally, the multimodal driving scene understanding model proposed in the present invention was obtained.

[0047] Example 1

[0048] Experimental scenario

[0049] This paper uses 300 images from the nuscenes dataset to qualitatively and quantitatively analyze the driving scene understanding and analysis capabilities of the proposed model. The images used did not appear in the visual command fine-tuning training phase to ensure the objectivity and fairness of the experimental results. Figure 5 One of the test images is shown.

[0050] Comparative experimental results

[0051] The present invention uses the closed-source large model GPT-4 to evaluate the present invention and the other three mainstream open-source multimodal large language models on a complete test dataset, and obtains the results shown in Table 1 below.

[0052] Table 1 Comparative experimental results

[0053]

[0054] It can be seen from the above experimental results that compared with the mainstream open source models, the present invention has achieved the highest score on the driving scenario test dataset, proving the advanced nature of the present invention.

[0055] Ablation experiment

[0056] In order to further verify the effectiveness of the present invention, the present invention designed an ablation experiment to evaluate the effect of visual instruction fine-tuning on the performance of the model. Specifically, the present invention compared the performance of the present invention and other three open source models before and after visual instruction fine-tuning in the same driving scenario, and obtained the results shown in Table 2 below.

[0057] Table 2 Ablation experiment results

[0058]

[0059] It can be seen from the experimental results that the four models have significantly improved their driving scene understanding and analysis capabilities after being fine-tuned with the visual instructions proposed in the present invention, which proves the effectiveness of the invention.

[0060] This paper proposes a driving scene understanding method based on a multimodal large language model. This method introduces LoRA fine-tuning technology to efficiently fine-tune the InternVL2-26B model, aiming to improve the understanding ability of the autonomous driving system in complex and changeable driving environments, especially the adaptability and accuracy when facing different driving conditions and diverse scene elements. In a wide range of evaluations of driving scenarios, this paper demonstrates its significant potential in improving model performance, which helps to improve the scene understanding ability of the autonomous driving system in complex environments, and is of great significance to promoting the development of autonomous driving technology.

[0061] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A driving scene understanding method based on a multimodal large language model, characterized in that: The implementation process includes the following steps: Step 1: Design of driving scenario understanding problem set Guide the multimodal large language model to make accurate analysis and decisions in complex driving scenarios, design natural language questions at three different levels: perception, prediction, and planning, and comprehensively evaluate important factors in driving scenarios; Natural language questions include description of the current driving environment, judgment of weather and road conditions, identification of the most important objects in the scene, detection and interpretation of traffic signs, judgment of the driving state of the vehicle, identification of other vehicles, potential risk analysis, and driving suggestions based on the current scene; Through natural language questions, we make full use of the thinking chain mechanism of the large language model to guide the model to think and answer in a logical order, ensuring that the model can understand and accurately analyze complex driving scenarios; Use unified terminology and standards to ensure clarity and consistency of natural language questions, ensure consistency of annotations between different data, and limit the model to output only limited answers for questions about the vehicle's motion status and action suggestions to avoid misunderstandings; Natural language questions include: 1.Describe the current driving scenario. 2.What is the current weather condition? 3.What is the current road condition? 4.What is the single most important object in the scene? 5.Are there any traffic signs in the image? If so, identify and explain them and their impact on the ego vehicle. 6.What is the current driving status of the ego vehicle? Answer with one word only:moving,turning_left,turning_right,stationary.Provide no other explanation. 7.Are there any other vehicles in the image? If so,identify and describe their state and direction. 8.Are there any potential risks in the current driving scenario? If so, identify and describe these risks. 9.Based on the current driving scene, provide an action recommendation for the vehicle.Answer with one word only:accelerate,decelerate,stop,change_lane_left,change_lane_right,continue_straight.Provide no other explanation. Step 2: Construction of multimodal driving scenario dataset Based on a total of 34,149 images taken by the CAM_FRONT sensor in the nuScenes open source dataset, which covers a variety of driving scenarios, including urban streets, highways, and rural roads, the closed-source multimodal large model GPT-4O's API interface is used to automatically annotate the image data in combination with the natural language question set in step 1; The manual review process was introduced, and 15% of the annotated data was selected for detailed review using a sampling inspection method; For each sample extracted, check the matching degree between the natural language annotation data generated by the API and the actual image content, and evaluate the accuracy and consistency of the annotation; If any labeling errors or inaccuracies are found, the extracted sample content will be manually corrected; Step 3: Fine-tune the multimodal large language model instructions Select the InternVL2-26B model as the base model and use the multimodal driving scenario dataset proposed in step 2 to fine-tune the visual instructions based on this model; First, the format of the multimodal dataset was converted according to the training format template used by InternVL2-26B in the pre-training stage; The InternVL2-26B model was then trained on 8 Nvidia A40 GPUs using the LoRA method. The single GPU training batch size was set to 8, the gradient accumulation was set to 2, the global batch size was set to 128, and 1 epoch was trained on the complete dataset. Finally, a multimodal driving scene understanding model was obtained.

Images