Data construction and training method of multi-mode large language model for automatic driving

Abstract

The invention discloses a data construction and training method of a multi-modal large language model for automatic driving, and the method comprises the steps: S10: data selection: selecting positive sample scene training data and negative sample scene training data from original video data collected by a vehicle, the positive sample scene refers to a scene in which a driving decision needs to be changed in the vehicle driving process, and the negative sample scene refers to a scene in which the driving decision does not need to be changed in the vehicle driving process; s20, data cleaning: extracting the positive sample scene training data and the negative sample scene training data selected in the step S10 into continuous frames of image data according to a preset frequency, and synchronously cleaning the vehicle speed and the navigation instruction of each frame; and S30: data annotation: obtaining a true value through a manual annotation mode. According to the data construction method, the diversity and balance of the data can be ensured, and the generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a data construction and training method for a multimodal large language model for autonomous driving. Background Art

[0002] As an important direction in the future transportation field, autonomous driving (AD) technology has gone through years of development and has made significant progress. In the early days, the focus was on the feasibility of autonomous driving technology and the establishment of a basic framework. After 2010, with the improvement of computing power and the development of sensor technology, autonomous driving technology has made significant breakthroughs, and some companies have begun to conduct autonomous driving tests on actual roads. At the current stage, autonomous driving technology is gradually commercialized, and many companies have launched cars with a certain level of automation. New technologies and algorithms are constantly being introduced, such as deep learning and perception fusion.

[0003] The basic principle of autonomous driving technology is to perceive the situation of the vehicle and the surrounding environment in real time through sensors, then make planning decisions through intelligent systems, and finally execute driving operations through control systems. This includes three links: perception, decision-making, and control. Its application scenarios include trunk logistics, terminal logistics, ports, unmanned taxis (Robotaxi), etc. For example, at the port, autonomous driving trucks can transport containers at low speeds. Since the port environment is relatively closed and the routes are fixed, the implementation of autonomous driving technology is relatively easy.

[0004] The application of multimodal large language models (MLLMs) in the field of autonomous driving is a rapidly developing research direction. These models combine the complex reasoning capabilities of large language models (LLMs) with image, video and audio data, enabling them to perform a variety of tasks such as image classification, matching text to corresponding videos, and speech detection. The development of autonomous driving technology has gone through early exploration to the application of modern deep learning techniques. Early research was limited by sensor accuracy and computing power, but in the past two decades, with the introduction of deep neural networks (DNNs), autonomous driving systems have made rapid progress. The application of MLLMs in autonomous driving is mainly concentrated in the following aspects:

[0005] 1. Perception: MLLMs are able to understand non-text data such as images and point clouds through text analysis, which has made significant progress in image classification, segmentation, and object detection.

[0006] 2. Decision Making: MLLMs can understand traffic scenarios and improve the driving decision-making process. These models are trained on a large amount of traffic scenario data and can extract valuable information from multiple sources such as maps, videos, and traffic rules.

[0007] Although MLLMs have great potential in the field of autonomous driving, there are still some challenges and problems to be solved, including how to better apply these models to actual autonomous driving systems.

[0008] In general, the application prospects of MLLMs in the field of autonomous driving are promising, but further research and development are still needed to address the current challenges.

[0009] Patent CN202410243702 discloses a multimodal perception and decision-making method for autonomous driving based on a large language model. The method uses a target detection model, a segmentation model, and a fusion model to obtain visual-related feature maps. The multimodal large language model is then used to fuse the first prompt information representing the user's intention, the second prompt information representing the priority of the target detection category, and the fused feature map to obtain text information. Finally, the text information is input into the large language model to obtain a decision result, which improves the priority of traffic participants in the perception decision-making of autonomous driving vehicles and improves the safety of autonomous driving vehicles.

[0010] Patent CN202410386647 discloses a generative large language model training method for autonomous driving. The method uses supervised fine-tuning and human preference reinforcement learning to enable the generative large language model to have the same driving experience and common sense as human drivers. Based on the reasoning ability of the generative large language model, it can effectively deal with long-tail cases that appear in real environments, improve the safety and explainability of autonomous driving or assisted driving, and reduce training and deployment costs.

[0011] However, the existing patents related to multimodal large language models for autonomous driving only have model training methods, but lack relevant data construction methods. However, data is more important to large language models than training methods. Summary of the invention

[0012] In response to the above technical problems, the present invention proposes a multimodal large language data construction and training strategy in the field of autonomous driving, which can ensure the diversity and balance of data and improve the generalization ability of the model.

[0013] In order to achieve the above-mentioned purpose, the technical solution of the present invention provides a data construction method for a multimodal large language model for autonomous driving, which includes: S10: data selection, selecting positive sample scene training data and negative sample scene training data from the original video data collected by the vehicle, the positive sample scene refers to a scene where the driving decision needs to be changed during the vehicle's driving, and the negative sample scene refers to a scene where the driving decision does not need to be changed during the vehicle's driving; S20: data cleaning, extracting the positive sample scene training data and the negative sample scene training data selected in step S10 into continuous frame image data at a predetermined frequency, and synchronously cleaning out the vehicle speed and navigation instructions of each frame; S30: data labeling, obtaining the true value through manual labeling.

[0014] Furthermore, in step S10, when performing data selection, the selection of positive sample scene training data adopts the following two methods: manual selection, controlling part of the data collection vehicle through the automatic driving algorithm, and manually taking over the vehicle when the automatic driving control algorithm cannot make a reasonable decision, and synchronously generating the data slice at the current moment as the positive sample scene training data; model selection, using the image-text multimodal model to build a image-text retrieval platform to retrieve the full amount of original data, wherein before the retrieval, firstly obtain the corresponding image representation vector for all the original image data through the image-text multimodal model, and then obtain the text representation vector for the set keywords through the image-text multimodal model, and use the distance measurement method to calculate the distance between the image representation vector and the text representation vector, and then obtain the similarity of the two vectors, that is, obtain the semantic similarity between the corresponding image and text, and find the original data of the image corresponding to the keyword on the image-text retrieval platform according to the similarity threshold.

[0015] Furthermore, in step S30, manual annotation includes the following dimensions: (1) key frame selection: selecting positive sample key frames that require driving decision changes and negative sample key frames that do not require driving decision changes from the continuous frame data obtained in step S20; (2) key targets: using 2D boxes to mark the targets in the positive sample key frames that will affect the driving decision, and giving a description of the relevant content; (3) scene description: used to describe the scene information that can be observed in the current frame image; (4) environment description: used to explain the objective environment information of the current frame image; (5) driving decision meta-operation sequence: including various meta-operation instructions in the driving process, selecting appropriate instructions for orderly arrangement and combination to form a suitable meta-operation sequence; (6) driving decision description: performing a structured decision description based on the driving decision meta-operation sequence, requiring a comprehensive and complete restoration of the meta-operation sequence.

[0016] Furthermore, in step S30, when performing manual labeling, the scene description is in the form of a subjective description question, and the environment description is in the form of an objective multiple-choice question.

[0017] The technical solution of the present invention also provides a method for training a multimodal large language model for autonomous driving, which uses the data obtained by the data construction method as described above to perform model training, wherein the training method includes:

[0018] S11: Data preprocessing, obtaining a training data set by processing the labeled data set obtained in step S30 into a format acceptable to the model;

[0019] S21: Pre-training, using a public data set in the field of autonomous driving and an internal target detection data set to pre-train the model to obtain an autonomous driving basic model, wherein the internal target detection data set is obtained by performing target detection on the image data of the continuous frames obtained in step S20;

[0020] S31: Model fine-tuning: Continue training the autonomous driving basic model obtained in step S21 using the training data set obtained in step S11 to fine-tune the model.

[0021] Furthermore, in step S11, the labeled data set obtained in step S30 is processed into a format acceptable to the model by organizing objective multiple-choice questions and subjective description questions into question-answer pairs; and a plurality of different question-answer pairs are spliced ​​as a sample by random combination, or the same question is repeatedly asked in a sample to generate a training data set online.

[0022] Furthermore, in step S31, the labeled data set obtained through preprocessing in step S11 and the training data set generated online are used to fine-tune the model.

[0023] Furthermore, in step S21, the public data sets used include: a public Chinese and English image description data set, a public target counting data set, a public Chinese and English 2D target detection data set, and a public Chinese and English autonomous driving field knowledge question and answer data set.

[0024] Furthermore, in step S21, the internal target detection data set is obtained by the following two methods: manual labeling, sending the image data of the continuous frames obtained in step S20 to labeling, and the labelers mark the driving-related targets with 2D boxes; the real value large model, performing target detection on the image data of the continuous frames obtained in step S20 through the existing large detection model to obtain the predicted 2D box. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 It is a flow chart of the data construction method of the present invention;

[0027] Figure 2 It is a flow chart of the model training method of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] like Figure 1 The data construction method proposed in the present invention includes three steps:

[0030] 1) Data selection

[0031] All data described in the present invention are collected by vehicles equipped with relevant visual sensors. The amount of raw data collected by a large number of vehicles is bound to be huge. Therefore, how to select data containing important scenes is the first step to be taken. Specifically, the above-mentioned "important scenes" refer to scenes where driving decisions need to be changed during vehicle driving. We call such data "positive sample scenes". In order to avoid overfitting the model to positive samples, it is also necessary to select some scenes where driving decisions do not need to be changed. We call them "negative sample scenes". For the selection of positive samples, we use the following two methods:

[0032] a. Manual selection

[0033] Some of the data collection vehicles are controlled by autonomous driving algorithms. The autonomous driving algorithms mentioned here adopt the paradigm of perception + prediction + planning + control. Each vehicle is equipped with a safety officer. When the autonomous driving algorithm cannot make reasonable decisions in certain situations, the safety officer will take over the vehicle to prevent danger and synchronously generate data slices at the current moment. The slice length can be set by yourself. The default here is 1 minute, that is, 30 seconds before and after the takeover time. The data slice of 1 minute is the positive sample.

[0034] b. Model selection

[0035] The data screened out by the above manual selection method is of high quality, but its cost is also high, and the amount of data cannot reach the level that can train the model in a short time, so it is necessary to use another automated method to retrieve the full amount of original data. Here we use the text-image multimodal model CLIP (Contrastive Language-Image Pre-Training) to build a text-image retrieval platform, through which text can be used to retrieve semantically similar images (text search image), or use images to retrieve other similar images (image search image). The principle is to map text and images to the same semantic space, generate a vector of fixed length, and calculate the cosine similarity of the two vectors to obtain the semantic similarity between text and image. Before searching, we need to run the CLIP model on all data frames to obtain the corresponding image representation vectors and store them in the vector database. Since the total amount of data may be very large, we can choose to run 1 frame every 10 frames to greatly speed up the speed of vector generation. Finally, we can use some artificially set keywords, such as "animals ahead" and "obstacles ahead", to input them into the CLIP model to obtain text representation vectors, calculate the cosine similarity between the image representation vector and the text representation vector, and find the original data of the corresponding image on the image-text retrieval platform according to different similarity thresholds. In addition, this method can also very conveniently supplement the lack of data in a certain aspect and improve the problem of unbalanced data distribution to a certain extent. For example, if there is a lack of night data in the existing data, it can be retrieved from the platform through the keyword "night".

[0036] 2) Data cleaning

[0037] Considering that the output of driving decisions is a sequential process, that is, it is necessary to use the historical frame situation (such as the relative position change between the vehicle and other targets in the picture, the change of the vehicle speed, etc.) to more reasonably give the driving decision of the current frame, so the data structure of our data cleaning output is a continuous frame, and the vehicle speed and navigation instructions of each frame need to be cleaned out synchronously. The vehicle speed is used to determine whether the current situation requires acceleration or deceleration, and the navigation instructions are used to guide the model vehicle in the direction it should travel when there are multiple drivable routes, including but not limited to going straight, turning left, turning right, turning around, etc. Specifically, according to a predetermined frequency (for example, 10Hz), the positive sample video data and negative sample video data selected above are extracted into image data of continuous frames.

[0038] 3) Data labeling

[0039] After we have the data, we need the ground truth (GT) to train the model. We obtain the ground truth through manual annotation, which mainly includes the following dimensions:

[0040] a. Key frame selection: In the above “data selection” step, continuous frame scenes of positive and negative samples have been selected, but not every frame needs to be used for training, otherwise it will be too redundant. Therefore, the first step of data annotation is to select “key frames” from continuous frames, which is divided into two parts:

[0041] a) Positive sample frame: the key frame that needs to change the driving decision. When there are multiple frames that meet the requirement, try to choose the earliest frame;

[0042] b) Negative sample frames: key frames that do not require changing driving decisions, such as "the road ahead is clear, maintain the current lane and speed and continue driving."

[0043] b. Key objectives: Use 2D boxes to mark the targets in the positive sample frames that may affect the self-driving decision, including but not limited to pedestrians, motor vehicles, non-motor vehicles, obstacles, traffic lights, speed limit signs, etc., and give the following descriptions of the relevant content to help the model better understand:

[0044] a) Category: the name of the key objective;

[0045] b) Risk point: the potential impact of the target on the vehicle;

[0046] c) Attributes: static attributes (such as semantic information of signboards), location, motion status, special behaviors (such as traffic police’s gestures), etc.

[0047] c. Scene description: used to describe the scene information that can be observed in the current image, and should try to avoid uncertain information.

[0048] d. Environment description: objective multiple-choice questions, used to describe the objective environment information of the current frame image, to help the model better understand the semantic information of the image, including but not limited to the following aspects:

[0049] a) Image quality: normal, blurry, dirty, etc.;

[0050] b) Weather: sunny, rainy, snowy, foggy, etc.;

[0051] c) Time: day, night;

[0052] d) Road type: urban, rural, highway, etc.;

[0053] e) Lane lines: whether there are lane lines on the left / right side and whether they can be crossed;

[0054] f) Road conditions: flat, potholes, waterlogged, slippery, etc.;

[0055] g) Sharp turn: Is there a sharp turn ahead?

[0056] h) Fork in the road: Is there a fork in the road ahead, including but not limited to road gaps, crossroads, T-intersections, Y-intersections, etc.;

[0057] i) Vehicle merging: Are there any vehicles merging ahead?

[0058] e. Driving decision meta-operation sequence: It includes various meta-operation instructions in the driving process. Select appropriate instructions and arrange and combine them in order to form a suitable meta-operation sequence:

[0059] a) Speed ​​related: uniform / low / high speed, rapid / slow acceleration, rapid / slow deceleration, stop and wait, parking, etc.;

[0060] b) Turning related: going straight, turning left / right, U-turn, reversing, etc.;

[0061] c) Lane related: left / right lane change, left / right lane borrowing, left / right detour, etc.

[0062] f. Driving decision description: A structured (meta-operation + object + time) decision description is made based on the driving decision meta-operation sequence, which requires a comprehensive and complete restoration of the meta-operation sequence. For example: stop and wait (meta-operation) for pedestrians to pass in front (object) and then (time) accelerate and change lanes to merge into the right lane (meta-operation) (object).

[0063] like Figure 2 As shown, the model training method proposed in the present invention includes the following three steps:

[0064] 1) Data preprocessing

[0065] The annotation results are processed into a format acceptable to the model, that is, objective multiple-choice questions and subjective description questions are organized into question-answer pairs. At the same time, considering that different contextual semantic information will affect the prediction results of the model, a random combination of multiple different question-answer pairs is used to splice as a sample, or the same question is asked repeatedly in a sample, or other data enhancement methods are used to generate training data sets online, increase data diversity, and to a certain extent reduce the impact of different contextual information on model reasoning. Therefore, there will be samples of various lengths of single-round and multi-round questions and answers in the training set.

[0066] Considering that the training data set generated offline is fixed, that is, the same data is fed to the model in each epoch of training, and there is a lack of data diversity in terms of contextual semantics, the training data set is generated online. That is, each iteration in the training process will generate a unique training sample according to the above random combination method. As the number of training rounds increases, the samples input to the model become richer. This greatly expands the data set when the amount of original data is not enough, which can effectively improve the generalization ability of the model.

[0067] 2) Pre-training

[0068] Due to the high cost and low efficiency of labeling, it is impossible to accumulate a large amount of data in a short period of time to directly train the model. Therefore, a two-stage training strategy of pre-training + fine-tuning is adopted. We use a large number of public data sets in the field of autonomous driving and internal object detection data sets to pre-train the model, hoping to better align the model's image and text understanding capabilities in the field of autonomous driving and obtain a basic model for autonomous driving. The data sets used are roughly divided into the following categories:

[0069] a. Public Chinese and English image description dataset (to improve the model's macroscopic understanding of images)

[0070] b. Public target counting dataset (to improve the counting ability of the model)

[0071] c. Public Chinese and English 2D object detection datasets (to improve the model's object detection capabilities)

[0072] d. Public Chinese and English autonomous driving knowledge question-answering dataset (to improve the model’s understanding of driving)

[0073] e. Internal target detection data set (including pedestrians, motor vehicles, non-motor vehicles, common obstacles, traffic lights, road surface anomalies, road surface garbage and other categories, to improve the model's target detection capabilities in the target data domain)

[0074] In particular, the internal target detection dataset is obtained in two ways. One is manual labeling, that is, the original image (the continuous frame image obtained by the above data cleaning) is sent for labeling, and the labeler is asked to mark all driving-related targets in the image with 2D boxes, including but not limited to pedestrians, motor vehicles, non-motor vehicles, obstacles, traffic lights, etc.; the second is to use large-scale detection models in other directions to infer the image (the continuous frame image obtained by the above data cleaning) to obtain the predicted 2D box, including but not limited to pedestrian detection model, vehicle detection model, obstacle detection model, traffic light detection model, etc. These large-scale detection models are obtained through continuous iterative optimization on their own ultra-large-scale datasets. As long as the confidence of the detected 2D box is screened by the threshold, the accuracy requirements can be basically met and used as training data for the large language model.

[0075] 3) Fine-tuning

[0076] The basic model for autonomous driving trained in the previous step has laid a good foundation in terms of image and text understanding, counting, target detection, and knowledge in the field of autonomous driving. Based on it, we continue to use the labeled dataset obtained in the "data preprocessing" step and the online generated dataset to fine-tune the model, and we can quickly train a usable model for our specific autonomous driving task.

[0077] In an embodiment of the present invention, a data construction method for a multimodal large language model for autonomous driving is provided, which includes: S10: data selection, selecting positive sample scene training data and negative sample scene training data from the original video data collected by the vehicle, the positive sample scene refers to a scene where the driving decision needs to be changed during the vehicle's driving, and the negative sample scene refers to a scene where the driving decision does not need to be changed during the vehicle's driving; S20: data cleaning, extracting the positive sample scene training data and the negative sample scene training data selected in step S10 into continuous frame image data at a predetermined frequency, and synchronously cleaning the vehicle speed and navigation instructions of each frame; S30: data labeling, obtaining the true value through manual labeling.

[0078] Further, in step S10, when performing data selection, the selection of positive sample scene training data adopts the following two methods: manual selection, controlling part of the data collection vehicle through the automatic driving algorithm, and manually taking over the vehicle when the automatic driving control algorithm cannot make a reasonable decision, and synchronously generating the data slice at the current moment as the positive sample scene training data; model selection, using the image-text multimodal model to build a picture-text retrieval platform, including but not limited to CLIP, to build a picture-text retrieval platform to retrieve the full amount of original data, wherein before the retrieval, firstly obtain the corresponding image representation vector for all the original image data through the image-text multimodal model, and then obtain the text representation vector for the set keywords through the model, select a certain distance measurement method, including but not limited to the cosine distance, calculate the distance between the image representation vector and the text representation vector, and then obtain the similarity of the two vectors, that is, obtain the semantic similarity between the corresponding image and text, and find the original data of the image corresponding to the keyword on the picture-text retrieval platform according to the similarity threshold.

[0079] Furthermore, in step S30, manual annotation includes the following dimensions: (1) key frame selection: selecting positive sample key frames that require driving decision changes and negative sample key frames that do not require driving decision changes from the continuous frame data obtained in step S20; (2) key targets: using 2D boxes to mark the targets in the positive sample key frames that will affect the driving decision, and giving a description of the relevant content; (3) scene description: used to describe the scene information that can be observed in the current frame image; (4) environment description: used to explain the objective environment information of the current frame image; (5) driving decision meta-operation sequence: including various meta-operation instructions in the driving process, selecting appropriate instructions for orderly arrangement and combination to form a suitable meta-operation sequence; (6) driving decision description: performing a structured decision description based on the driving decision meta-operation sequence, requiring a comprehensive and complete restoration of the meta-operation sequence.

[0080] Furthermore, in step S30, when performing manual labeling, the scene description is in the form of a subjective description question, and the environment description is in the form of an objective multiple-choice question.

[0081] In another embodiment of the present invention, a training method for a multimodal large language model for autonomous driving is provided, which uses the data obtained by the data construction method as described above to perform model training, wherein the training method includes: S11: data preprocessing, obtaining a training data set by processing the labeled data set obtained in step S30 into a format acceptable to the model; S21: pre-training, pre-training the model using a public data set in the field of autonomous driving and an internal target detection data set to obtain an autonomous driving basic model, wherein the internal target detection data set is obtained by performing target detection on the image data of continuous frames obtained in step S20; S31: model fine-tuning, continuing to train the autonomous driving basic model obtained in step S21 with the training data set obtained in step S11 to fine-tune the model.

[0082] Furthermore, in step S11, the labeled data set obtained in step S30 is processed into a format acceptable to the model by organizing objective multiple-choice questions and subjective description questions into question-answer pairs; and a plurality of different question-answer pairs are spliced ​​as a sample in a random combination manner, or the same question is asked multiple times in a sample to generate a training data set online.

[0083] Furthermore, in step S31, the labeled data set obtained through preprocessing in step S11 and the training data set generated online are used to fine-tune the model.

[0084] Furthermore, in step S21, the public data sets used include: a public Chinese and English image description data set, a public target counting data set, a public Chinese and English 2D target detection data set, and a public Chinese and English autonomous driving field knowledge question and answer data set.

[0085] Furthermore, in step S21, the internal target detection data set is obtained by the following two methods: manual labeling, sending the image data of the continuous frames obtained in step S20 to labeling, and the labelers mark the driving-related targets with 2D boxes; the real value large model, performing target detection on the image data of the continuous frames obtained in step S20 through the existing large detection model to obtain the predicted 2D box.

[0086] Beneficial effects brought by the technical solution of the present invention:

[0087] 1) The selection of positive samples in the data set meets the diversity and the data meets the balance;

[0088] 2) Continuous frame data is conducive to the model outputting more reasonable decisions;

[0089] 3) Online generation of training data can greatly expand the data set and improve the generalization ability of the model.

[0090] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data construction method for a multimodal large language model for autonomous driving, characterized in that: include: S10: data selection, selecting positive sample scene training data and negative sample scene training data from the original video data collected by the vehicle, the positive sample scene refers to the scene where the driving decision needs to be changed during the driving process of the vehicle, and the negative sample scene refers to the scene where the driving decision does not need to be changed during the driving process of the vehicle; S20: Data cleaning: extracting the positive sample scene training data and the negative sample scene training data selected in step S10 into image data of continuous frames at a predetermined frequency, and simultaneously cleaning out the vehicle speed and navigation instructions of each frame; S30: Data labeling, obtaining the true value through manual labeling.

2. The data construction method according to claim 1, characterized in that: In step S10, when selecting data, the positive sample scene training data is selected in the following two ways: Manual selection: Use the autonomous driving algorithm to control some data collection vehicles. When the autonomous driving control algorithm cannot make a reasonable decision, the vehicle is manually taken over and the data slice at the current moment is synchronously generated as the positive sample scene training data; Model selection, using the image-text multimodal model to build a image-text retrieval platform to retrieve the full amount of original data. Before the retrieval, the corresponding image representation vectors are obtained for all original image data through the image-text multimodal model, and then the set keywords are used to obtain text representation vectors through the image-text multimodal model. The distance measurement method is used to calculate the distance between the image representation vector and the text representation vector, and then the similarity of the two vectors is obtained, that is, the semantic similarity between the corresponding image and text is obtained, and the original data of the image corresponding to the keyword is found on the image-text retrieval platform according to the similarity threshold.

3. The data construction method according to claim 2, characterized in that: In step S30, manual annotation includes the following dimensions: (1) Key frame selection: selecting positive sample key frames for which the driving decision needs to be changed and negative sample key frames for which the driving decision does not need to be changed from the continuous frame data obtained in step S20; (2) Key targets: Use 2D boxes to mark the targets in the positive sample keyframes that will affect driving decisions, and give descriptions of the relevant content; (3) Scene description: used to describe the scene information that can be observed in the current frame image; (4) Environmental description: used to describe the objective environmental information of the current frame image; (5) Driving decision meta-operation sequence: It includes various meta-operation instructions in the driving process. Appropriate instructions are selected and arranged in order to form an appropriate meta-operation sequence; (6) Driving decision description: A structured decision description is performed based on the driving decision meta-operation sequence, which requires a comprehensive and complete restoration of the meta-operation sequence.

4. The data construction method according to claim 3, characterized in that: In step S30, when performing manual labeling, the scene description is in the form of a subjective description question, and the environment description is in the form of an objective multiple-choice question.

5. A method for training a multimodal large language model for autonomous driving, characterized in that: Model training is performed using data obtained by the data construction method according to any one of claims 1 to 4, wherein the training method comprises: S11: Data preprocessing, obtaining a training data set by processing the labeled data set obtained in step S30 into a format acceptable to the model; S21: Pre-training, using a public data set in the field of autonomous driving and an internal target detection data set to pre-train the model to obtain an autonomous driving basic model, wherein the internal target detection data set is obtained by performing target detection on the image data of the continuous frames obtained in step S20; S31: Model fine-tuning: Continue training the autonomous driving basic model obtained in step S21 using the training data set obtained in step S11 to fine-tune the model.

6. The training method according to claim 5, characterized in that: In step S11, the labeled data set obtained in step S30 is processed into a format acceptable to the model by organizing objective multiple-choice questions and subjective description questions into question-answer pairs; and a plurality of different question-answer pairs are spliced ​​as a sample by random combination, or the same question is repeatedly asked in a sample to generate a training data set online.

7. The training method according to claim 6, characterized in that: In step S31, the labeled data set obtained through preprocessing in step S11 and the training data set generated online are used to fine-tune the model.

8. The training method according to claim 7, characterized in that: In step S21, the public data sets used include: Public Chinese and English image description datasets, public object counting datasets, public Chinese and English 2D object detection datasets, and public Chinese and English autonomous driving knowledge question-answering datasets.

9. The training method according to claim 8, characterized in that: In step S21, the internal object detection dataset is obtained in the following two ways: Manual labeling: sending the image data of the continuous frames obtained in step S20 to labeling, and having the labeler label the driving-related objects with 2D boxes; The real value large model performs target detection on the image data of the continuous frames obtained in step S20 through the existing large detection model to obtain a predicted 2D box.

Images