An autonomous driving planning method based on reasoning decision consistency

By constructing a positive and negative sample set and utilizing the cross entropy loss function and the inference decision alignment loss function, the inference and decision consistency problems in autonomous driving planning are solved, and the accuracy and safety of vehicle path prediction are improved.

CN118628890BInactive Publication Date: 2024-11-12SUN YAT SEN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411053899.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing autonomous driving planning methods have consistency problems in the reasoning and decision-making process, resulting in vehicle path prediction errors. Especially in the field of autonomous driving with high safety requirements, the existing methods ignore the inconsistency of causal relationships.

Method used

A self-driving planning method based on inference decision consistency is proposed. By obtaining the training sample data set, a positive and negative sample set is constructed, and the average token score is calculated. The cross entropy loss function and inference decision alignment loss function are used to make the initial autonomous driving planning model. Optimize to ensure consistency in reasoning and decision making.

Benefits of technology

It improves the consistency of model reasoning and decision-making, ensures the accuracy and safety of vehicle path prediction, reduces error values, and improves the reliability of the autonomous driving system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118628890B_ABST
    Figure CN118628890B_ABST
Patent Text Reader

Abstract

The present invention discloses an automatic driving planning method based on the consistency of reasoning and decision making, comprising: obtaining a training sample data set, constructing a first positive and negative sample set, and calculating the average true token score of each first positive and negative sample; obtaining the visual token and language token of the training sample, inputting them into a multimodal large language model, setting a cross entropy loss function for training optimization, outputting corresponding k prediction results, and constructing a second positive and negative sample set, calculating the average predicted token score of each second positive and negative sample; constructing an inference decision alignment loss function according to the average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples, using the inference decision alignment loss function to train and optimize the initial automatic driving planning model, inputting the multi-view image of the self-vehicle to be tested into the trained automatic driving planning model, and obtaining the driving trajectory of the self-vehicle. The present invention can improve the consistency of model reasoning and decision making.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning model algorithm design, and more specifically, to an autonomous driving planning method based on reasoning decision consistency. Background Art

[0002] There are two methods for combining large language models with autonomous driving. One method first uses the basic model to obtain the perception results of the scene's object position information and the prediction results of the object's motion state. These results are converted into text expressions, combined with the prompt words of the thought chain and the vehicle status information, and input into the large language model to obtain the reasoning results and the final predicted trajectory results. The other type of end-to-end planning model does not require an additional basic model to obtain the basic prediction results. It directly inputs the sensor's collection results, uses the encoder to obtain the input token of the large language model, and then obtains the reasoning and trajectory predictions.

[0003] These methods all use the rich knowledge of the large language model itself, adopt a pre-training plus fine-tuning strategy, and use the chain of thought deduction process in the fine-tuning process of the large language model to enhance the model's interpretability and scene understanding capabilities. However, existing methods often ignore the inconsistency problem in causal relationships, which is particularly critical in the field of autonomous driving, which has extremely high safety requirements. Although the model performed correctly in the overall reasoning process, it incorrectly predicted the vehicle's forward path in the final trajectory prediction stage. Summary of the invention

[0004] In order to overcome the above-mentioned problem of inconsistency between reasoning and decision-making in the prior art, the present invention provides an automatic driving planning method based on reasoning and decision-making consistency, which can improve the consistency of reasoning and decision-making.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] The present invention proposes an automatic driving planning method based on reasoning decision consistency, comprising:

[0007] S1: Obtain a training sample data set, each training sample includes a multi-view image of the self-vehicle, a number of preset thought chain prompt words, and a real reasoning result and a real decision result corresponding to each preset thought chain prompt word;

[0008] Construct the first positive and negative sample sets based on the training sample data set, and calculate the average true token score of each first positive and negative sample;

[0009] Constructing an initial autonomous driving planning model, wherein the initial autonomous driving planning model includes a BEV encoder, an Adapter module, a word segmenter, and a multimodal large language model;

[0010] S2: For each training sample, the multi-view image of the self-vehicle is input into the BEV encoder to obtain BEV features, and then the BEV features are input into the Adapter module to obtain corresponding visual tokens; the plurality of preset thinking chain prompt words and the real reasoning results and real decision results corresponding to each preset thinking chain prompt word are input into the word segmenter to obtain corresponding language tokens;

[0011] S3: inputting the visual token and the language token into the multimodal large language model, setting the cross entropy loss function for training optimization, and obtaining a trained multimodal large language model; the trained multimodal large language model outputs k prediction results corresponding to each training sample, each of which includes a prediction reasoning result and a prediction decision result;

[0012] S4: construct a second positive and negative sample set according to all the training samples and their corresponding k prediction results, and calculate the average predicted token score of each second positive and negative sample;

[0013] S5: construct an inference decision alignment loss function according to the average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples, and use the inference decision alignment loss function to train and optimize the initial autonomous driving planning model to obtain a trained autonomous driving planning model;

[0014] S6: Obtain multi-view images of the self-vehicle to be tested, input them into the trained autonomous driving planning model, and obtain the driving trajectory of the self-vehicle.

[0015] Preferably, in S1, the plurality of preset thought chain prompt words include:

[0016] Identify key objects and their locations, predict key object motion and trajectory, predict ego vehicle driving behavior and predict ego vehicle trajectory.

[0017] Preferably, in S1, constructing a first positive and negative sample set according to the training sample data set includes:

[0018] For two training samples of the same scene in the training sample dataset and , Indicates training samples, Indicates training samples, , Respectively represent , Multi-view images of the ego vehicle in training samples, , Respectively represent The actual decision results in the training samples are , Respectively represent , Several preset thinking chain prompt words in the training samples and the real reasoning results corresponding to each preset thinking chain prompt word;

[0019] Combine several preset thinking chain prompt words in the two training samples and the real reasoning results and real decision results corresponding to each preset thinking chain prompt word to obtain a negative sample , ,in, Composition of positive samples Negative sample set , Composition of positive samples Negative sample set , and , and Both are the first positive and negative samples.

[0020] Preferably, the average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples are calculated in the same way, including:

[0021]

[0022] in, represents the average true token score of the first positive and negative samples or the average predicted token score of the second positive and negative samples. represents the multi-view image of the ego vehicle, Indicates several preset thinking chain prompt words and the reasoning results and decision results corresponding to each preset thinking chain prompt word, represents the model weight, Tokens representing the first positive and negative samples or the second positive and negative samples, including a number of preset thought chain prompt words and the language tokens corresponding to the reasoning results and decision results corresponding to each preset thought chain prompt word, is the number of tokens, express No. Tokens, express No. All tokens before the token, Indicated in , , as well as Prediction under the condition probability.

[0023] Preferably, in S2, for each training sample, the multi-view image of the ego vehicle is input into a BEV encoder to obtain a BEV feature, and then the BEV feature is input into an Adapter module to obtain a corresponding visual token, including:

[0024] The multi-view image of the ego vehicle is input into the BEV encoder to obtain the BEV features, which are divided into grids of preset size, and the features of each point in the grid are flattened and spliced, and then input into the Adapter module in units of grids to generate visual tokens.

[0025] Preferably, in S3, the method for determining the cross entropy loss function includes:

[0026]

[0027] in, represents the cross entropy loss function, Indicates the training sample data set tokens of training samples, For the The length of tokens of training samples, express No. Tokens, express No. All tokens before the token, express No. The tokens before the token correspond to several preset thought chain prompts and the real reasoning results and real decision results corresponding to each thought chain prompt. Indicates the training sample data set training samples of multi-view images of the ego vehicle, Indicates that the weight of the multimodal large language model is , when inputting , and Predicted under the condition probability.

[0028] Preferably, in S4, constructing a second positive and negative sample set according to all the training samples and their corresponding k prediction results includes:

[0029] The k prediction results corresponding to each training sample are used as negative samples. The error value between the predicted decision result in each prediction result and the actual decision result of the training sample is calculated respectively. The k prediction results are sorted from small to large according to the error value to obtain the sorted k prediction results. , the corresponding sorted tokens of the k prediction results are ,in Indicates the prediction result of sequence number k, Tokens representing the prediction results with sequence number k, It means that the prediction result with sequence number 1 is ranked before the prediction result with sequence number 2, that is, the error value between the prediction decision result in the prediction result with sequence number 1 and the actual decision result of the training sample is less than or equal to the error value between the prediction decision result in the prediction result with sequence number 2 and the actual decision result of the training sample. The tokens representing the prediction result with sequence number 1 are ranked before the tokens representing the prediction result with sequence number 2;

[0030] Each training sample and the corresponding k sorted prediction results constitute a second positive and negative sample, and all the training samples and the corresponding k sorted prediction results constitute a second positive and negative sample set.

[0031] Preferably, in S5, the method for determining the inference decision alignment loss function includes:

[0032]

[0033] in, represents the inference decision alignment loss function, represents the binary alignment loss function, represents the ranking alignment loss function.

[0034] Preferably, the method for determining the binary alignment loss function includes:

[0035]

[0036] in, represents the binary alignment loss function, Indicates the first positive and negative sample set The tokens of the positive samples in the first positive and negative samples, Indicates the first positive and negative sample set The tokens set corresponding to the negative sample set of the first positive and negative samples, express Middle n Tokens of negative samples, represents a gradient separation operation, and Respectively and The average true token score.

[0037] Preferably, the method for determining the sorting alignment loss function includes:

[0038]

[0039] in, , Respectively represent the negative samples of the second positive and negative samples The tokens of the prediction results and the Tokens of prediction results, Indicates the negative sample of the second positive and negative sample The tokens of the predicted results are ranked first in the negative samples of the second positive and negative samples. Before the tokens of the prediction results; express The average predicted token score, express The average predicted token score.

[0040] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0041] The present invention proposes an autonomous driving planning method based on inference decision consistency, firstly obtaining a training sample data set, constructing a first positive and negative sample set, and calculating the average true token score of each first positive and negative sample; obtaining the visual token and language token of the training sample, inputting them into a multimodal large language model, setting a cross entropy loss function for training optimization, outputting the corresponding k prediction results, and constructing a second positive and negative sample set, calculating the average predicted token score of each second positive and negative sample; constructing an inference decision alignment loss function according to the average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples, using the inference decision alignment loss function to train and optimize the initial autonomous driving planning model, inputting the multi-view image of the self-vehicle to be tested into the trained autonomous driving planning model, and obtaining the driving trajectory of the self-vehicle. The present invention can improve the consistency of model reasoning and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of the automatic driving planning method based on reasoning decision consistency described in Example 1;

[0043] Figure 2 This is an overall block diagram of the autonomous driving planning method based on reasoning decision consistency described in Example 2;

[0044] Figure 3 The multi-view image of the self-vehicle described in Example 2;

[0045] Figure 4 The driving trajectory diagram of the self-vehicle described in Example 2;

[0046] Figure 5 The thought chain prompt words and the prediction and reasoning result diagram corresponding to each prompt word described in Example 2;

[0047] Figure 6 This is the thought chain prompt words and predicted decision result diagram described in Example 2. DETAILED DESCRIPTION

[0048] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0049] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0050] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0051] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0052] Example 1

[0053] This embodiment provides an automatic driving planning method based on reasoning decision consistency, such as Figure 1 As shown, including:

[0054] S1: Obtain a training sample data set, each training sample includes a multi-view image of the self-vehicle, a number of preset thought chain prompt words, and a real reasoning result and a real decision result corresponding to each preset thought chain prompt word;

[0055] Construct the first positive and negative sample sets based on the training sample data set, and calculate the average true token score of each first positive and negative sample;

[0056] Constructing an initial autonomous driving planning model, wherein the initial autonomous driving planning model includes a BEV encoder, an Adapter module, a word segmenter, and a multimodal large language model;

[0057] S2: For each training sample, the multi-view image of the self-vehicle is input into the BEV encoder to obtain BEV features, and then the BEV features are input into the Adapter module to obtain corresponding visual tokens; the plurality of preset thinking chain prompt words and the real reasoning results and real decision results corresponding to each preset thinking chain prompt word are input into the word segmenter to obtain corresponding language tokens;

[0058] S3: inputting the visual token and the language token into the multimodal large language model, setting the cross entropy loss function for training optimization, and obtaining a trained multimodal large language model; the trained multimodal large language model outputs k prediction results corresponding to each training sample, each of which includes a prediction reasoning result and a prediction decision result;

[0059] S4: construct a second positive and negative sample set according to all the training samples and their corresponding k prediction results, and calculate the average predicted token score of each second positive and negative sample;

[0060] S5: construct an inference decision alignment loss function according to the average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples, and use the inference decision alignment loss function to train and optimize the initial autonomous driving planning model to obtain a trained autonomous driving planning model;

[0061] S6: Obtain multi-view images of the self-vehicle to be tested, input them into the trained autonomous driving planning model, and obtain the driving trajectory of the self-vehicle.

[0062] In the specific implementation process, first obtain the training sample data set, then construct the first positive and negative sample set, and construct the inference decision alignment loss function according to the average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples. Use the inference decision alignment loss function to train and optimize the initial autonomous driving planning model to obtain the trained autonomous driving planning model. Input the multi-view image of the self-vehicle to be tested into the trained autonomous driving planning model to obtain the driving trajectory of the self-vehicle.

[0063] Example 2

[0064] This embodiment provides an autonomous driving planning method based on reasoning decision consistency, including:

[0065] S1: Obtain a training sample data set, each training sample includes a multi-view image of the self-vehicle, a number of preset thought chain prompt words, and the real reasoning results and real decision results corresponding to each prompt word;

[0066] Construct the first positive and negative sample sets based on the training sample data set, and calculate the average true token score of each first positive and negative sample;

[0067] Constructing an initial autonomous driving planning model, wherein the initial autonomous driving planning model includes a BEV encoder, an Adapter module, a word segmenter, and a multimodal large language model;

[0068] The preset thought chain prompt words are:

[0069] Identify key objects and their locations, predict key object motion and trajectory, predict ego vehicle driving behavior and predict ego vehicle trajectory.

[0070] The actual decision result is the driving trajectory of the self-vehicle. In the process of human driving, multiple factors are usually considered when making driving decisions. For example, people usually first perceive nearby key objects (such as cars, pedestrians, traffic lights, etc.), then determine the specific meaning of these objects or judge their driving trends, and finally infer their specific driving behavior and plan their driving routes. In order to enable the model to have similar explanatory capabilities, this embodiment introduces a thinking chain prompt word strategy, which first imitates the first step of human perception tasks, guides the model to identify key objects, and then continues to perform fine-grained motion predictions on these objects, distinguishing the relationship and priority of each object's impact on the self-driving car, thereby inferring the self-driving car's speed and direction consistent with the final planned route. Driving behavior.

[0071] Identify key objects and their locations: When driving on the road, humans instinctively identify key aspects of the scene, focusing on elements that directly affect our driving. These factors may include traffic signals or other situations that interact with the self-vehicle. Identifying key objects in the scene is the basis for understanding the driving environment and planning driving behavior. This embodiment uses "What are the important objects in the current scene? These objects will be used for future reasoning and driving decisions" as a prompt in the perception part, requiring the system to identify important objects in the scene and their locations.

[0072] Predicting the movement and trajectory of key objects: How to handle interactions with key objects is an important aspect of how people understand scenes. Drivers will focus on elements that may affect driving decisions, logically describe the specific meanings of these objects, and prioritize these instances. For example, in a traffic light scenario, different colors are the determining factors for driving behavior, and pedestrians in front are more worthy of attention than parallel vehicles. The system accurately analyzes driving interactions as part of the thinking chain, with prompts like "When the vehicle drives to the next possible location, which object should the vehicle pay attention to first / secondary / third? What state is the object that the vehicle pays attention to first / secondary / third now, and what response should the vehicle make?" Logical reasoning can guide the model to make correct judgments and avoid hidden risks.

[0073] Predicting the driving behavior of the self-vehicle: Determine the appropriate driving action based on the perception results and the predicted future state. The decision-making process of human driving is highly logical and inductive, involving the driver's ability to analyze various factors and come up with appropriate actions. Therefore, we choose a simple prompt "predict the behavior of the self-vehicle" to let the autonomous driving system summarize the perception and prediction results and complete the underlying thinking process. The model needs to answer the direction of the self-vehicle (such as forward or turning) and the speed of the self-vehicle (such as stopping or accelerating). This logical ability to imitate human decision-making makes the system's judgment more reliable.

[0074] Planning driving trajectory: The final output of the complete driving model should be a direct output of the control signal or trajectory. Therefore, while ensuring the correctness of natural language logic reasoning, it is necessary to output the future driving trajectory by prompting "planning a safe and feasible 3-second trajectory including 6 coordinate points".

[0075] It can be seen that the multi-round thinking chain prompt words take into account the perception, prediction and decision-making parts of the autonomous driving model.

[0076] During driving, there will be samples with similar scenes but completely different decisions. This embodiment combines the inference results and decision results of similar scenes in the training set to form a set of positive and negative samples. and , Indicates training samples, Indicates training samples, , Respectively represent , Multi-view images of the ego vehicle in training samples, , Respectively represent The actual decision results in the training samples are , Respectively represent , Several preset thinking chain prompt words in the training samples and the real reasoning results corresponding to each preset thinking chain prompt word;

[0077] Combine several preset thinking chain prompt words in the two training samples and the real reasoning results and real decision results corresponding to each preset thinking chain prompt word to obtain a negative sample , ,in, Composition of positive samples Negative sample set , Composition of positive samples Negative sample set , and , and Both are the first positive and negative samples.

[0078] S2: For each training sample, the multi-view image of the self-vehicle is input into the BEV encoder to obtain the BEV feature, and then the BEV feature is input into the Adapter module to obtain the corresponding visual token; the plurality of preset thought chain prompt words and the real reasoning result and the real decision result corresponding to each prompt word are input into the word segmenter to obtain the corresponding language token;

[0079] Among visual encoders, the pre-trained CLIP model is the most widely used encoder. However, predicting the trajectory requires predicting the trajectory coordinate points under the BEV perspective. Since the end-to-end framework inputs multi-view images taken from the camera perspective, there is a perspective conversion problem with the final predicted BEV perspective. For a multimodal large language model, it is more difficult to learn the perspective conversion between the two. Therefore, this embodiment uses the BEV encoder to encode the multi-view images to obtain BEV features and finally convert them into visual tokens. For the multi-view images of the ego vehicle, if these images are to be uniformly transformed to the BEV perspective, it is necessary to project each pixel in the two-dimensional image into the three-dimensional voxel space according to the predicted depth distribution of each pixel, and then use the sum pooling operation to integrate the three-dimensional voxels obtained from each perspective to finally obtain the BEV features. ,in is the feature dimension, and In order to make the BEV encoder have stronger semantic expression ability, this embodiment uses target detection as a task to pre-train the encoder.

[0080] The BEV features obtained from multi-view images need to be converted into token inputs that can be accepted by the large language model. The most widely used method is to use the attention method for token expression. However, this method will lose some fine-grained detail information on the one hand, and the training samples are relatively limited on the other hand. Therefore, it is difficult to train such a complex module. The method proposed in this embodiment adopts a simple and efficient region encoding method. For the BEV features, we use The image is divided into grids, and the features of each point in the grid are flattened and spliced, and visual tokens are generated in grid units. Correspondingly, we can get the visual input token as , thereby increasing the number of tokens from Reduce to Next, a 2-layer fully connected layer is used to align the feature dimensions of the visual token and the language token.

[0081] S3: inputting the visual token and the language token into the multimodal large language model, setting the cross entropy loss function for training optimization, and obtaining a trained multimodal large language model; the trained multimodal large language model outputs k prediction results corresponding to each training sample, each of which includes a prediction reasoning result and a prediction decision result;

[0082] An open source model such as Llama can be used as a multimodal large language model. The cross entropy loss function calculation formula is:

[0083]

[0084] in, represents the cross entropy loss function, Indicates the training sample data set tokens of training samples, For the The length of tokens of training samples, express No. Tokens, express No. All tokens before the token, express No. The tokens before the token correspond to several preset thought chain prompts and the real reasoning results and real decision results corresponding to each thought chain prompt. Indicates the training sample data set training samples of multi-view images of the ego vehicle, Indicates that the weight of the multimodal large language model is , when inputting , and Predicted under the condition probability.

[0085] First, an initial model is fine-tuned using the cross entropy loss in the training sample data set, and then this initial model is used to generate k different prediction results for each training sample in the training sample data set. Then, these prediction results are sorted based on the error between the predicted decision results and the actual decision results, and finally generated data of different qualities can be obtained.

[0086] S4: construct a second positive and negative sample set according to all the training samples and their corresponding k prediction results, and calculate the average predicted token score of each second positive and negative sample;

[0087] The average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples are calculated in the same way:

[0088]

[0089] in, represents the average true token score of the first positive and negative samples or the average predicted token score of the second positive and negative samples. represents the multi-view image of the ego vehicle, Indicates several preset thinking chain prompt words and the reasoning results and decision results corresponding to each preset thinking chain prompt word, represents the model weight, Tokens representing the first positive and negative samples or the second positive and negative samples, including a number of preset thought chain prompt words and the language tokens corresponding to the reasoning results and decision results corresponding to each preset thought chain prompt word, is the number of tokens, express No. Tokens, express No. All tokens before the token, Indicated in , , as well as Prediction under the condition probability.

[0090] The k prediction results corresponding to each training sample are used as negative samples. The error value between the predicted decision result in each prediction result and the actual decision result of the training sample is calculated respectively. The k prediction results are sorted from small to large according to the error value to obtain the sorted k prediction results. , the corresponding sorted tokens of the k prediction results are ,in Indicates the prediction result of sequence number k, Tokens representing the prediction results with sequence number k, It means that the prediction result with sequence number 1 is ranked before the prediction result with sequence number 2, that is, the error value between the prediction decision result in the prediction result with sequence number 1 and the actual decision result of the training sample is less than or equal to the error value between the prediction decision result in the prediction result with sequence number 2 and the actual decision result of the training sample. The tokens representing the prediction result with sequence number 1 are ranked before the tokens representing the prediction result with sequence number 2;

[0091] Each training sample and the corresponding k sorted prediction results constitute a second positive and negative sample, and all the training samples and the corresponding k sorted prediction results constitute a second positive and negative sample set.

[0092] S5: construct an inference decision alignment loss function according to the average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples, and use the inference decision alignment loss function to train and optimize the initial autonomous driving planning model to obtain a trained autonomous driving planning model;

[0093] The model needs to ensure that the token score of the positive sample is higher than the token score of the negative sample to ensure the consistency of reasoning and decision. Specifically, for the first positive and negative samples, this embodiment uses a simple binary alignment loss for training: for the second positive and negative samples, a sorting alignment loss is used to ensure that the token score of the positive sample is higher than that of the negative sample. The reasoning decision alignment loss function is:

[0094]

[0095] in, represents the inference decision alignment loss function, represents the binary alignment loss function, represents the ranking alignment loss function. The calculation formula is:

[0096]

[0097] in, represents the binary alignment loss function, Indicates the first positive and negative sample set The tokens of the positive samples in the first positive and negative samples, Indicates the first positive and negative sample set The tokens set corresponding to the negative sample set of the first positive and negative samples, express Middle n Tokens of negative samples, represents a gradient separation operation, and Respectively and The average true token score.

[0098] The calculation formula is:

[0099]

[0100] in, , Respectively represent the negative samples of the second positive and negative samples The tokens of the prediction results and the Tokens of prediction results, Indicates the negative sample of the second positive and negative sample The tokens of the predicted results are ranked first in the negative samples of the second positive and negative samples. Before the tokens of the prediction results; express The average predicted token score, express The average predicted token score.

[0101] S6: Obtain multi-view images of the self-vehicle to be tested, input them into the trained autonomous driving planning model, and obtain the driving trajectory of the self-vehicle.

[0102] like Figure 2 As shown, it is an overall block diagram of the method proposed in this embodiment. During the specific implementation process, the driving model needs to accurately perceive the surrounding vehicles and understand the changes in the surrounding environment, and make appropriate decisions. The thought chain prompt words establish a causal relationship between decision-making and reasoning, helping the model to understand the surrounding environment, so as to effectively plan the driving route. Through thought chain reasoning, the model can more accurately predict the possible consequences in various situations and make corresponding adjustments, thereby improving the safety and reliability of the model. During the reasoning process, it is necessary to ensure that the reasoning and decision of the model are consistent rather than contradictory. This is a constraint condition missing in the original large language model prediction. In order to ensure the consistency of thought chain reasoning and decision-making, this embodiment adopts the reasoning decision alignment loss function to achieve it. As shown in Figure 3 As shown, it is a multi-view image of the self-vehicle. Figure 4 The predicted decision result of this embodiment is the planned driving trajectory of the self-vehicle within 3s, which is the two-dimensional coordinate value of 6 points. Figure 5 The present invention relates to a thinking chain prompt word and a predicted reasoning result corresponding to each thinking chain prompt word, including a first thinking chain prompt word, a second thinking chain prompt word and a third thinking chain prompt word, and a predicted reasoning result corresponding to each thinking chain prompt word. Figure 6 To predict the decision result diagram, the answer corresponding to the fourth thinking chain prompt word is the predicted decision result, that is, the two-dimensional coordinate values ​​of the 6 points. Figure 5 The predicted reasoning result of the third thought chain prompt word is that the vehicle is turning left and the vehicle is moving slowly. Figure 6The two-dimensional coordinate values ​​of the six points in the predicted decision result are not all 0, that is, the reasoning and decision are consistent. The model can perceive the traffic lights and vehicles that affect driving and correctly analyze their meanings and behaviors, and finally make the correct decision. If the predicted self-vehicle driving behavior in the predicted reasoning result is stationary, and the predicted decision result, that is, the two-dimensional coordinate values ​​of the six points within 3s of the self-vehicle, are not all 0, then this situation belongs to the situation where the reasoning and decision are inconsistent, because the two-dimensional coordinate values ​​of the six points are all 0, which means that the self-vehicle remains stationary. Similarly, if the predicted self-vehicle driving behavior in the predicted reasoning result is the movement of the self-vehicle, and the predicted decision result, that is, the two-dimensional coordinate values ​​of the six points within 3s of the self-vehicle, are all 0, this situation also belongs to the situation where the reasoning and decision are inconsistent. Through the above correct reasoning analysis, the model can plan the correct trajectory to ensure that the reasoning result and the final decision are consistent.

[0103] Example 3

[0104] This embodiment conducts a simulation experiment on the method proposed in Embodiment 2, using the outdoor real scene autonomous driving nuScenes dataset and DriveLM-nuScenes dataset as training datasets. The nuScenes dataset is an authoritative dataset in the field of autonomous driving. This dataset is a multi-sensor dataset containing 1000 scenes, each lasting 20 seconds, with a total of 28130 training samples and 6019 verification samples.

[0105] The nuScenes dataset was used as a test set to experimentally test the method proposed in Example 2. The data were grouped according to different evaluation methods and compared with the prior art. The performance indicators are shown in Table 1. The decision result of this embodiment is to plan the driving trajectory of the ego vehicle within 3s. Two indicators are used to evaluate the quality of the driving trajectory: L2 error and collision rate, where the L2 error measures the similarity between the predicted driving trajectory and the actual driving trajectory. In order to determine the collision frequency between the ego vehicle and other objects, this embodiment places a box representing the ego vehicle at each path point of the predicted trajectory and checks whether it collides with the boxes of other objects in the scene. Similar to most trajectory prediction work in nuScenes, this embodiment evaluates the planning results within a 3-second time frame and evaluates the quality of the driving trajectory for time ranges of 1 second, 2 seconds, and 3 seconds.

[0106] This embodiment adopts two evaluation methods (ST-P3 evaluation method and UniAD evaluation method) to make fair comparisons with different models. As can be seen from Table 1, compared with traditional methods such as ST-P3 and UniAD that train on all 28,130 data in the training set, the method proposed in this application only uses 15% of the training data containing reasoning annotations for training, and finally achieves good results in both L2 error and collision rate. Compared with VAD, the traditional model with the best performance in the ST-P3 evaluation series, the collision rate of the method proposed in this application drops from 0.14 to 0.10; compared with the traditional UniAD model with the best performance in the UniAD evaluation series, the L2 error of the method proposed in this application drops from 1.03 to 0.80.

[0107] In addition, the GPT-Driver model combined with a large language model uses ChatGPT, which requires the integration of the results of other basic models (such as perception and prediction). In contrast, the method proposed in this application has achieved excellent performance in both indicators of the two evaluation methods without relying on these prior knowledge. Specifically, in the ST-P3 evaluation method, the L2 error and collision rate of the method proposed in this application are reduced by 10% and 41% respectively compared with GPT-Driver; in the UniAD evaluation method, the L2 error and collision rate of the method proposed in this application are reduced by 5% and 28% respectively compared with GPT-Driver. The DME-Driver model combines an end-to-end autonomous driving decision model with a large language model. It is similar in structure to the model of the method proposed in this application. It also inputs multimodal information into a large language model for end-to-end autonomous driving decision-making. The L2 error of the method proposed in this application is significantly better than that of DME-Driver (0.80 vs 0.98), and can achieve basically the same effect in terms of collision rate.

[0108] In two different evaluation methods, the L2 error and collision rate of the method proposed in this application are comparable to those of existing methods, and most performance indicators are better than existing methods. It is worth noting that the collision rate of the method proposed in this application is 0 within 1 second, which exceeds the performance of all existing models. The experimental results show that the method proposed in this application can effectively utilize the knowledge contained in the large language model to complete the autonomous driving planning task. It is feasible to use the method proposed in this application to generate human-like driving trajectories and has the ability to plan safe driving paths. The method proposed in this application achieves a lower collision rate and significantly reduces the L2 error by constructing an inference-decision alignment loss function, which indirectly proves the importance of consistency between reasoning and decision-making for autonomous driving.

[0109]

[0110] The DriveLM-nuScenes dataset is obtained by selecting key frames from the complete nuScenes data, in which the intention of the self-vehicle changes significantly, and the historical records cannot effectively predict future behavior or driving trajectory, which helps to alleviate the "self-state" problem. There are 4072 training samples and 799 validation samples in the dataset, covering all aspects of the driving process, including perception, prediction, and planning.

[0111] In order to alleviate the problem of dependence on the state of the self-vehicle in the open-loop evaluation, the DriveLM-nuScenes dataset is used as a test set to experimentally test the method proposed in Example 2, and compared with the prior art, and grouped according to different evaluation methods. The performance indicators are shown in Table 2. The end-to-end models UniAD and ST-P3 based on learning have poor test results under this dataset. It can be seen that the L2 error of the UniAD model increased from 1.03 to 3.00, and the collision rate increased from 0.31 to 0.72, which also proves that the traditional model does have the problem of dependence on the state of the self-vehicle. Similarly, the collision rate of GPT-Driver combined with a large language model also changed significantly, from 0.44 to 0.67. Compared with these methods, the method proposed in this application has achieved the best performance in all evaluation methods and all indicators in this dataset. For the ST-P3 evaluation method, the L2 error of the method proposed in this application is 0.41, and the collision rate is 0.14; for the UniAD evaluation method, the L2 error of the method proposed in this application is 0.82, and the collision rate is 0.38. Moreover, the proposed method in this application only shows slight fluctuations in the two datasets of nuScenes and DriveLM-nuScenes, with the corresponding L2 error fluctuations of 0.01 and 0.02, and the corresponding collision rate fluctuations of 0.04 and 0.06, respectively. Therefore, it can be seen that the proposed method in this application performs better than other existing methods and alleviates the problem of the model being dependent on the state of the vehicle.

[0112] Table 2 Performance indicators of the proposed method on the DriveLM-nuScenes dataset

[0113]

[0114] The same or similar reference numerals correspond to the same or similar components;

[0115] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0116] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An autonomous driving planning method based on reasoning decision consistency, characterized in that: include: S1: Obtain a training sample data set, each training sample includes a multi-view image of the self-vehicle, a number of preset thought chain prompt words, and a real reasoning result and a real decision result corresponding to each preset thought chain prompt word; Construct the first positive and negative sample sets based on the training sample data set, and calculate the average true token score of each first positive and negative sample; Constructing an initial autonomous driving planning model, wherein the initial autonomous driving planning model includes a BEV encoder, an Adapter module, a word segmenter, and a multimodal large language model; S2: For each training sample, the multi-view image of the self-vehicle is input into the BEV encoder to obtain BEV features, and then the BEV features are input into the Adapter module to obtain corresponding visual tokens; the plurality of preset thinking chain prompt words and the real reasoning results and real decision results corresponding to each preset thinking chain prompt word are input into the word segmenter to obtain corresponding language tokens; S3: inputting the visual token and the language token into the multimodal large language model, setting the cross entropy loss function for training optimization, and obtaining a trained multimodal large language model; the trained multimodal large language model outputs k prediction results corresponding to each training sample, each of which includes a prediction reasoning result and a prediction decision result; S4: construct a second positive and negative sample set according to all the training samples and their corresponding k prediction results, and calculate the average predicted token score of each second positive and negative sample; S5: construct an inference decision alignment loss function according to the average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples, and use the inference decision alignment loss function to train and optimize the initial autonomous driving planning model to obtain a trained autonomous driving planning model; S6: Obtain multi-view images of the self-vehicle to be tested, input them into the trained autonomous driving planning model, and obtain the driving trajectory of the self-vehicle; In S5, the method for determining the inference decision alignment loss function includes: in, represents the inference decision alignment loss function, represents the binary alignment loss function, represents the sorting alignment loss function; The method for determining the binary alignment loss function includes: in, represents the binary alignment loss function, Indicates the first positive and negative sample set The tokens of the positive samples in the first positive and negative samples, Indicates the first positive and negative sample set The tokens set corresponding to the negative sample set of the first positive and negative samples, express Middle n Tokens of negative samples, represents a gradient separation operation, and Respectively and The average real token score of The method for determining the sorting alignment loss function includes: in, , Respectively represent the negative samples of the second positive and negative samples The tokens of the prediction results and the Tokens of prediction results, Indicates the negative sample of the second positive and negative sample The tokens of the predicted results are ranked first in the negative samples of the second positive and negative samples. Before the tokens of the prediction results; express The average predicted token score, express The average predicted token score.

2. The method for autonomous driving planning based on reasoning decision consistency according to claim 1, characterized in that: In S1, the plurality of preset thought chain prompt words include: Identify key objects and their locations, predict key object motion and trajectory, predict ego vehicle driving behavior and predict ego vehicle trajectory.

3. The method for autonomous driving planning based on reasoning decision consistency according to claim 2, characterized in that: In S1, constructing a first positive and negative sample set according to the training sample data set includes: For two training samples of the same scene in the training sample dataset and , Indicates training samples, Indicates training samples, , Respectively represent , Multi-view images of the ego vehicle in training samples, , Respectively represent The actual decision results in the training samples are , Respectively represent , Several preset thinking chain prompt words in the training samples and the real reasoning results corresponding to each preset thinking chain prompt word; Combine several preset thinking chain prompt words in the two training samples and the real reasoning results and real decision results corresponding to each preset thinking chain prompt word to obtain a negative sample , ,in, Composition of positive samples Negative sample set , Composition of positive samples Negative sample set , and , and Both are the first positive and negative samples.

4. The method for autonomous driving planning based on reasoning decision consistency according to claim 3, characterized in that: The average true token score of the first positive and negative samples and the average predicted token score of the second positive and negative samples are calculated in the same way, including: in, represents the average true token score of the first positive and negative samples or the average predicted token score of the second positive and negative samples. represents the multi-view image of the ego vehicle, Indicates several preset thinking chain prompt words and the reasoning results and decision results corresponding to each preset thinking chain prompt word, represents the model weight, Tokens representing the first positive and negative samples or the second positive and negative samples, including a number of preset thought chain prompt words and the language tokens corresponding to the reasoning results and decision results corresponding to each preset thought chain prompt word, is the number of tokens, express No. Tokens, express No. All tokens before the token, Indicated in , , as well as Prediction under the condition probability.

5. The method for autonomous driving planning based on reasoning decision consistency according to claim 3, characterized in that: In S2, for each training sample, the multi-view image of the ego vehicle is input into the BEV encoder to obtain the BEV feature, and then the BEV feature is input into the Adapter module to obtain the corresponding visual token including: The multi-view image of the ego vehicle is input into the BEV encoder to obtain the BEV features, which are divided into grids of preset size, and the features of each point in the grid are flattened and spliced, and then input into the Adapter module in units of grids to generate visual tokens.

6. The method for autonomous driving planning based on reasoning decision consistency according to claim 5, characterized in that: In S3, the method for determining the cross entropy loss function includes: in, represents the cross entropy loss function, Indicates the training sample data set tokens of training samples, For the The length of tokens of training samples, express No. Tokens, express No. All tokens before the token, express No. The tokens before the token correspond to several preset thought chain prompts and the real reasoning results and real decision results corresponding to each thought chain prompt. Indicates the training sample data set training samples of multi-view images of the ego vehicle, Indicates that the weight of the multimodal large language model is , when inputting , and Predicted under the condition probability.

7. The method for autonomous driving planning based on reasoning decision consistency according to claim 6, characterized in that: In S4, constructing a second positive and negative sample set according to all the training samples and their corresponding k prediction results includes: The k prediction results corresponding to each training sample are used as negative samples. The error value between the predicted decision result in each prediction result and the actual decision result of the training sample is calculated respectively. The k prediction results are sorted from small to large according to the error value to obtain the sorted k prediction results. , the corresponding sorted tokens of the k prediction results are ,in Indicates the prediction result of sequence number k, Tokens representing the prediction results with sequence number k, It means that the prediction result with sequence number 1 is ranked before the prediction result with sequence number 2, that is, the error value between the prediction decision result in the prediction result with sequence number 1 and the actual decision result of the training sample is less than or equal to the error value between the prediction decision result in the prediction result with sequence number 2 and the actual decision result of the training sample. The tokens representing the prediction result with sequence number 1 are ranked before the tokens representing the prediction result with sequence number 2; Each training sample and the corresponding k sorted prediction results constitute a second positive and negative sample, and all the training samples and the corresponding k sorted prediction results constitute a second positive and negative sample set.