Vehicle driving track prediction method based on large language model

Abstract

The invention discloses a vehicle driving track prediction method based on a large language model, and belongs to the technical field of automatic driving. The invention aims to provide a vehicle driving track prediction method based on a large language model, which is used for inputting a vehicle track into the large language model in a text format and evaluating the capability of the large language model in understanding the time dynamic aspect of a traffic agent. According to the method, tracks of different vehicles in a traffic scene are encoded by using a track encoder, track encoding features are mapped into a text embedding space of a large language model by using a reprogramming adapter, and the encoding features are translated into lexical combination which can be understood by the large language model. According to the method, powerful support is provided for realizing reliable and safe driving decisions by the large language model.

Description

Technical Field

[0001] The invention belongs to the technical field of automatic driving. Background Art

[0002] Large language models such as ChatGPT have shown great potential in the field of autonomous driving, especially in applications such as human-computer interaction in the cockpit and traffic scene understanding. In terms of human-computer interaction, large language models can achieve personalized user interaction by parsing user commands and combining database-based memory modules, greatly improving the user's driving experience. In terms of scene understanding, large language models can integrate multimodal data from different sensors and combine them with common common sense obtained from network pre-training to understand and reason about long-tail traffic scenes. At the same time, trajectory prediction, as an important part of autonomous driving, aims to provide strong support for path planning and obstacle avoidance of intelligent vehicles by accurately predicting the motion status of various participants such as vehicles and pedestrians on the road. Therefore, how to effectively use large language models to achieve accurate trajectory prediction is a key prerequisite for its safe and reliable application in autonomous driving.

[0003] Patent CN 1 1358 12 13A proposes a method for predicting vehicle motion trajectories. The basic idea of ​​this type of method is to predict trajectories by modeling the physical properties and dynamic equations of moving objects and combining them with observed historical data. However, it is only applicable to simple traffic scenarios with clear rules and few influencing factors, and cannot handle complex traffic scenarios with dense vehicles and pedestrians and changeable environments.

[0004] Patent CN 1 15523934A proposes a vehicle trajectory prediction method and system based on deep learning. This method inputs the position information encoding vector and the lane line information encoding vector into the first-layer recurrent neural network module for training, so that it outputs a latent vector that represents the driving habit decision basis of the target vehicle; then the latent vector and the historical trajectory of the target vehicle are input into the second-layer fully connected neural network module to generate a predicted trajectory. Compared with traditional methods, neural networks predict trajectories in a data-driven manner and no longer rely on manually designed feature extraction or physical models. Typical neural network methods include recurrent neural networks (RNNs), long short-term memory networks (LSTMs), etc., which can automatically learn vehicle motion patterns and time dependencies. However, such methods usually rely on the training of vehicle big data and lack generalization capabilities when dealing with long-tail traffic scenarios.

[0005] In addition to traditional methods and deep learning methods, patent CN 1 18394874A discloses a vehicle trajectory prediction method based on a large language model. The method first obtains the historical trajectory data of the vehicle, generates the historical trajectory after enhancement processing, and inputs it into the large language model to generate the embedding vector of the historical trajectory. The embedded vector is then encoded, and finally the future trajectory is predicted through the decoder. This patent uses a large language model to generate the embedding vector of the historical trajectory, and fails to give full play to the reasoning advantages of the large language model in trajectory prediction, which may lead to insufficient context understanding, limited dynamic learning ability, insufficient cross-domain knowledge integration, and challenges in data sparsity, thereby affecting the overall prediction performance.

[0006] In summary, accurate trajectory prediction is the key to the safe and reliable application of large language models in the field of autonomous driving. However, with the increasing complexity of environmental information such as vehicles and pedestrians, the number of words required to describe also increases. At the same time, considering the limitations of language models in processing numbers, this requires the model to have strong scene understanding and reasoning capabilities, so it faces many difficulties and challenges in the reasoning process. Therefore, it is urgent to explore vehicle trajectory prediction methods based on large language models to help the model accurately predict the behavior of traffic participants, thereby improving the decision-making ability of autonomous driving systems in changeable and complex road conditions, and ultimately achieving an intelligent and safe driving experience. Summary of the invention

[0007] The purpose of this invention is to propose a framework for inputting vehicle trajectories in text format into a large language model and to evaluate the ability of the large language model in understanding the temporal dynamics of traffic agents.

[0008] The steps of the present invention are: S1. Traffic scene data encoding: First, vectorize the target vehicle 0 and its neighboring agents i∈{1,2,…,I} for each timestamp t∈{1,2,…,T} as follows: in represents the spatial coordinates, I and T represent the number of neighboring agents and the observed timestamp, respectively; S2, Interaction encoding: Use a cross-attention layer to encode the interactions between the target vehicle and neighboring agents into vectors Among them, the query vector derived from the target vehicle 0 is Key vector Sum value vector as follows: Among them, Φ: is a multi-layer fully connected neural network. The scenarios are respectively learnable query, key and value weight matrices; Fusion of the interactions between neighboring agents and the spatial state of the target vehicle: In the formula is a learnable vector, represents the Hadamard product; S3, Reprogramming adapter: The scene feature h extracted from equation (4) t Projection into the text embedding space of LLM Further decomposed into two steps S31, Reprogram adapter Reprogram compresses vocabulary symbols and converts them into a set of text prototypes Β prot =W prot Β text (6) in, represents J text words in the LLM vocabulary, W prot ∈R M×J is a trainable matrix, Represents a set of M text prototypes S32, using the cross attention layer, reprogramming the adapter Reprogram the scene features Programming as scene words in, are query, key and value vectors, given by: in, are the learnable weight matrices for query, key, and value conversion, respectively, and the function Φ rep : represents a fully connected network; S4, Linear Track Decoder: in is the processed scene word The frozen LLM trunk; First, Flatten it onto the vector, and then project it onto the vector t using a learnable matrix Project it onto the predicted trajectory where N represents the number of timestamps in the prediction time range.

[0009] The present invention can be directly applied to trajectory prediction without secondary training, which significantly saves training resources; the proposed encoding method simplifies the word units required for scene representation and improves the reasoning speed; the trajectory prediction task is used to quantitatively evaluate the ability of the large language model to understand the dynamic information of the traffic scene, and a simple linear model is used as a decoder to avoid the complexity of the task, thereby more accurately evaluating the prediction ability of the model. Experiments show that the framework is superior to existing methods in terms of prediction accuracy (ADE and FDE indicators) in different time ranges (see Table 1 for details), providing strong support for large language models to achieve reliable and safe driving decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flowchart of a large language model predicting a vehicle’s trajectory; Figure 2 This is a diagram of the vehicle trajectory prediction framework based on the large language model; Figure 3 It is a scene reprogramming processing diagram. DETAILED DESCRIPTION

[0011] The technical problem solved by the present invention is to use the understanding and reasoning capabilities of a large language model to predict vehicle trajectories. Specifically, a framework is proposed to input vehicle trajectories in text format into a large language model and evaluate the ability of the large language model in understanding the temporal dynamics of traffic agents.

[0012] The present invention proposes a technical solution for vehicle trajectory prediction using a large language model. Figure 1 , the specific technical solutions are as follows: Step 1: Use the trajectory encoder to encode the trajectories of different vehicles in the traffic scene and capture the spatial distribution and interaction relationship between multiple vehicles at each moment.

[0013] Step 2: Use the reprogramming adapter to map the trajectory encoding features into the text embedding space of the large language model, and translate the encoding features into word combinations that the large language model can understand.

[0014] Step 3: These reprogrammed scene features, called scene tags, are concatenated with text tags describing the prediction task and then fed into a pre-trained, frozen large language model backbone.

[0015] Step 4: Finally, the decoder that performs matrix transformation converts the output of the large language model into a predicted trajectory.

[0016] The present invention proposes a framework for vehicle trajectory prediction using a large language model (LLM). The specific implementation is as follows: Step 1 Traffic scene data encoding: First, vectorize the target vehicle 0 and its neighboring agents i∈{1,2,…,I} for each timestamp t∈{1,2,…,T} as follows: in represents the spatial coordinates, I and T represent the number of neighboring agents and the observed timestamp, respectively. The displacement vector of agent i and its position relative to the target vehicle are integrated. In addition, all vectorized state vectors are rotated to align with the direction of the target vehicle, ensuring that the trajectory encoding is invariant to both translation and rotation.

[0017] Step 2 Interaction encoding: Use a cross-attention layer to encode the interactions between the target vehicle and neighboring agents into vectors Among them, the query vector derived from the target vehicle 0 is

[0018] The key vector derived from neighboring agent i Sum value vector The calculation is as follows: Among them, Φ: is a multi-layer fully connected neural network (FCN), The scenarios are learnable query, key, and value weight matrices, respectively.

[0019] Finally, a learnable fusion scheme is adopted to fuse the interactions of neighboring agents with the spatial state of the target vehicle as follows: In the formula is a learnable vector, Denotes the Hadamard product.

[0020] Step 3 Reprogramming adapter: The framework proposed in the present invention uses a trainable reprogramming adapter Reprogram: The scene feature h extracted from formula (4) t Projected into the text embedding space of LLM, it is expressed as:

[0021] Specifically, this process can be further decomposed into two steps, such as Figure 3 shown.

[0022] First, to alleviate the complexity of the large and potentially dense reprogramming space, the reprogramming adapter Reprogram compresses and converts vocabulary symbols into a set of textual prototypes. Β prot =W prot Β text (6) in, represents J text words in the LLM vocabulary, W prot ∈R M×J is a trainable matrix, Represents a set of M text prototypes Each text prototype can be viewed as a linear combination of vocabulary tokens.

[0023] Secondly, using the cross-attention layer, the reprogramming adapter Reprograms the scene features Programming as scene words Expressed as: in, are query, key, and value vectors.

[0024] The query, key, and value vectors are represented as: in, are the learnable weight matrices for query, key, and value conversion, respectively. Function Φ rep : Denotes a fully connected network (FCN). In practice, the reprogramming process is implemented using a multi-head cross-attention layer, and Equation (7) illustrates a single-head version.

[0025] In essence, the scene feature h t Reprogram the learning mapping function into a combination of multiple LLM tokens This conversion produces h t The compact text representation of enhances the local semantic richness of the input token and promotes the effective understanding of scene information by LLM.

[0026] Step 4 Linear Trajectory Decoder: The proposed framework uses a simple linear model to decode the output of LLM in the following way: in is the processed scene word The frozen LLM trunk. The decoder LinearDecoder first Flatten it onto the vector, and then project it onto the vector t using a learnable matrix Project it onto the predicted trajectory where N represents the number of timestamps in the prediction time range.

[0027] Step 5: Trajectory data adaptation: Due to the structured and serialized characteristics of trajectory data, the present invention chooses to convert vehicle dynamics into text format and directly input it into the large language model.

[0028] Step 6: Model training: Through the training of encoder, reprogramming adapter, and decoder, the model can understand and predict the trajectory of the vehicle.

[0029] The present invention improves the prediction accuracy and reaction speed in the traffic environment by using a large language model to predict vehicle trajectories, provides rapid support for real-time intelligent driving decisions, and helps to improve the safety and robustness of intelligent driving systems. This method reduces the reliance on high-precision maps, shows strong adaptability and robustness in complex scenarios, and has inspiring significance for the optimization of intelligent transportation systems and multi-vehicle collaborative path planning. At the same time, this method is clear and easy to expand, providing a useful reference for the development and implementation of industry applications and providing new ideas for trajectory prediction.

[0030] This paper exploits the reasoning power of large language models: This paper proposes a framework to predict vehicle trajectories using large, off-the-shelf language models for text only. This is the first attempt to apply these models to understand the temporal dynamics of traffic agents.

[0031] Trainable Reprogramming Adapter: We propose a trainable adapter that transforms features extracted from traffic scenes into text embedding space vectors for a large language model, thereby improving the local semantic richness of input tokens.

[0032] Simplified Linear Trajectory Decoder: To avoid assigning the prediction task to a complex decoder, we adopt a simple linear model to decode the output of the large language model, ensuring that the temporal dependencies are fully managed by the large language model.

[0033] The present invention aims to directly utilize the text representation ability of a large language model to handle trajectory prediction problems, and avoid the interference that may be caused by a complex decoding process through a simple linear decoder, so as to more accurately evaluate the performance of the large language model on the trajectory prediction task.

[0034] Table 1 is the qualitative comparison data of the present invention and other methods Note: ADE±STD(2s) is the average distance error (ADE) and its standard deviation (STD) within 2 seconds. The same applies to 4s and 6s.

[0035] A total of four comparison methods are considered in this work: 1) ZS-LLM, which leverages large language models in zero-shot training by embedding text-formatted trajectories into templates fine-tuned for chat interactions.

[0036] 2) IC-LLM incorporates task-specific examples into the input text and extends ZC-LLM by leveraging the contextual learning capabilities of large language models.

[0037] 3) CoTLLM extends ZC-LLM and leverages the thought chaining capability of large language models to infer future trajectories.

[0038] 4) Time-LLM represents a pioneering approach that leverages off-the-shelf large language models for time series forecasting and is therefore identified as a baseline approach in this work.

[0039] Evaluation Metrics: The trajectory prediction performance is mainly evaluated by the Average Displacement Error (ADE) and Final Displacement Error (FDE) metrics. ADE measures the average Euclidean distance between the predicted waypoints and the true waypoints within a specified prediction horizon. FDE quantifies the Euclidean distance between the final predicted waypoint and the final true waypoint. In addition, the Missing Rate (MR) metric is introduced to quantify the proportion of prediction failures, where a prediction failure occurs when the number of waypoints generated by the large language model deviates from the expected count of the prediction timestamp.

Claims

1. A vehicle driving trajectory prediction method based on a large language model, characterized by: The steps are: S1. Traffic scene data encoding: First, vectorize the target vehicle 0 and its neighboring agents i∈{1,2,…,I} for each timestamp t∈{1,2,…,T} as follows: in represents the spatial coordinates, I and T represent the number of neighboring agents and the observed timestamp, respectively; S2, Interaction encoding: Use a cross-attention layer to encode the interactions between the target vehicle and neighboring agents into vectors Among them, the query vector derived from the target vehicle 0 is Key vector Sum value vector as follows: in, is a multi-layer fully connected neural network. The scenarios are respectively learnable query, key and value weight matrices; Fusion of the interactions between neighboring agents and the spatial state of the target vehicle: In the formula is a learnable vector, represents the Hadamard product; S3, Reprogramming adapter: The scene feature h extracted from equation (4) t Projection into the text embedding space of LLM Further decomposed into two steps S31, Reprogram adapter Reprogram compresses vocabulary symbols and converts them into a set of text prototypes B prot =W prot B text (6) in, represents J text words in the LLM vocabulary, W prot ∈R M×J is a trainable matrix, Represents a set of M text prototypes S32, using the cross attention layer, reprogramming the adapter Reprogram the scene features Programming as scene words in, are query, key and value vectors, given by: in, are the learnable weight matrices for query, key, and value conversion, respectively, and the function Φ rep : represents a fully connected network; S4, linear track decoder: in is the processed scene word b t scene The frozen LLM trunk; First, Flatten it onto the vector, and then project it onto the vector t using a learnable matrix Project it onto the predicted trajectory where N represents the number of timestamps in the prediction time range.

Images