Interactive vehicle trajectory prediction method based on multi-modal large language model
Through the multimodal large language model, the problem of inaccurate vehicle trajectory prediction in the prior art is solved, and high-quality and interpretable vehicle trajectory prediction is achieved.
Patent Information
- Application Number
- CN202510803231.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-16
- Publication Date
- 2025-09-02
AI Technical Summary
The existing large language models rely only on single modal input in vehicle trajectory prediction, do not fully utilize map information and cannot clearly express the interactive behavior between traffic participants, thus resulting in inaccurate vehicle trajectory prediction results.
A multimodal large language model is adopted to extract map features through an image encoder, and a text encoder extracts the state and interaction information of moving objects, and performs feature fusion in the shared embedding space. The large language model is input to generate vehicle behavior descriptions to predict trajectories.
Improves the accuracy and interpretability of vehicle trajectory prediction, enhances environmental perception, and can capture complex interactions between multiple vehicles.
Smart Images
Figure CN120578728A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of vehicle trajectory prediction, and in particular relates to an interactive vehicle trajectory prediction method based on a multimodal large language model. Background Art
[0002] During the development of autonomous driving systems, the construction of simulation scenarios is crucial for algorithm testing and validation. Ideal simulation scenarios should be high-fidelity, interactive, and controllable. In recent years, with the rapid development of large language models (LLMs) and multimodal large language models (MLLMs), some research efforts have attempted to incorporate them into scenario generation tasks. These models can generate traffic scenarios based on language descriptions, enabling a mapping between abstract and interactive trajectories. Furthermore, trained LLMs can be used to generate vehicle behavior in actual autonomous driving scenarios, thereby predicting vehicle trajectories. While existing technologies have achieved certain success, LLMs generally rely on a single modality for input and lack full utilization of map information such as road structure and lane topology. Furthermore, implicit modeling approaches cannot clearly express the causal relationships between the behaviors of various traffic participants. Consequently, the LLMs obtained in existing technologies cannot accurately predict vehicle trajectories and remain difficult to meet the requirements for high-quality and interpretable vehicle trajectory prediction. Summary of the Invention
[0003] The purpose of this invention is to solve the problem of poor accuracy of vehicle trajectory prediction results due to the fact that the large language model in the existing methods only relies on a single modal input and does not adequately model the interactive behaviors between multiple traffic participants. An interactive vehicle trajectory prediction method based on a multimodal large language model is proposed.
[0004] The technical solution adopted by the present invention to solve the above technical problems is: an interactive vehicle trajectory prediction method based on a multimodal large language model, the method specifically comprising the following steps: Step 1: Use the map of the study area as the input of the image encoder, and then pass the output of the image encoder through a multi-layer perceptron to obtain the map semantic features; The current state of each moving object and the interaction information between each moving object are used as the input of the text encoder to obtain the embedding features of the current state and the embedding features of the interaction information; Step 2: Fuse the embedded features of the current state and the embedded features of the interactive information to obtain motion features , and then fuse the motion features with the map semantic features to obtain the final scene semantic representation ; Step 3: Represent the scene semantics As the input of the large language model LLM, the large language model LLM outputs the next behavior description of each moving object; According to the output of the large language model LLM, the next state information of each moving object and the next interaction information between each moving object are obtained, and then the newly obtained state information and interaction information are used to return to execute step one.
[0005] Furthermore, the image encoder is a visual Transformer model.
[0006] Furthermore, the map of the study area is used as the input of the image encoder, and the output of the image encoder is passed through a multi-layer perceptron to obtain the map semantic features; the specific process is:
[0007] in, represents the input map, represents the visual Transformer model, represents a multilayer perceptron, Represents map semantic features.
[0008] Furthermore, the current state of each moving object includes the type of the moving object, the two-dimensional position coordinates of the moving object, the direction in which the moving object is traveling, and the speed of the moving object; The types of moving objects include motor vehicles, non-motor vehicles and pedestrians.
[0009] Furthermore, the current state of each moving object and the interaction information between each moving object are used as inputs of the text encoder to obtain the embedding features of the current state and the embedding features of the interaction information; the specific process is:
[0010] in, Indicates the current status information of each moving object, Represents the interaction information between various moving objects, Embedding features representing the current state, Embedding features representing interactive information, Represents a text encoder.
[0011] Furthermore, the embedding features of the current state and the embedding features of the interactive information are fused to obtain the motion features ; The specific process is:
[0012] in, Represents the multi-head self-attention mechanism.
[0013] Furthermore, the motion features are fused with the map semantic features to obtain the final scene semantic representation. ; The specific process is:
[0014] Furthermore, the parameter update method during the large language model LLM training process is:
[0015] in, represents the updated model parameters, represents the pre-trained weight matrix, , represents a real number, Represents the weight matrix Dimensions, represents the amount of model parameter adjustment, ,matrix ,matrix , ; Through the training matrix and To update the parameters of the large language model LLM.
[0016] Furthermore, the optimization goal during the large language model LLM training process is for:
[0017] in, represents the cross entropy loss function, Represents the behavior of the large language model LLM output, Indicates real behavior.
[0018] Furthermore, the method obtains a predicted trajectory of each moving object based on the behavior of each moving object at each step output by the large language model LLM.
[0019] The beneficial effects of the present invention are: The present invention uses an image encoder to encode the map and a text encoder to extract features from the current state of each moving object and the interaction information between them. These features are then fused in a shared embedding space. This fused multimodal scene semantic representation is then fed into a large multimodal language model, completing the generative mapping from scene to behavior. The output is a description of each vehicle's behavior, which in turn generates trajectory predictions based on the vehicle's behavior.
[0020] This invention supports the efficient integration of map and text information, enhancing environmental perception. Furthermore, the large language model can capture the complex interactions between multiple vehicles, significantly improving the accuracy of vehicle trajectory prediction compared to existing methods. Furthermore, the behavior based on which vehicle trajectory prediction is performed is interpretable and logical. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 This is a flow chart of an interactive vehicle trajectory prediction method based on a multimodal large language model of the present invention. DETAILED DESCRIPTION
[0022] Specific implementation method 1: Combination Figure 1 This embodiment describes an interactive vehicle trajectory prediction method based on a multimodal large language model, and the method specifically includes the following steps: Step 1: A map containing the static environment of the study area, including road structure, lane lines, obstacles, etc., is used as the input of the image encoder. The output of the image encoder is then passed through a multi-layer perceptron to obtain the map semantic features. The current state of each moving object and the interaction information between each moving object are used as the input of the text encoder to obtain the embedding features of the current state and the embedding features of the interaction information; The interactive information is the interactive relationship between vehicles expressed in natural language, such as following, overtaking, avoiding, etc. Step 2: Fuse the embedded features of the current state and the embedded features of the interactive information to obtain motion features , and then fuse the motion features with the map semantic features to obtain the final scene semantic representation ; Step 3: Represent the scene semantics As the input of the large language model LLM, the large language model LLM outputs the next behavior description of each moving object; According to the output of the large language model LLM, the next state information of each moving object and the next interaction information between each moving object are obtained, and then the newly obtained state information and interaction information are used to return to execute step one.
[0023] A driving scenario can be formally defined as:
[0024] in, Represents the map area information in the scene, consisting of multiple lane segments; Represents the collection of intelligent agents participating in the scene, including vehicles, bicycles, pedestrians, etc.
[0025] At the initial time step , a series of interactive behaviors will occur between the agents, defined as:
[0026] in Representing an agent and Interactions between vehicles, such as avoiding, overtaking, and following.
[0027] Each agent The behavior can be expressed as ,in, It is the set of all generative behaviors, and the behaviors themselves are represented by text or semantics that are logically interpretable.
[0028] The above problem can be formalized as and interactive information Let’s start by learning a mapping function:
[0029] That is, building a multimodal driven generative model , used to extract interaction features and generate agent behaviors with interaction attributes Therefore, based on the above definition, an interactive vehicle behavior generation model based on a multimodal large language model is proposed in the present invention. The core idea is to realize the transformation from the initial scene to the behavior of the moving object by fusing multimodal input information.
[0030] Specific embodiment 2: This embodiment differs from specific embodiment 1 in that the image encoder is a visual Transformer model (ViT).
[0031] Other steps and parameters are the same as those in the first embodiment.
[0032] Specific embodiment three: This embodiment differs from specific embodiments one or two in that the map of the study area is used as the input of the image encoder, and the output of the image encoder is passed through a multi-layer perceptron to obtain map semantic features. The specific process is as follows:
[0033] in, represents the input map, represents the visual Transformer model, represents a multilayer perceptron, Represents map semantic features.
[0034] Other steps and parameters are the same as those in the first or second embodiment.
[0035] This implementation uses a visual encoder based on Visual Transformer (ViT) to extract map features, and then aligns the map features through a multi-layer perceptron projector to obtain map semantic features.
[0036] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that the current state of each moving object includes the type of the moving object, the two-dimensional position coordinates of the moving object, the direction in which the moving object is traveling, and the speed of the moving object; The types of moving objects include motor vehicles, non-motor vehicles and pedestrians.
[0037] The other steps and parameters are the same as those in the first to third embodiments.
[0038] Specific embodiment 5: This embodiment differs from any one of specific embodiments 1 to 4 in that the current state of each moving object and the interaction information between each moving object are used as inputs of the text encoder to obtain the embedding features of the current state and the embedding features of the interaction information; the specific process is as follows:
[0039] in, Indicates the current status information of each moving object, Represents the interaction information between various moving objects, Embedding features representing the current state, Embedding features representing interactive information, Represents a text encoder.
[0040] The other steps and parameters are the same as those in the first to fourth embodiments.
[0041] Specific embodiment 6: This embodiment differs from the specific embodiments 1 to 5 in that the embedded features of the current state and the embedded features of the interactive information are fused to obtain the motion features. ; The specific process is:
[0042] in, Represents the multi-head self-attention mechanism.
[0043] The other steps and parameters are the same as those in the first to fifth embodiments.
[0044] In this embodiment, the current state of the embedded feature Mapped to Q vector, embedding features of interactive information Mapped to the key vector K, and then embedding the interactive information into features Map it to a value vector V, and then perform multi-head self-attention calculation based on the Q vector, key vector K and value vector V obtained by the mapping.
[0045] Specific embodiment 7: This embodiment differs from specific embodiments 1 to 6 in that the motion features are fused with map semantic features to obtain the final scene semantic representation. ; The specific process is:
[0046] The other steps and parameters are the same as those in the first to sixth embodiments.
[0047] In this embodiment, the motion feature Mapped into Q vector, the map semantic features Mapped to key vector K, and then map semantic features Map it to a value vector V, and then perform multi-head self-attention calculation based on the Q vector, key vector K and value vector V obtained by the mapping.
[0048] The final scene semantic representation obtained in this embodiment can enhance the model's comprehensive understanding of multimodal information, thereby further improving the accuracy of vehicle trajectory prediction.
[0049] Specific embodiment eight: This embodiment differs from any one of specific embodiments one to seven in that the parameter updating method during the large language model LLM training process is as follows:
[0050] in, represents the updated model parameters, represents the pre-trained weight matrix, , represents a real number, Represents the weight matrix Dimensions, represents the amount of model parameter adjustment, ,matrix ,matrix ,
[0051] Through the training matrix and To update the parameters of the large language model LLM.
[0052] The other steps and parameters are the same as those in the first to seventh embodiments.
[0053] To reduce the number of parameters to be trained during the training phase, this implementation introduces LoRA (Low-Rank Adaptation) to perform low-rank adaptation on the attention layer and the MLP layer. This significantly reduces the number of parameters to be trained while maintaining the model's expressive power.
[0054] Specific embodiment 9: This embodiment differs from specific embodiments 1 to 8 in that the optimization goal in the large language model LLM training process is for:
[0055] in, represents the cross entropy loss function, Represents the behavior of the large language model LLM output, Indicates real behavior.
[0056] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.
[0057] The behavior generation process of the large language model LLM is an autoregressive sequence modeling process, and the final output is the joint probability of the behavior sequence for:
[0058] The method of the present invention achieves high-quality and explainable behavior generation under multimodal drive by finely modeling the interaction logic and scene semantics between moving objects, providing semantic support for subsequent trajectory generation and scene simulation.
[0059] Specific embodiment ten: This embodiment differs from any one of specific embodiments one to nine in that the method obtains the predicted trajectory of each moving object based on the behavior of each moving object at each step output by the large language model LLM.
[0060] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.
[0061] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.
Claims
1. An interactive vehicle trajectory prediction method based on a multimodal large language model, characterized in that: The method specifically comprises the following steps: Step 1: Use the map of the study area as the input of the image encoder, and then pass the output of the image encoder through a multi-layer perceptron to obtain the map semantic features; The current state of each moving object and the interaction information between each moving object are used as the input of the text encoder to obtain the embedding features of the current state and the embedding features of the interaction information; Step 2: Fuse the embedded features of the current state and the embedded features of the interactive information to obtain motion features , and then fuse the motion features with the map semantic features to obtain the final scene semantic representation ; Step 3: Represent the scene semantics As the input of the large language model LLM, the large language model LLM outputs the next behavior description of each moving object; According to the output of the large language model LLM, the next state information of each moving object and the next interaction information between each moving object are obtained, and then the newly obtained state information and interaction information are used to return to execute step one.
2. The interactive vehicle trajectory prediction method based on a multimodal large language model according to claim 1, characterized in that: The image encoder is a visual Transformer model.
3. The interactive vehicle trajectory prediction method based on a multimodal large language model according to claim 2, characterized in that: The map of the study area is used as the input of the image encoder, and the output of the image encoder is passed through a multi-layer perceptron to obtain map semantic features; The specific process is: in, represents the input map, represents the visual Transformer model, represents a multilayer perceptron, Represents map semantic features.
4. The interactive vehicle trajectory prediction method based on a multimodal large language model according to claim 3 is characterized in that: The current state of each moving object includes the type of the moving object, the two-dimensional position coordinates of the moving object, the direction of the moving object, and the speed of the moving object; The types of moving objects include motor vehicles, non-motor vehicles and pedestrians.
5. The interactive vehicle trajectory prediction method based on a multimodal large language model according to claim 4 is characterized in that: The current state of each moving object and the interaction information between each moving object are used as inputs of the text encoder to obtain the embedding features of the current state and the embedding features of the interaction information; The specific process is: in, Indicates the current status information of each moving object, Represents the interaction information between various moving objects, Embedding features representing the current state, Embedding features representing interactive information, Represents a text encoder.
6. The interactive vehicle trajectory prediction method based on a multimodal large language model according to claim 5, characterized in that: The embedding features of the current state and the embedding features of the interactive information are fused to obtain the motion features ; The specific process is: in, Represents the multi-head self-attention mechanism.
7. The interactive vehicle trajectory prediction method based on a multimodal large language model according to claim 6, characterized in that: The motion features are fused with the map semantic features to obtain the final scene semantic representation ; The specific process is: 。 8. The interactive vehicle trajectory prediction method based on a multimodal large language model according to claim 7, characterized in that: The parameter update method during the large language model LLM training process is: in, represents the updated model parameters, represents the pre-trained weight matrix, , represents a real number, Represents the weight matrix Dimensions, represents the amount of model parameter adjustment, ,matrix ,matrix , ; Through the training matrix and To update the parameters of the large language model LLM.
9. The interactive vehicle trajectory prediction method based on a multimodal large language model according to claim 8, characterized in that: The optimization goal during the large language model LLM training process for: in, represents the cross entropy loss function, Represents the behavior of the large language model LLM output, Indicates real behavior.
10. The interactive vehicle trajectory prediction method based on a multimodal large language model according to claim 9, characterized in that: The method obtains a predicted trajectory of each moving object based on the behavior of each moving object at each step output by a large language model (LLM).