Vehicle motion trail prediction method and system based on big language model semantic guidance

Abstract

The invention provides a vehicle motion track prediction method and system based on large language model semantic guidance, and belongs to the field of automatic driving vehicle motion prediction. Comprising the following steps: collecting own vehicle data, roadside equipment data, participating vehicle data and map data, and carrying out multi-view space-time alignment and vectorization feature extraction to obtain position features, motion features and map features of each vehicle under the same coordinate system; inputting the features into GNN to generate a perception vector; inputting the perception vector into a lightweight neural network to obtain a motion prediction vector; generating a high-dimensional motion prediction vector of the predicted vehicle by using the LLM in combination with the position features of the region near the predicted vehicle; determining a final prediction vector according to the high-dimensional motion prediction vector and the motion prediction vector, and decoding the final prediction vector and the perception vector to obtain a final prediction track of the predicted vehicle; and after determining that the lightweight neural network has the LLM reasoning capability by using the loss adjustment parameter, stopping calling the LLM to predict the trajectory.

Description

Technical Field

[0001] This invention belongs to the field of autonomous vehicle motion prediction, and particularly relates to a method and system for predicting vehicle motion trajectory based on semantic guidance of a large language model. Background Technology

[0002] In recent years, vehicle-road cooperation has become an important development trend in the field of autonomous driving. Its advantage lies in the fact that by integrating the perception information of infrastructure and other vehicles, it can effectively overcome the limitations of single-vehicle perception, especially in complex scenarios such as obscured intersections.

[0003] Currently, in the field of autonomous vehicle motion prediction technology, Graph Neural Networks (GNNs) have gradually become the mainstream method for modeling complex interactions between vehicles due to the inherent graph structure of traffic scenarios. GNNs, through message passing mechanisms between nodes, can efficiently aggregate spatiotemporal features within a local neighborhood, thereby improving the reliability of trajectory prediction results. However, GNNs essentially rely on data-driven feature associations to build interaction models, lacking explicit guidance from high-level semantic patterns. Specifically, GNNs aggregate features through spatial proximity but cannot encode semantic interaction types (e.g., "following" or "lane changing"). This lack of semantic patterns ultimately leads to biased trajectory predictions, thus reducing the accuracy of long-term trajectory predictions. Summary of the Invention

[0004] In view of this, the present invention provides a vehicle motion trajectory prediction method and system based on semantic guidance of a large language model. It uses LLM to infer a logical trajectory prediction vector to guide trajectory generation, thereby meeting the accuracy and real-time requirements of autonomous vehicle motion trajectory prediction.

[0005] The technical solution adopted by the embodiments of the present invention to solve its technical problem is as follows:

[0006] A vehicle trajectory prediction method based on semantic guidance of a large language model includes:

[0007] Data from vehicles, roadside equipment, participating vehicles, and maps is collected and subjected to multi-view spatiotemporal alignment and vectorized feature extraction to obtain the position features, motion features, and map features of each vehicle in the same coordinate system. The vehicle data includes the vehicle's historical trajectory sequence, vehicle type, and vehicle size information. The participating vehicle data includes historical trajectory, type, and size information. The roadside equipment data consists of the vehicle's historical trajectory sequence, vehicle type, and vehicle size information from the roadside perspective. The map includes lane line vectors, turning information, and whether it is an intersection.

[0008] Using participating vehicles as nodes, the position features, motion features, and map features of each vehicle are input into a graph neural network (GNN) for feature extraction to generate a perception vector.

[0009] The perception vector is input into a lightweight neural network for feature extraction to obtain the motion prediction vector;

[0010] By using the Large Language Model (LLM) and combining the location features of the area near the predicted vehicle, a high-dimensional motion prediction vector with high-level semantics is generated for the predicted vehicle.

[0011] The final prediction vector is determined based on the high-dimensional motion prediction vector and the motion prediction vector. The final predicted trajectory of the vehicle is obtained by decoding the final prediction vector and the perception vector. At the same time, the decoder parameters are adjusted using trajectory loss and motion prediction loss. After confirming that the lightweight neural network has the inference ability of LLM based on the motion prediction loss, the LLM is stopped from being called for trajectory prediction.

[0012] Preferably, the multi-view spatiotemporal alignment and vectorized feature extraction includes:

[0013] Step S11: Acquire roadside equipment data, participating vehicle data, and map data, and align the multi-source data using a unified timestamp;

[0014] Step S12: Assign a unique graph node index to all participating vehicles appearing within the historical time window. N is the number of participating vehicles;

[0015] Step S13: Perform multi-view spatiotemporal alignment of multi-source data:

[0016] The position of the vehicle at the last time step of the historical window is taken as the origin of the new local coordinate system for the entire scene;

[0017] Using the vehicle's heading angle at the same moment Define the X and Y axes of the new coordinate system and calculate the rotation matrix. : ;

[0018] Using the rotation matrix R, all participating vehicles Rotate and translate the absolute coordinates at all time steps to the new local coordinate system;

[0019] Step S14: Extract the position features, motion features, and map features of each participating vehicle in the new local coordinate system. The position features are the trajectories in the new local coordinate system, the motion features are differential coordinates, and the map features include lane line vectors, steering information, and intersection information in the new local coordinate system.

[0020] Preferably, the step of using participating vehicles as nodes and inputting the position features, motion features, and map features of each vehicle into a graph neural network (GNN) for feature extraction includes:

[0021] Step S21: Input the position features, motion features, and map features of each vehicle into the GNN for encoding to obtain trajectory motion features. Spatiotemporal characteristics of trajectory and lane segment spatial characteristics :

[0022] Based on motion characteristics Extracting trajectory motion features : ;

[0023] In the formula, Indicates vehicle The trajectory position, where T is the window length. This represents the heading vector at time step t. This indicates the embedding at a learnable location at time step t. It is a multi-head self-attention module. Represents a multilayer perceptron;

[0024] Based on location features Extracting trajectory spatiotemporal features : ;

[0025] Lane segment spatial features extracted from map features : ;

[0026] In the formula, The trajectory in the current frame and its lane section The relative heading vector between them; and These represent the start and end points of a lane segment, respectively.

[0027] Step S22: Use spatiotemporal coding to capture the spatial-temporal correlation of the trajectories of vehicles i and j at each time step to determine whether the two cross-view trajectories are the same object; use relative spatial coding to describe the spatial relationship between participants or between participants and lanes.

[0028] Represented as: ; ;

[0029] In the formula, Represents spatiotemporal coding. This indicates relative spatial encoding, and `concat` represents the concatenation operation; for relative spatial encoding... When indicating the relationship between participants, and Represented as and When represented as the relationship between a participant and a lane, and Represented as and ;

[0030] Step S23: Construct associated component A, motion fusion subgraph, lane fusion subgraph, and cooperative interaction subgraph for feature fusion: ;

[0031] In the formula, This is an MLP used for binary classification to determine whether there is a correlation between two trajectories; and This indicates the trajectory of the vehicle and the participating vehicles;

[0032] The motion fusion subgraph aggregates cooperative motion feature representations of cross-view associated trajectories by considering explicit associations and temporal-spatial correlations. The feature fusion and update process is as follows: ;

[0033] In the formula, This represents the k+1 update data of the motion fusion vector. It is a feedforward network. This represents the cross-attention mechanism;

[0034] The lane fusion subgraph uses relative spatial encoding to enable proxy motion features to query relevant lane segment interaction features. The feature fusion and update process is as follows: ;

[0035] In the formula, This is the k+1 update data for the lane fusion vector. Learnable tags representing semantic attributes associated with the corresponding lane segment, such as whether it is located at an intersection, turning direction, and traffic control.

[0036] The cooperative interaction subgraph represents the dense interactions between different agents across all perspectives. The feature fusion and update process is as follows: ;

[0037] In the formula, This represents the k+1 update data of the cooperative interaction vector. This represents the learnable interactive attribute features associated between two trajectories;

[0038] Step S24: Concatenate trajectory motion feature vectors Trajectory spatiotemporal feature vector Motion fusion vector Lane fusion vector Collaborative interaction vector The concatenated result is fed into an MLP to generate a perceptual vector. .

[0039] Preferably, the generation of high-dimensional motion prediction vectors and motion prediction vectors with high-level semantics includes:

[0040] Step S31: Obtain the predicted vehicle from any viewpoint. The state of the predicted vehicle Centered on the status, all participating vehicles World coordinates ( , Convert to coordinates in a local coordinate system centered on the predicted vehicle. : ;

[0041] In the formula, the vehicle being predicted The state includes position ( , ) and heading angle ;

[0042] Step S32: Divide the area around the predicted vehicle into W key sectors, centered on the predicted vehicle. Select the key target closest to the predicted vehicle within each key sector and record the distance between the key target and the predicted vehicle.

[0043] Step S33: Preset high-rise driving targets, and generate text prompts based on key targets and distances. ;

[0044] Step S34, add text prompts Inputting LLM for inference yields the prediction vector. Input mapping network Mapped to high-dimensional motion prediction vectors : ; .

[0045] Preferably, the step of determining the final prediction vector based on the high-dimensional motion prediction vector and the motion prediction vector, and obtaining the final predicted trajectory of the predicted vehicle by decoding the final prediction vector, and adjusting the decoder parameters using trajectory loss and motion prediction loss includes:

[0046] Maintain a loss value for storing the motion prediction values ​​of the most recent Z steps. Fixed-size queue ,when When filled, the standard deviation is calculated. To determine whether a lightweight neural network possesses the inference capability of an LLM, thereby identifying the flag bits. : ; ;

[0047] In the formula, The motion prediction loss value for step z; For queue The mean loss; Not less than the stability threshold At that time, the flag position Setting it to True indicates that the lightweight neural network does not yet possess the inference capabilities of an LLM; when Less than the stability threshold At that time, the flag position Setting it to False indicates that the lightweight neural network already possesses the inference capabilities of LLM;

[0048] Combined with flag bits Based on high-dimensional motion prediction vectors Motion prediction vector Determine the final prediction vector and the final predicted trajectory : ; ;

[0049] Training process loss function Including trajectory loss and motion prediction loss : ; ;

[0050] In the formula, trajectory loss Used to compare the final predicted trajectory With the actual trajectory The difference between them is due to the regression loss. and classification loss composition, For hyperparameter weights; It's a hyperparameter. It is an indicator function;

[0051] The loss function is adjusted to meet the requirements. < Then, stop calling LLM.

[0052] A vehicle trajectory prediction system based on semantic guidance from a large language model includes:

[0053] The data acquisition module is used to collect data from vehicles, roadside equipment, participating vehicles, and map data, and to perform multi-view spatiotemporal alignment and vectorized feature extraction to obtain the position features, motion features, and map features of each vehicle in the same coordinate system. The vehicle data includes the vehicle's historical trajectory sequence, vehicle type, and vehicle size information. The participating vehicle data includes historical trajectory, type, and size information. The roadside equipment data is the vehicle's historical trajectory sequence, vehicle type, and vehicle size information from the roadside perspective. The map includes lane line vectors, turning information, and whether it is an intersection.

[0054] The GNN network feature extraction module is used to extract features by taking the participating vehicles as nodes and inputting the position features, motion features, and map features of each vehicle into the graph neural network (GNN) to generate a perception vector.

[0055] The perception vector is input into a lightweight neural network for feature extraction to obtain the motion prediction vector;

[0056] The LLM prediction module is used to generate a high-dimensional motion prediction vector of the predicted vehicle with high-level semantics by using the large language model LLM and combining the location features of the area near the predicted vehicle.

[0057] The trajectory prediction module is used to determine the final prediction vector based on the high-dimensional motion prediction vector and the motion prediction vector, and to obtain the final predicted trajectory of the vehicle being predicted by decoding the final prediction vector and the perception vector. At the same time, the decoder parameters are adjusted using trajectory loss and motion prediction loss. After confirming whether the lightweight neural network has the inference ability of LLM based on the motion prediction loss, the LLM is stopped from being called for trajectory prediction.

[0058] As described above, the vehicle trajectory prediction method and system based on semantic guidance of a large language model provided in this invention first performs multi-view spatiotemporal alignment and vectorized feature extraction on the original trajectory data of the vehicle, roadside equipment, and other vehicles to facilitate efficient loading and training of the GNN model. Then, the preprocessed data is simultaneously fed into both the GNN and LLM. The GNN is responsible for encoding nodes and edges and extracting more detailed features about the relationships between them, generating perception vectors to provide the underlying perception foundation for trajectory prediction. The LLM is responsible for reasoning about the traffic situation around the predicted vehicle, generating trajectory prediction vectors with high-level semantics, providing high-level semantic guidance for trajectory prediction, and designing a lightweight neural network to learn the inference ability of the LLM to achieve fast inference. Finally, in the trajectory decoding stage, the perception vector generated by the GNN and the trajectory prediction vector inferred by the LLM are concatenated and then decoded to obtain the predicted trajectory. This invention utilizes the deep understanding of traffic scenarios and logical reasoning capabilities of the LLM to compensate for the trajectory deviations generated by the GNN's trajectory prediction, improving the accuracy of the predicted vehicle trajectory. Attached Figure Description

[0059] Figure 1 This is a flowchart of the vehicle trajectory prediction method based on semantic guidance of a large language model according to the present invention.

[0060] Figure 2 This is an example diagram of a complex road traffic environment.

[0061] Figure 3 This is an example diagram of vehicle motion prediction generated by LLM. The solid lines represent the actual trajectory, the dashed lines represent the six predicted trajectories, and a series of square points represent the historical trajectories. Detailed Implementation

[0062] The technical solution and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0063] Addressing the lack of high-level semantics in trajectory prediction by Generative Neural Networks (GNNs), Large Language Models (LLMs) possess inherent advantages in semantic understanding and reasoning. LLMs can perform logical reasoning within textualized scenarios, much like human drivers. For example, LanguageMPC successfully utilizes LLMs as decision-makers; when a vehicle is in motion, the system inputs surrounding environmental information into the LLM, which then directly outputs decision commands such as "change lanes" or "follow the car." These works demonstrate the unique advantages of LLMs in handling long-tail scenarios requiring common-sense reasoning, enabling them to deduce the next course of action much like a human.

[0064] This invention leverages the deep understanding and logical reasoning capabilities of the LLM (Large Language Model) of traffic scenarios to compensate for trajectory biases in GNN (Generative Neural Network) trajectory prediction, providing a vehicle trajectory prediction method based on semantic guidance from a large language model. First, the raw trajectory data of the vehicle, roadside equipment, and other vehicles are spatiotemporally aligned from multiple perspectives and vectorized for feature extraction, facilitating efficient loading and training of the GNN model. Then, the preprocessed data is simultaneously fed into both the GNN and LLM. The GNN is responsible for encoding nodes and edges and extracting more detailed features about their relationships, generating perceptual vectors to provide the underlying perceptual foundation for trajectory prediction. The LLM is responsible for reasoning about the traffic situation surrounding the predicted vehicle, generating trajectory prediction vectors with high-level semantics, providing high-level semantic guidance for trajectory prediction, and a lightweight neural network is designed to learn the LLM's reasoning capabilities for rapid reasoning. Finally, in the trajectory decoding stage, the perceptual vectors generated by the GNN and the trajectory prediction vectors inferred by the LLM are concatenated to decode the predicted trajectory. The following section, in conjunction with the appendix... Figure 1 The flowchart shown further illustrates the concept of this invention:

[0065] This invention uses the real-world public motion trajectory prediction dataset V2X-Traj as an example. The V2X-Traj dataset contains 10,102 scenarios, which are randomly divided into training, validation, and test sets, containing 6,062, 2,020, and 2,020 scenarios respectively. Each scenario consists of three independent trajectories from two autonomous vehicles and one infrastructure-side perception device, achieving collaborative perception between V2I (vehicle-to-infrastructure) and V2V (vehicle-to-vehicle). Each scenario lasts for 8 seconds with a sampling rate of 10 Hz. Four seconds of observation data from each viewpoint are used to predict future motion within the next four seconds.

[0066] During the LLM invocation, this invention uses a locally deployed large language model, llama3.1:8B, as an example. Based on the summary and induction of information about the scene surrounding the vehicle being predicted, a motion prediction vector containing risk, acceleration, and steering suggestions is generated. The lightweight neural network learns the predictive capabilities of the LLM, with a loss function for supervision. The system continuously monitors the standard deviation of the loss value over the most recent 6000 training steps. When this value falls below a threshold... When the value is 0.001, it indicates that the lightweight neural network has achieved the predictive capability of LLM, so the LLM is stopped and the lightweight neural network is used for motion prediction.

[0067] This invention uses the AdamW optimizer as an example, with an initial learning rate set to 1e-3 and a cosine annealing strategy for decay. The weight decay is 1e-4, the batch size is 2, and the total training time is 64 epochs.

[0068] Step 1: Data Preprocessing

[0069] Data is collected from vehicles, roadside equipment, participating vehicles, and map data, and multi-view spatiotemporal alignment and vectorized feature extraction are performed to obtain the position features, motion features, and map features of each vehicle in the same coordinate system; specific implementation includes:

[0070] Step S11: Acquire roadside equipment data, participating vehicle data, and map data, aligning the multi-source data using a unified timestamp (e.g., selecting the first 40 historical time steps); vehicle data includes the vehicle's historical trajectory sequence, vehicle type, and vehicle size information; participating vehicle data includes historical trajectory, type, and size information; roadside equipment data is the vehicle's historical trajectory sequence from the roadside perspective, vehicle type, and vehicle size information; the map includes lane line vectors, turning information, and whether it is an intersection information in the scene.

[0071] Step S12: Participants are screened. To ensure the stability of the graph structure, a unique graph node index is assigned to all participating vehicles that appear within the historical time window. N is the number of participating vehicles;

[0072] Step S13: Perform multi-view spatiotemporal alignment of multi-source data:

[0073] The position of the vehicle at the last time step of the historical window is taken as the origin of the new local coordinate system for the entire scene;

[0074] Using the vehicle's heading angle at the same moment Define the X and Y axes of the new coordinate system and calculate the rotation matrix. : (1)

[0075] Using the rotation matrix R, all participating vehicles Rotate and translate the absolute coordinates at all time steps to transform them all into this new local coordinate system centered on the vehicle and oriented towards normalization;

[0076] Step S14: Extract the position features, motion features, and map features of each participating vehicle in the new local coordinate system. The position features are the trajectories in the new local coordinate system, which are used to calculate spatial relationships in the future. The motion features are differential coordinates, which convert the historical trajectories into differential coordinates, i.e., the displacement vectors of each step, so that the features are more focused on the motion patterns. The map features include lane line vectors, steering information, and intersection information (e.g., extracting information within a 50-meter radius in the scene) in the new local coordinate system, and align them to the local coordinate system using the same rotation matrix R and translation.

[0077] Step 2: Establish a motion feature extraction system based on a graph neural network (GNN). Using participating vehicles as nodes, input the position features, motion features, and map features of each vehicle into the GNN for feature extraction, generating a perception vector. Specific implementation includes:

[0078] In step S21, during the GNN feature extraction stage, in order to convert the low-dimensional "raw numerical features" generated in the preprocessing stage into high-dimensional "embedding vectors" that can be understood by the neural network, the position features, motion features, and map features of each vehicle are input into the GNN for encoding to obtain trajectory motion features. Spatiotemporal characteristics of trajectory and lane segment spatial characteristics :

[0079] Based on motion characteristics Extracting trajectory motion features The formula is as follows: (2)

[0080] In the formula, Indicates vehicle The trajectory position, where T is the window length. This represents the heading vector at time step t. This indicates the embedding at a learnable location at time step t. It is a multi-head self-attention module. Represents a multilayer perceptron;

[0081] Based on location features Extracting trajectory spatiotemporal features Spatial-temporal features are encoded by incorporating the temporal dependencies between the self-center normalized coordinates of the trajectory. In the attention module, missing frames are masked, as shown in the following formula: (3)

[0082] Lane segment spatial features extracted from map features To enhance the representation of trajectories for future intent reasoning, features from a vector map structure are incorporated. Specifically, the spatial features of lane segments are represented by coordinates relative to the center point (from the start to the end of each lane segment) and further encoded as nodes for feature aggregation, as shown in the following formula: (4)

[0083] In the formula, The trajectory in the current frame and its lane section The relative heading vector between them; and These represent the start and end points of a lane segment, respectively.

[0084] Step S22: Encode the graph edges from two perspectives: spatiotemporal encoding. and relative spatial encoding :

[0085] First is spatiotemporal coding To determine whether two cross-view trajectories belong to the same object, this invention uses an attention module to aggregate the spatiotemporal codes between each pair of cross-view trajectories. The spatiotemporal codes capture the spatial-temporal correlation between the two trajectories at each time step, which facilitates motion feature fusion. Secondly, relative spatial coding is used. This describes the spatial relationship between participants or between a participant and the lane. It is calculated by plotting the relative position vectors of the participants at the current moment and encoding them using an MLP. The formula is as follows: (5) (6)

[0086] In the formula, Represents spatiotemporal coding. Relative spatial encoding is represented by `concat`, which concatenates the spatiotemporal feature vectors of the trajectory from different perspectives. When indicating the relationship between participants, and Represented as and When represented as the relationship between a participant and a lane, and Represented as and ;

[0087] Step S23: To effectively fuse the features of the encoded nodes and edges to obtain a more accurate and robust scene understanding, this invention constructs an associated component A, a motion fusion subgraph, a lane fusion subgraph, and a cooperative interaction subgraph for feature fusion.

[0088] Related components It is used to solve the core problem of distinguishing trajectories from different sources in multi-view data fusion of vehicle-road cooperative systems, and provides clear guidance for the fusion of motion and interaction features. The formula is as follows: (7)

[0089] In the formula, This is an MLP used for binary classification to determine whether there is a correlation between two trajectories; and This indicates the trajectory of the vehicle and the participating vehicles;

[0090] The motion fusion subgraph aggregates cooperative motion feature representations of cross-view associated trajectories by considering explicit associations and temporal-spatial correlations, aiming to comprehensively represent the historical motion of the predicted vehicle. The feature fusion and update process is as follows: (8)

[0091] In the formula, This represents the k+1 update data of the motion fusion vector. It is a feedforward network. This represents the cross-attention mechanism;

[0092] The lane fusion submap uses relative spatial encoding to enable proxy motion features to query relevant lane segment interaction features, combining map information for cooperative motion prediction. The feature fusion and update process is as follows: (9)

[0093] In the formula, This is the k+1 update data for the lane fusion vector. Learnable tags representing semantic attributes associated with the corresponding lane segment, such as whether it is located at an intersection, turning direction, and traffic control.

[0094] The cooperative interaction subgraph represents the dense interaction between different agents across all perspectives. It combines explicit associations and relative spatial correlations to model the dense interaction features in cooperative scenarios. The feature fusion and update process is as follows: (10)

[0095] In the formula, This represents the k+1 update data of the cooperative interaction vector. This represents the learnable interactive attribute features associated between two trajectories; The relative motion geometric features between two different vehicle trajectories, namely the relative heading angle, are represented by the following calculation method: First, calculate the difference in heading angles between the two vehicles at the current moment. Then, calculate the sine and cosine values ​​of the difference to obtain a two-dimensional vector. Finally, input the two-dimensional vector into the MLP to map it into a high-dimensional feature vector.

[0096] Step S24: Concatenate trajectory motion feature vectors Trajectory spatiotemporal feature vector Motion fusion vector Lane fusion vector Collaborative interaction vector The concatenated result is fed into an MLP to generate a perceptual vector. .

[0097] Step 3: Input the perception vector into a lightweight neural network for feature extraction to obtain the motion prediction vector:

[0098] The specific processing flow in a lightweight neural network includes the following four operations: First, the input perceptual vector... After a linear transformation layer, non-linearity is introduced through the ReLU activation function; then, a layer normalization module is applied; finally, a second linear mapping layer projects the normalized internal features onto the final 256-dimensional vector, outputting the motion prediction vector. .

[0099] Step 4: Establish motion feature extraction based on LLM. Using the Large Language Model (LLM), combined with the location features of the area near the vehicle being predicted, a high-dimensional motion prediction vector with high-level semantics is generated for the vehicle being predicted.

[0100] Step S31: Obtain the predicted vehicle from any viewpoint. The state of the predicted vehicle Centered on the status, all participating vehicles World coordinates ( , Transform the coordinates into a local coordinate system centered on the predicted vehicle. : (11)

[0101] In the formula, the vehicle being predicted The state includes position ( , ) and heading angle Any perspective can refer to data from the vehicle's cameras, LiDAR, and roadside equipment (the raw trajectory information is jointly provided by LiDAR and cameras, with LiDAR providing geometric information and cameras providing semantic information).

[0102] Step S32: Divide the area around the predicted vehicle into W key sectors centered on the predicted vehicle (for example, divide the 360-degree area around the predicted vehicle into 8 key sectors, including front, left front, right front, left, right, rear, left rear, and right rear). Select the key target closest to the predicted vehicle within each key sector (the straight-line distance between the center coordinates of the predicted vehicle and the center coordinates of the target vehicle), and record the distance between the key target and the predicted vehicle. This method can reduce scene complexity and effectively preserve key information.

[0103] Step S33: Preset high-level driving objectives (e.g., "maintain current lane and drive straight at a steady speed"), and combine them with key objectives and distances to convert them into a concise and structured text prompt. , as input to LLM;

[0104] Step S34, add text prompts Inputting LLM data for inference yields prediction vectors in JSON format. Input mapping network Mapped to high-dimensional motion prediction vectors : (12) (13)

[0105] Step 5: Motion prediction trajectory decoding. Based on the high-dimensional motion prediction vector and the motion prediction vector, the final prediction vector is determined, and the final predicted trajectory of the vehicle is obtained by decoding the final prediction vector. Simultaneously, the decoder parameters are adjusted using trajectory loss and motion prediction loss. After confirming that the lightweight neural network has the inference capability of LLM based on the motion prediction loss, the LLM invocation for trajectory prediction is stopped. Specifically:

[0106] Maintain a loss value for storing the motion prediction values ​​of the most recent Z steps. Fixed-size queue ,when When filled, the standard deviation is calculated. To determine whether a lightweight neural network possesses the inference capability of an LLM, thereby identifying the flag bits. : (14) (15)

[0107] In the formula, The motion prediction loss value for step z; For queue The mean loss; Not less than the stability threshold At that time, the flag position Setting it to True indicates that the lightweight neural network does not yet possess the inference capabilities of an LLM; when Less than the stability threshold At that time, the flag position Setting it to False indicates that the lightweight neural network already possesses the inference capabilities of LLM;

[0108] Combined with flag bits Based on high-dimensional motion prediction vectors Motion prediction vector Determine the final prediction vector and the final predicted trajectory : (16) (17)

[0109] Including high-level semantic information can effectively reduce trajectory prediction bias;

[0110] Training process loss function Including trajectory loss and motion prediction loss :

[0111] The first part is trajectory loss. This is a standard supervision signal, which primarily compares the trajectories predicted by the model. The difference between the actual trajectory Y and the regression loss is determined by the regression loss. and classification loss This component is used to optimize the behavior of the entire model. For all elements in the scene... Nodes: (18)

[0112] In the formula, For hyperparameter weights;

[0113] The second part is the prediction of loss. This is a loss term specifically designed for knowledge distillation. It is calculated using LLM-generated loss terms. and lightweight neural network generation The mean squared error (MSE) between the two is used for backpropagation. The role of this loss is to "distill" the inference power of the LLM into a lightweight neural network: (19)

[0114] In the formula, It's a hyperparameter; It is an indicator function, which is invoked when the condition is true (i.e., when the condition is true). The value is 1 when it is True (i.e., LLM is used), otherwise it is 0.

[0115] The loss function is adjusted to meet the requirements. < After that, stop calling LLM and enter the efficient late training stage.

[0116] Furthermore, this invention provides a vehicle trajectory prediction system based on semantic guidance from a large language model, for implementation. Figure 1 The method described above, and the specific data processing methods of each module in the system, refer to the foregoing content. The system includes:

[0117] The data acquisition module is used to collect data from vehicles, roadside equipment, participating vehicles, and map data, and to perform multi-view spatiotemporal alignment and vectorized feature extraction to obtain the position features, motion features, and map features of each vehicle in the same coordinate system. The vehicle data includes the vehicle's historical trajectory sequence, vehicle type, and vehicle size information. The participating vehicle data includes historical trajectory, type, and size information. The roadside equipment data is the vehicle's historical trajectory sequence, vehicle type, and vehicle size information from the roadside perspective. The map includes lane line vectors, turning information, and whether it is an intersection.

[0118] The GNN network feature extraction module is used to extract features by taking the participating vehicles as nodes and inputting the position features, motion features, and map features of each vehicle into the graph neural network (GNN) to generate a perception vector.

[0119] The perception vector is input into a lightweight neural network for feature extraction to obtain the motion prediction vector;

[0120] The LLM prediction module is used to generate a high-dimensional motion prediction vector of the predicted vehicle with high-level semantics by using the large language model LLM and combining the location features of the area near the predicted vehicle.

[0121] The trajectory prediction module is used to determine the final prediction vector based on the high-dimensional motion prediction vector and the motion prediction vector, and to obtain the final predicted trajectory of the vehicle being predicted by decoding the final prediction vector and the perception vector. At the same time, the decoder parameters are adjusted using trajectory loss and motion prediction loss. After confirming whether the lightweight neural network has the inference ability of LLM based on the motion prediction loss, the LLM is stopped from being called for trajectory prediction.

[0122] Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects.

[0123] By combining the powerful perception capabilities of GNNs with the logical reasoning capabilities of LLMs, the predicted trajectories become more accurate.

[0124] LLM-based feature extraction can effectively extract key information from a scene and transform it into structured text prompts, thereby enabling LLM to perform effective reasoning.

[0125] The knowledge distillation mechanism of this invention enables lightweight neural networks to perform fast motion reasoning, meeting the real-time requirements of autonomous driving systems.

[0126] The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that implementing all or part of the above-described embodiments and making equivalent changes in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A vehicle motion trajectory prediction method based on large language model semantic guidance, characterized in that, Comprise: Collecting ego vehicle data, roadside device data, participating vehicle data, map data, and performing multi-view spatio-temporal alignment and vectorized feature extraction to obtain position features, motion features, and map features of each vehicle in the same coordinate system; the ego vehicle data includes ego vehicle historical trajectory sequence, vehicle type and vehicle size information; the participating vehicle data includes historical trajectory, type and size information; the roadside device data is the vehicle historical trajectory sequence, vehicle type and vehicle size information in the roadside view; the map includes lane line vectors, turning information, and intersection information in the scene; Taking the participating vehicle as the node, inputting the position features, motion features, and map features of each vehicle into the graph neural network GNN for feature extraction to generate a perception vector; Inputting the perception vector into a lightweight neural network for feature extraction to obtain a motion prediction vector; Using a large language model LLM to generate a high-dimensional motion prediction vector with high-level semantics for the predicted vehicle in combination with the position features of the surrounding area of the predicted vehicle; According to the high-dimensional motion prediction vector and the motion prediction vector, a final prediction vector is determined, and the final prediction trajectory of the predicted vehicle is obtained by decoding the final prediction vector and the perception vector; at the same time, the trajectory loss and the motion prediction loss are used to adjust the decoder parameters, and after confirming that the lightweight neural network has the reasoning ability of the LLM according to the motion prediction loss, the LLM is stopped for trajectory prediction.

2. The vehicle motion trajectory prediction method based on large language model semantic guidance according to claim 1, wherein, The multi-view spatio-temporal alignment and vectorized feature extraction comprises: Step S11, acquiring roadside device data, participating vehicle data, and map data, and aligning multi-source data through a unified timestamp; Step S12, assigning a unique graph node index to all participating vehicles appearing within the historical time window N is the number of participating vehicles; Step S13, performing multi-view spatio-temporal alignment of multi-source data: Taking the position of the ego vehicle at the last time step in the historical window as the origin of the new local coordinate system of the entire scene; Using the orientation angle of the ego vehicle at the same time Defining the X, Y axes of the new coordinate system, calculating the rotation matrix : ; All participating vehicles are rotated by a rotation matrix R The absolute coordinates at all time steps are rotated and translated into a new local coordinate system; Step S14, extracting the position features, motion features, and map features of each participating vehicle in the new local coordinate system, the position features being the trajectory in the new local coordinate system, the motion features being the differential coordinates, and the map features including the lane line vectors, turning information, and intersection information in the new local coordinate system.

3. The vehicle motion trajectory prediction method based on large language model semantic guidance according to claim 2, wherein, The inputting of the position features, motion features, and map features of each vehicle into the graph neural network GNN for feature extraction comprises: Step S21, input the position features, motion features, and map features of each vehicle into the GNN for encoding to obtain trajectory motion features , trajectory spatiotemporal features , and lane segment spatial features : Based on motion features Extracting trajectory motion features : ; wherein denotes the trajectory position of a vehicle T is the window length, denotes the heading vector at time step t, denotes the learnable position embedding at time step t, is a multi-head self-attention module, denotes a multi-layer perceptron; Based on location features Extracting trajectory spatio-temporal features : ; Extracting lane segment spatial features based on map features : ; wherein is the relative heading vector between the trajectory and its lane segment in the current frame; and denote the start and end points of the lane segment, respectively. Step S22, using spatio-temporal encoding to capture the space-time correlation of the trajectories of vehicles i and j at each time step to determine whether the two cross-view trajectories are the same object; using relative spatial encoding to describe the spatial relationship between participants or between participants and lanes; Indicated as: ; ; wherein, represents a spatio-temporal encoding, represents a relative spatial encoding, concat represents a concatenation operation; for the relative spatial encoding when representing a relationship between participants, and is represented as and when represented as a relationship between a participant and a lane, and is represented as and ; Step S23, constructing an association component A, a motion fusion subgraph, a lane fusion subgraph, and a cooperative interaction subgraph for feature fusion: ; wherein is an MLP for binary classification to determine whether there is an association between two trajectories; and denotes the trajectories of the ego vehicle and the participating vehicle; The motion fusion subgraph aggregates the cooperative motion feature representation of the cross-view associated trajectories by considering the display association and the time-space correlation, and the feature fusion and updating process is: ; wherein k+1 -th update data of the motion fusion vector, is a feedforward network, denotes a cross-attention mechanism; The lane fusion subgraph enables the agent motion feature to query the relevant lane segment interaction feature by using the relative spatial encoding, and the feature fusion and updating process is: ; wherein k+1 order update data of the lane fusion vector, learnable labels representing semantic attributes associated with the respective lane segment, i.e. whether located at an intersection, turning direction, traffic control; The cooperative interaction subgraph represents the dense interaction representation between different agents in all views, and the feature fusion and updating process is: ; In the formula, represents the k+1 order update data of the cooperative interaction vector, represents the learnable interaction attribute feature related between two trajectories; Step S24, splicing trajectory motion feature vector , trajectory spatiotemporal feature vector , motion fusion vector , lane fusion vector , cooperative interaction vector , sending the splicing result into MLP to generate perception vector .

4. The vehicle motion trajectory prediction method based on large language model semantic guidance according to claim 3, characterized in that, The generating the high-dimensional motion prediction vector with high-level semantics and the motion prediction vector comprises: Step S31: Obtain the predicted vehicle from any viewpoint. The state of the predicted vehicle Centered on the status, all participating vehicles World coordinates ( , Convert to coordinates in a local coordinate system centered on the predicted vehicle. : ; wherein the predicted state of the vehicle includes a position , ) and a heading angle ; In step S32, the predicted vehicle is centered to divide the surroundings of the predicted vehicle into W key sectors, and the closest key target in each key sector is selected and the distance between the key target and the predicted vehicle is recorded; Step S33, preset high-level driving target, combine each key target and distance, generate text prompt word ; Step S34, text prompt word inputting the LLM for inference to obtain a prediction vector inputting the mapping network mapping to a high-dimensional motion prediction vector : ; 。 5. The vehicle motion trajectory prediction method based on large language model semantic guidance according to claim 4, characterized in that, The final prediction vector is determined according to the high-dimensional motion prediction vector and the motion prediction vector, and the final prediction trajectory of the predicted vehicle is obtained by decoding the final prediction vector and the perception vector, and the decoder parameters are adjusted by using the trajectory loss and the motion prediction loss, and the LLM inference ability of the lightweight neural network is confirmed according to the motion prediction loss, and the trajectory prediction is stopped by calling the LLM. Maintain a loss value for storing the motion prediction values ​​of the most recent Z steps. Fixed-size queue ,when When filled, the standard deviation is calculated. To determine whether a lightweight neural network possesses the inference capability of an LLM, thereby identifying the flag bits. : ; ; In the formula, is the motion prediction loss value of the zth step; is the loss mean value of the queue ; is not less than the stable threshold , the flag bit is set to True, indicating that the lightweight neural network does not currently have the inference ability of the LLM; when is less than the stable threshold , the flag bit is set to False, indicating that the lightweight neural network already has the inference ability of the LLM; combined flag , according to the high-dimensional motion prediction vector , motion prediction vector determining the final prediction vector and the final predicted trajectory : ; ; Training process loss function Including trajectory loss And motion prediction loss : ; ; where the trajectory loss is composed of a regression loss and a classification loss between the final predicted trajectory and the true trajectory , is a hyperparameter weight; is a hyperparameter, is an indicator function. is adjusted by a loss function to satisfy < After that, the calling of LLM is stopped.

6. A vehicle motion trajectory prediction system based on large language model semantic guidance, characterized in that, The method comprises the following steps: The acquisition module is used for acquiring the ego vehicle data, the roadside device data, the participant vehicle data and the map data, performing multi-view spatio-temporal alignment and vectorized feature extraction, and obtaining the position features, the motion features and the map features of each vehicle in the same coordinate system; the ego vehicle data comprises a historical trajectory sequence of the ego vehicle, vehicle type information and vehicle size information; the participant vehicle data comprises historical trajectories, type information and size information; the roadside device data comprises historical trajectory sequences of vehicles, vehicle type information and vehicle size information in a roadside view; the map comprises lane line vectors, turning information and intersection information in a scene; The GNN network feature extraction module is used for taking the participant vehicles as nodes, inputting the position features, the motion features and the map features of each vehicle into a graph neural network GNN for feature extraction, and generating a perception vector; The perception vector is input into a lightweight neural network for feature extraction to obtain a motion prediction vector; The LLM prediction module is used for generating a high-dimensional motion prediction vector with high-level semantics of the predicted vehicle by using a large language model LLM in combination with the position features of the surrounding area of the predicted vehicle; The trajectory prediction module is used for determining a final prediction vector according to the high-dimensional motion prediction vector and the motion prediction vector, and obtaining a final prediction trajectory of the predicted vehicle by decoding the final prediction vector; meanwhile, the decoder parameters are adjusted by using the trajectory loss and the motion prediction loss, and the LLM is stopped from being called for trajectory prediction after confirming whether the lightweight neural network has the LLM inference ability according to the motion prediction loss.

Images