Multi-modal vehicle trajectory prediction method based on graph interaction mechanism

Abstract

A multi-modal vehicle trajectory prediction method based on a graph interaction mechanism relates to the technical field of automatic driving, and comprises the following steps: constructing a deep learning database containing vehicle trajectory data, and dividing the deep learning database into a training set and a verification set; the method comprises the following steps: defining a vehicle trajectory prediction task, acquiring historical trajectory information and lane information of a target vehicle, encoding input information by using an LSTM encoder, processing vehicle interaction by using a multi-modal attention model, and extracting local features of lane and vehicle interaction by using a multi-modal attention model. And decoding by using a Bezier curve-based trajectory decoder to obtain a vehicle trajectory prediction model. According to the method, the attention capability of the model on important information is enhanced through the graph modeling interaction relationship, capturing the complex association between the tracks and combining with a multi-head attention mechanism, and the continuous track decoder based on the Bezier curve is introduced to decode the track prediction, so that the smooth and coherent tracks can be effectively generated; therefore, the prediction accuracy and feasibility are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile automatic driving, and specifically relates to a multimodal vehicle trajectory prediction method based on a graph interaction mechanism. Background Art

[0002] With the rapid development of intelligent transportation systems and autonomous driving technology, trajectory prediction has become an important research direction in the fields of traffic management and automatic navigation. Traditional trajectory prediction methods are usually based on physical models and rely on known kinematic and dynamic laws. However, in complex and dynamic environments, physical models are difficult to cope with uncertainty. In recent years, the rise of machine learning, especially deep learning, has made it possible to use big data for trajectory prediction. However, existing methods often require a large amount of labeled data for training and have high computational costs, and their real-time prediction capabilities are limited. At the same time, although methods based on historical trajectory data can capture certain laws, they are not adaptable enough to environmental changes and lack robustness for complex scenarios.

[0003] In order to solve these problems, a new trajectory prediction method is proposed. This method can accurately capture the complex correlation between trajectories by modeling interactive relationships through graphs, and combines the multi-head attention mechanism to enhance the model's ability to focus on important information. In addition, this method also introduces a continuous trajectory decoder based on Bezier curves to decode trajectory predictions. This decoder can effectively generate smooth and coherent trajectories, thereby improving the accuracy and feasibility of predictions. Although some studies have attempted to use social behavior models to improve prediction performance, such methods still face challenges in real-time applications. Therefore, the innovative method of using graph modeling, multi-head attention mechanism and Bezier curve-based decoder provides a new solution for trajectory prediction. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to propose a multimodal vehicle trajectory prediction method based on a graph interaction mechanism to solve the problem that the existing trajectory prediction cannot accurately model the interaction and may output a kinematically infeasible trajectory.

[0005] To achieve the above object, the present invention adopts the following technical solution: a multimodal vehicle trajectory prediction method based on a graph interaction mechanism, comprising the following steps:

[0006] S1. Build a deep learning database containing vehicle trajectory data and divide it into training set and validation set;

[0007] S2. Define the vehicle trajectory prediction task, obtain and process the historical trajectory information of traffic participants and the lane information of the target vehicle. The historical trajectory information of traffic participants includes the trajectory coordinates, speed, acceleration and yaw rate attributes of the vehicle in the past period of time; and the lane information of the target vehicle is the coordinates of the surrounding lane lines, turning lanes and whether it is an intersection extracted based on its position information;

[0008] S3. Use LSTM encoder to encode the input information, use graph multi-head attention model to process vehicle interaction, obtain the interaction features of vehicle trajectory, use multimodal sparse attention model to extract local features of lane and vehicle interaction, and finally use Bezier curve-based trajectory decoder to decode the trajectory;

[0009] S4. Construct a loss function, where the total loss is the weighted sum of regression loss and classification loss, and continuously calculate the loss value;

[0010] S5, loop S3 to S4 until the number of iterations reaches the preset value, then use the validation set for evaluation, select the weight with the highest accuracy on the validation set, and thus obtain the final vehicle trajectory prediction model.

[0011] Step S1 is specifically as follows:

[0012] S1.1. Extract the historical driving trajectory data of all vehicles in the scene where the target vehicle is located. Each vehicle corresponds to a data set. Each data set records in detail the specific location coordinates of the corresponding vehicle at each sampling time point in a predetermined period of time in the past. In this way, the historical driving trajectory of each vehicle in the scene can be fully grasped;

[0013] S1.2. The historical trajectory information collected in the scene is processed by converting it into a 2D coordinate system, thereby constructing a vehicle trajectory database suitable for deep learning algorithms. This database will serve as the basic data source for subsequent analysis and prediction of vehicle behavior;

[0014] S1.3. Divide the constructed vehicle trajectory database into two parts: one part is used as a training set to train the deep learning model so that it can learn and understand the characteristics and laws of vehicle trajectories; the other part is used as a validation set to evaluate the performance during the model training process, help adjust and optimize the model parameters, and ensure that the model has good generalization ability.

[0015] Step S2 is specifically as follows:

[0016] The goal of trajectory prediction is to predict multiple possible future trajectories of the target scene vehicle based on the state characteristics of the target scene vehicle and its surrounding scene vehicles. In a scenario with N scene vehicles, the state characteristics are defined as: is the historical state of the i-th vehicle, including the position of vehicle i in 2D coordinates Indicates vehicle speed Vehicle acceleration and the vehicle's yaw rate Right now M is the high-precision map information, that is, the lane centerline position obtained through the application program interface of the high-precision map. Based on the available features in this scene, the multimodal trajectory prediction task can be expressed as prediction T f is the forecast time range,

[0017] Step S3 is as follows:

[0018] S3.1. Use LSTM encoder to encode the input vehicle history status information and high-precision map information to extract features. For the vehicle history status, extract the features of the observation trajectory and the vehicle driving status features; for the high-precision map, the extracted feature information can reflect the spatial relationship. First, the input of the model is upgraded through the embedding layer to obtain an embedded vector of fixed length. Then use the LSTM encoder network to extract hidden state features The encoding process is as follows:

[0019]

[0020]

[0021]

[0022] φ represents the fully connected layer with Relu as the activation function, and are the output of the embedding layer and the hidden layer of the LSTM encoder, respectively. emb , W e are the learnable weights of the embedding layer and LSTM encoder;

[0023] S3.2. First, use a graph to represent the relationship between vehicles. Combine the graph attention mechanism with the multi-head attention mechanism to form a graph multi-head attention model to encode the interaction between vehicles. Each node in the node set V corresponds to an object in the traffic scene. Considering that each object may have different states at different time steps, define the node set Where n is the number of objects observed in the scene, th is the observed time step, is the hidden feature of the i-th vehicle at time t, and finally the vehicle interaction weight is calculated, and the expression is as follows:

[0024]

[0025]

[0026]

[0027] Where: Wqs, Wks and Wvs represent the learnable matrices of the corresponding matrices in vehicle interaction, and calculate the attention value of vehicle i at the hth attention head at the observation time at time t The expression is:

[0028]

[0029] The multi-head attention is operated in parallel h times and connected in series, the expression is:

[0030] C i =Con i (head 1 ,head 2 ,…,head h )∈R N×d ;

[0031] Where: Con is the concat operation, C i is the interaction feature at time i, and then the results are concatenated and weighted to obtain the interaction feature H in the observation time domain, expressed as:

[0032]

[0033] S3.3, the interaction module between vehicle and lane information is used to enable each vehicle to collect local static lane information, thereby generating prediction results of concerned lane information. In the vehicle and lane interaction encoder, Q is the interaction feature from the vehicle interaction encoder, K and V are the feature vectors from the map encoder, first sample the key matrix K, and obtain the sampled matrix Then the query matrix Q's i-th row qi is calculated with respect to The M value,

[0034]

[0035] Where: As an indicator to evaluate whether qi is important, The larger the value of , the more important qi is; for The jth row of is the scaling factor, d is the number of columns of K, L k is the number of rows of K, and then find the u q with the largest M value i , this u q i Form a new query matrix And Find the score value about K

[0036] Finally, we obtain the probabilistic sparse self-attention:

[0037]

[0038] The multi-head sparse attention is changed to multimodal sparse attention as shown in the following formula. Specifically, instead of connecting the results of a single head and projecting the connected vector onto a low-dimensional vector, the result of a single head is directly output. The final trajectory output depends on a single head.

[0039] Multimodel(Q,K,V)=(head 1 ,…,head h );

[0040] Where: Multimodel(Q,K,V) is multimodal sparse attention; head h For the hth "head", head h =Attention(Q h ,K h ,V h ), Q h ,K h ,V h are the Q, K, and V matrices of the h-th “head” respectively;

[0041] S3.4. After obtaining the vehicle interaction features and vehicle-image interaction features, they are concatenated with the features of the target vehicle and sent to the multimodal motion decoder to generate predictions for all actors. k possible futures are predicted, and for each mode, a simple MLP is applied, which has a regression head and a classification head for the trajectory, followed by a softmax function of the corresponding probability scores. The trajectory regression head uses a continuous parameterized representation based on Bezier curves.

[0042] The S3.4 specifically includes the following steps:

[0043] S3.4.1. In order to take advantage of the parameterized trajectory, the Bernstein basis polynomial is introduced, where the coefficients are control points with specific spatial meanings, so as to obtain better convergence. Specifically, the n-order Bezier curve is expressed as:

[0044]

[0045] In the formula, is a Bernstein basis of order i, is the binomial coefficient, t is the parametric curve variable, p iis the control point. For an n-order Bezier curve, there are a total of n+1 control points. The first and last control points are always the endpoints of the curve. Since the Bezier curve is defined on t∈[0,1], the present invention applies the actual time τ∈[0,τ max ] is normalized, so the parameter curve can be written as In addition, due to the properties of the sagittal graph, the derivative of the nth-order Bezier curve is still a Bezier curve, and its control points are defined as That is, the velocity profile of the trajectory can be calculated by the following formula:

[0046]

[0047] S3.4.2. Use a simple MLP as the regression head to perform the mapping from the concatenated vehicle features to the control points, and then the position coordinate Y of each predicted trajectory pos ∈R T×2 It can be obtained by using the constant basis matrix B∈R T×(n+1) and the corresponding predicted 2D control point P∈R (n+1)×2 Simply calculate,

[0048]

[0049] Where T is the number of sampling timestamps required to predict the trajectory, To normalize the time points, for agents with non-holonomic constraints, such as vehicles and bicycles, the yaw angle is aligned with the tangent vector of the trajectory. We can derive the heading angle of each state from the velocity estimate, and finally further transform the predicted trajectory back to the global coordinates according to the corresponding anchor pose of the actor.

[0050] Step S4 is specifically as follows:

[0051] S4.1. Setting the training parameters of the vehicle trajectory prediction model, including the number of iterations of the vehicle trajectory prediction model and the learning rate of the training;

[0052] S4.2. Construct a loss function and calculate its loss value.

[0053] The S4.2 specifically includes the following steps:

[0054] S4.2.1. The overall loss function is the weighted sum of regression loss and classification loss:

[0055]

[0056] Where ω∈[0,1] is the weight to balance these components. We use the WTA strategy to handle multimodality. For each agent, we find the best predicted trajectory k* among k hypotheses by selecting the one with the smallest final displacement error;

[0057] S4.2.2. For classification tasks, this patent uses the maximum margin loss to distinguish the positive mode from other similar modes.

[0058]

[0059] S4.2.3 For the trajectory regression task, in addition to the position coordinate regression, this method also introduces an optional yaw angle loss to provide auxiliary supervision, thereby obtaining

[0060]

[0061] where ˉY(·) represents the ground truth state, and Y k*(·) is the predicted position and yaw angle of the WTA mode. In this study, the smooth L1 loss is used as the position regression loss, and the yaw regression loss is specified as

[0062]

[0063] where cossim(·,·) is the cosine similarity metric, which takes the value 1 for two aligned yaw vectors and takes the value 0 for two opposite yaw vectors. ,Considering yaw angle loss implicitly enhances the consistency between consecutive states, making the predicted trajectory more smooth and kinematically feasible, and producing more realistic trajectories, especially for low-speed vehicles.

[0064] The beneficial effects of the present invention are: by using the LSTM encoder to encode the input information in time series, by modeling the interactive relationship through the graph, the multi-head attention mechanism of the graph is used to enhance the capture of important interactive features, and the local features of the lane and the vehicle are extracted through the multimodal sparse attention model to improve the accuracy of trajectory prediction. Finally, the trajectory decoder based on the Bezier curve generates a smooth and natural trajectory, which enhances the interpretability and operability of the prediction results. This method has broad application prospects, can effectively support autonomous driving and intelligent transportation systems, provide a scientific basis for improving traffic safety and efficiency, and enhance decision-making support capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of the present invention.

[0066] Figure 2 This is the overall framework diagram of the vehicle trajectory prediction model of the present invention.

[0067] Figure 3 This is the structural diagram of the vehicle and lane interaction encoder. DETAILED DESCRIPTION

[0068] The technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of this specification. It should be noted that the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0069] See also Figure 1-Figure 3 , a multimodal vehicle trajectory prediction method based on graph interaction mechanism, comprising the following steps:

[0070] S1. Build a deep learning database containing vehicle trajectory data and divide it into a training set and a validation set. Step S1 is as follows:

[0071] S1.1. Extract the historical driving trajectory data of all vehicles in the scene where the target vehicle is located. Each vehicle corresponds to a data set. Each data set records in detail the specific location coordinates of the corresponding vehicle at each sampling time point in a predetermined period of time in the past. In this way, the historical driving trajectory of each vehicle in the scene can be fully grasped;

[0072] S1.2. The historical trajectory information collected in the scene is processed by converting it into a 2D coordinate system, thereby constructing a vehicle trajectory database suitable for deep learning algorithms. This database will serve as the basic data source for subsequent analysis and prediction of vehicle behavior;

[0073] S1.3. Divide the constructed vehicle trajectory database into two parts: one part is used as a training set to train the deep learning model so that it can learn and understand the characteristics and laws of vehicle trajectories; the other part is used as a validation set to evaluate the performance during the model training process, help adjust and optimize the model parameters, and ensure that the model has good generalization ability.

[0074] S2, define the vehicle trajectory prediction task, obtain and process the historical trajectory information of the traffic participants and the lane information of the target vehicle. The historical trajectory information of the traffic participants includes the trajectory coordinates, speed, acceleration and yaw rate attributes of the vehicle in the past period of time; and the lane information of the target vehicle is the coordinates of the surrounding lane lines, the turning lane and whether it is an intersection extracted based on its position information; step S2 is as follows:

[0075] The goal of trajectory prediction is to predict multiple possible future trajectories of the target scene vehicle based on the state characteristics of the target scene vehicle and its surrounding scene vehicles. In a scenario with N scene vehicles, the state characteristics are defined as: is the historical state of the i-th vehicle, including the position of vehicle i in 2D coordinates Indicates vehicle speed Vehicle acceleration and the vehicle's yaw rate Right now M is the high-precision map information, that is, the lane centerline position obtained through the application program interface of the high-precision map. Based on the available features in this scene, the multimodal trajectory prediction task can be expressed as prediction T f is the forecast time range,

[0076] S3, using the LSTM encoder to encode the input information, using the graph multi-head attention model to process the vehicle interaction, obtaining the interaction features of the vehicle trajectory, using the multimodal sparse attention model to extract the local features of the lane and vehicle interaction, and finally using the trajectory decoder based on the Bezier curve to decode the trajectory; step S3 is as follows:

[0077] S3.1. Use LSTM encoder to encode the input vehicle history status information and high-precision map information to extract features. For the vehicle history status, extract the features of the observation trajectory and the vehicle driving status features; for the high-precision map, the extracted feature information can reflect the spatial relationship. First, the input of the model is upgraded through the embedding layer to obtain an embedded vector of fixed length. Then use the LSTM encoder network to extract hidden state features The encoding process is as follows:

[0078]

[0079]

[0080]

[0081] Where: φ represents the fully connected layer with Relu as the activation function, and are the output of the embedding layer and the hidden layer of the LSTM encoder, W emb , W e are the learnable weights of the embedding layer and LSTM encoder;

[0082] S3.2. First, use a graph to represent the relationship between vehicles. Combine the graph attention mechanism with the multi-head attention mechanism to form a graph multi-head attention model to encode the interaction between vehicles. Each node in the node set V corresponds to an object in the traffic scene. Considering that each object may have different states at different time steps, define the node set Where n is the number of objects observed in the scene, th is the observed time step, is the hidden feature of the i-th vehicle at time t, and finally the vehicle interaction weight is calculated, and the expression is as follows:

[0083]

[0084]

[0085]

[0086] Where: Wqs, Wks and Wvs represent the learnable matrices of the corresponding matrices in vehicle interaction, and calculate the attention value of vehicle i at the hth attention head at the observation time at time t The expression is:

[0087]

[0088] The multi-head attention is operated in parallel h times and connected in series, the expression is:

[0089] C i =Con i (head 1 ,head 2 ,…,head h )∈R N×d ;

[0090] Where: Con is the concat operation, C i is the interaction feature at time i, and then the results are concatenated and weighted to obtain the interaction feature H in the observation time domain, expressed as:

[0091]

[0092] S3.3, the interaction module between vehicle and lane information is used to enable each vehicle to collect local static lane information, thereby generating prediction results of concerned lane information. In the vehicle and lane interaction encoder, Q is the interaction feature from the vehicle interaction encoder, K and V are the feature vectors from the map encoder, first sample the key matrix K, and obtain the sampled matrix Then the query matrix Q's i-th row qi is calculated with respect to The M value,

[0093]

[0094] Where: To evaluate q i Is it an important indicator? The larger the value of q i The more important; for The jth row of is the scaling factor, d is the number of columns of K, L k is the number of rows of K, and then find the u q with the largest M valuei , this u q i Form a new query matrix And Find the score value about K

[0095] Finally, we obtain the probabilistic sparse self-attention:

[0096]

[0097] The multi-head sparse attention is changed to multimodal sparse attention as shown in the following formula. Specifically, instead of connecting the results of a single head and projecting the connected vector onto a low-dimensional vector, the result of a single head is directly output. The final trajectory output depends on a single head.

[0098] Multimodel(Q,K,V)=(head 1 ,…,head h );

[0099] Where: Multimodel(Q,K,V) is multimodal sparse attention; head h For the hth "head", head h =Attention(Q h ,K h ,V h ), Q h ,K h ,V h are the Q, K, and V matrices of the h-th “head” respectively;

[0100] S3.4. After obtaining the vehicle interaction features and vehicle-image interaction features, they are concatenated with the features of the target vehicle and sent to the multimodal motion decoder to generate predictions for all actors. k possible futures are predicted, and for each mode, a simple MLP is applied, which has a regression head and a classification head for the trajectory, followed by a softmax function of the corresponding probability scores. The trajectory regression head uses a continuous parameterized representation based on Bezier curves.

[0101] The S3.4 specifically includes the following steps:

[0102] S3.4.1. In order to take advantage of the parameterized trajectory, the Bernstein basis polynomial is introduced, where the coefficients are control points with specific spatial meanings, so as to obtain better convergence. Specifically, the n-order Bezier curve is expressed as:

[0103]

[0104] Where: is a Bernstein basis of order i, is the binomial coefficient, t is the parametric curve variable, p i is the control point. For an n-order Bezier curve, there are a total of n+1 control points. The first and last control points are always the endpoints of the curve. Since the Bezier curve is defined on t∈[0,1], the present invention applies the actual time τ∈[0,τ max ] is normalized, so the parameter curve can be written as In addition, due to the properties of the sagittal graph, the derivative of the nth-order Bezier curve is still a Bezier curve, and its control points are defined as That is, the velocity profile of the trajectory can be calculated by the following formula:

[0105]

[0106] S3.4.2. Use a simple MLP as the regression head to perform the mapping from the concatenated vehicle features to the control points, and then the position coordinate Y of each predicted trajectory pos ∈R T×2 It can be obtained by using the constant basis matrix B∈R T×(n+1) and the corresponding predicted 2D control point P∈R (n+1)×2 Simply calculate,

[0107]

[0108] Where T is the number of sampling timestamps required to predict the trajectory, To normalize the time points, for agents with non-holonomic constraints, such as vehicles and bicycles, the yaw angle is aligned with the tangent vector of the trajectory. We can derive the heading angle of each state from the velocity estimate, and finally further transform the predicted trajectory back to the global coordinates according to the corresponding anchor pose of the actor.

[0109] S4, construct a loss function, where the total loss is the weighted sum of the regression loss and the classification loss, and the loss value is continuously calculated; step S4 is as follows:

[0110] S4.1. Setting the training parameters of the vehicle trajectory prediction model, including the number of iterations of the vehicle trajectory prediction model and the learning rate of the training;

[0111] S4.2. Construct a loss function and calculate its loss value.

[0112] The S4.2 specifically includes the following steps:

[0113] S4.2.1. The overall loss function is the weighted sum of regression loss and classification loss:

[0114]

[0115] Where ω∈[0,1] is the weight to balance these components. We use the WTA strategy to handle multimodality. For each agent, we find the best predicted trajectory k* among k hypotheses by selecting the one with the smallest final displacement error;

[0116] S4.2.2. For classification tasks, this patent uses the maximum margin loss to distinguish the positive mode from other similar modes.

[0117]

[0118] S4.2.3 For the trajectory regression task, in addition to the position coordinate regression, this method also introduces an optional yaw angle loss to provide auxiliary supervision, thereby obtaining

[0119]

[0120] where ˉY(·) represents the ground truth state, and Y k*(·) is the predicted position and yaw angle of the WTA mode. In this study, the smooth L1 loss is used as the position regression loss, and the yaw regression loss is specified as

[0121]

[0122] where cossim(·,·) is the cosine similarity metric, which takes the value of 1 for two aligned yaw vectors and −1 for two opposite yaw vectors. Considering the yaw angle loss implicitly enhances the consistency between consecutive states, making the predicted trajectory more smooth and kinematically feasible and producing more realistic trajectories, especially for low-speed vehicles.

[0123] S5, loop S3 to S4 until the number of iterations reaches the preset value, then use the validation set for evaluation, select the weight with the highest accuracy on the validation set, and thus obtain the final vehicle trajectory prediction model.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0125] Parts of the present invention not described in detail are prior art.

Claims

1. A multimodal vehicle trajectory prediction method based on a graph interaction mechanism, characterized by: The steps include: S1. Build a deep learning database containing vehicle trajectory data and divide it into training set and validation set; S2. Define the vehicle trajectory prediction task, obtain and process the historical trajectory information of traffic participants and the lane information of the target vehicle. The historical trajectory information of traffic participants includes the trajectory coordinates, speed, acceleration and yaw rate attributes of the vehicle in the past period of time; and the lane information of the target vehicle is the coordinates of the surrounding lane lines, turning lanes and whether it is an intersection extracted based on its position information; S3. Use LSTM encoder to encode the input information, use graph multi-head attention model to process vehicle interaction, obtain the interaction features of vehicle trajectory, use multimodal sparse attention model to extract local features of lane and vehicle interaction, and finally use Bezier curve-based trajectory decoder to decode the trajectory; S4. Construct a loss function, where the total loss is the weighted sum of regression loss and classification loss, and continuously calculate the loss value; S5, loop S3 to S4 until the number of iterations reaches the preset value, then use the validation set for evaluation, select the weight with the highest accuracy on the validation set, and thus obtain the final vehicle trajectory prediction model.

2. The multimodal vehicle trajectory prediction method based on graph interaction mechanism according to claim 1 is characterized by: Step S1 is specifically as follows: S1.

1. Extract the historical driving trajectory data of all vehicles in the scene where the target vehicle is located. Each vehicle corresponds to a data set. Each data set records in detail the specific location coordinates of the corresponding vehicle at each sampling time point in a predetermined period of time in the past. In this way, the historical driving trajectory of each vehicle in the scene can be fully grasped; S1.

2. The historical trajectory information collected in the scene is processed by converting it into a 2D coordinate system, thereby constructing a vehicle trajectory database suitable for deep learning algorithms. This database will serve as the basic data source for subsequent analysis and prediction of vehicle behavior; S1.

3. Divide the constructed vehicle trajectory database into two parts: one part is used as a training set to train the deep learning model so that it can learn and understand the characteristics and laws of vehicle trajectories; the other part is used as a validation set to evaluate the performance during the model training process, help adjust and optimize the model parameters, and ensure that the model has good generalization ability.

3. The multimodal vehicle trajectory prediction method based on graph interaction mechanism according to claim 1 is characterized by: Step S2 is specifically as follows: The goal of trajectory prediction is to predict multiple possible future trajectories of the target scene vehicle based on the state characteristics of the target scene vehicle and its surrounding scene vehicles. In a scenario with N scene vehicles, the state characteristics are defined as: is the historical state of the i-th vehicle, including the position of vehicle i in 2D coordinates Indicates vehicle speed Vehicle acceleration and the vehicle's yaw rate Right now M is the high-precision map information, that is, the lane centerline position obtained through the application program interface of the high-precision map. Based on the available features in this scene, the multimodal trajectory prediction task can be expressed as prediction T f is the forecast time range, 4. The multimodal vehicle trajectory prediction method based on graph interaction mechanism according to claim 1 is characterized by: Step S3 is as follows: S3.

1. Use LSTM encoder to encode the input vehicle history status information and high-precision map information to extract features. For the vehicle history status, extract the features of the observation trajectory and the vehicle driving status features; for the high-precision map, the extracted feature information can reflect the spatial relationship. First, the input of the model is upgraded through the embedding layer to obtain an embedded vector of fixed length. Then use the LSTM encoder network to extract hidden state features The encoding process is as follows: Where: φ represents the fully connected layer with Relu as the activation function, and are the output of the embedding layer and the hidden layer of the LSTM encoder, W emb , W e are the learnable weights of the embedding layer and LSTM encoder; S3.

2. First, use a graph to represent the relationship between vehicles. Combine the graph attention mechanism with the multi-head attention mechanism to form a graph multi-head attention model to encode the interaction between vehicles. Each node in the node set V corresponds to an object in the traffic scene. Considering that each object may have different states at different time steps, define the node set Where n is the number of objects observed in the scene, th is the observed time step, is the hidden feature of the i-th vehicle at time t, and finally the vehicle interaction weight is calculated, and the expression is as follows: Where: Wqs, Wks and Wvs represent the learnable matrices of the corresponding matrices in vehicle interaction, and calculate the attention value of vehicle i at the hth attention head at the observation time at time t The expression is: The multi-head attention is operated in parallel h times and connected in series, the expression is: C i =Con i (head1,head2,…,head h )∈R N×d ; Where: Con is the concat operation, C i is the interaction feature at time i, and then the results are concatenated and weighted to obtain the interaction feature H in the observation time domain, expressed as: S3.3, the interaction module between vehicle and lane information is used to enable each vehicle to collect local static lane information, thereby generating prediction results of concerned lane information. In the vehicle and lane interaction encoder, Q is the interaction feature from the vehicle interaction encoder, K and V are the feature vectors from the map encoder, first sample the key matrix K, and obtain the sampled matrix Then the query matrix Q's i-th row qi is calculated with respect to The M value, Where: To evaluate q i Is it an important indicator? The larger the value of q i The more important; for The jth row of is the scaling factor, d is the number of columns of K, L k is the number of rows of K, and then find the u q with the largest M value i , this u q i Form a new query matrix And Find the score value about K Finally, we obtain the probabilistic sparse self-attention: The multi-head sparse attention is changed to multimodal sparse attention as shown in the following formula. Specifically, instead of connecting the results of a single head and projecting the connected vector onto a low-dimensional vector, the result of a single head is directly output. The final trajectory output depends on a single head. Multimodel(Q,K,V)=(head1,…,head h ); Where: Multimodel(Q,K,V) is multimodal sparse attention; head h For the hth "head", head h =Attention(Q h ,K h ,V h ), Q h ,K h ,V h are the Q, K, and V matrices of the h-th "head" respectively; S3.

4. After obtaining the vehicle interaction features and vehicle-image interaction features, they are concatenated with the features of the target vehicle and sent to the multimodal motion decoder to generate predictions for all actors. k possible futures are predicted, and for each mode, a simple MLP is applied, which has a regression head and a classification head for the trajectory, followed by a softmax function of the corresponding probability scores. The trajectory regression head uses a continuous parameterized representation based on Bezier curves.

5. The multimodal vehicle trajectory prediction method based on graph interaction mechanism according to claim 4 is characterized by: The S3.4 specifically includes the following steps: S3.4.

1. In order to take advantage of the parameterized trajectory, the Bernstein basis polynomial is introduced, where the coefficients are control points with specific spatial meanings, so as to obtain better convergence. Specifically, the n-order Bezier curve is expressed as: In the formula, is a Bernstein basis of order i, is the binomial coefficient, t is the parametric curve variable, p i is the control point. For an n-order Bezier curve, there are a total of n+1 control points. The first and last control points are always the endpoints of the curve. Since the Bezier curve is defined on t∈[0,1], the present invention applies the actual time τ∈[0,τ max ] is normalized, so the parameter curve can be written as In addition, due to the properties of the sagittal graph, the derivative of the nth-order Bezier curve is still a Bezier curve, and its control points are defined as That is, the velocity profile of the trajectory can be calculated by the following formula: S3.4.

2. Use a simple MLP as the regression head to perform the mapping from the concatenated vehicle features to the control points, and then the position coordinate Y of each predicted trajectory pos ∈R T×2 It can be obtained by using the constant basis matrix B∈R T×(n+1) and the corresponding predicted 2D control point P∈R (n+1)×2 Simply calculate, Where T is the number of sampling timestamps required to predict the trajectory, To normalize the time points, for agents with non-holonomic constraints, such as vehicles and bicycles, the yaw angle is aligned with the tangent vector of the trajectory. We can derive the heading angle of each state from the velocity estimate, and finally further transform the predicted trajectory back to the global coordinates according to the corresponding anchor pose of the actor.

6. The multimodal vehicle trajectory prediction method based on graph interaction mechanism according to claim 1 is characterized by: Step S4 is specifically as follows: S4.

1. Setting the training parameters of the vehicle trajectory prediction model, including the number of iterations of the vehicle trajectory prediction model and the learning rate of the training; S4.

2. Construct a loss function and calculate its loss value.

7. The multimodal vehicle trajectory prediction method based on graph interaction mechanism according to claim 6 is characterized by: The S4.2 specifically includes the following steps: S4.2.

1. The overall loss function is the weighted sum of regression loss and classification loss: Where ω∈[0,1] is the weight to balance these components. We use the WTA strategy to handle multimodality. For each agent, we find the best predicted trajectory k* among k hypotheses by selecting the one with the smallest final displacement error; S4.2.

2. For classification tasks, this patent uses the maximum margin loss to distinguish the positive mode from other similar modes. S4.2.3 For the trajectory regression task, in addition to the position coordinate regression, this method also introduces an optional yaw angle loss to provide auxiliary supervision, thereby obtaining where ˉY(·) represents the ground truth state, and Y k*(·) is the predicted position and yaw angle of the WTA mode. In this study, the smooth L1 loss is used as the position regression loss, and the yaw regression loss is specified as where cossim(·,·) is the cosine similarity metric, which takes the value of 1 for two aligned yaw vectors and −1 for two opposite yaw vectors. Considering the yaw angle loss implicitly enhances the consistency between consecutive states, making the predicted trajectory more smooth and kinematically feasible and producing more realistic trajectories, especially for low-speed vehicles.

Images