Multi-modal trajectory prediction method based on vehicle interaction diagram space-time decoupling coding

Abstract

The invention relates to a multi-modal trajectory prediction method based on vehicle interaction diagram space-time decoupling coding, and belongs to the technical field of automatic driving automobiles. The method comprises the following steps: generating graph structure data according to a traffic scene trajectory data set; constructing an interaction graph considering multiple traffic factors based on the graph structure data; when the characteristics of the interaction graph are updated, capturing the change of the node characteristics of the interaction graph on a time sequence through a time sequence coding module; multi-modal prediction is completed by constructing a trajectory decoding module and a fractional decoding module; and designing a prediction model loss objective function and carrying out training. According to the method, the influence of multi-source factors on motion prediction is combined through a heterogeneous graph attention method, decoupling coding is carried out on spatial-temporal characteristics, and the characteristics on a time sequence are better captured. Compared with a traditional vehicle track prediction method, the method can complete more refined prediction, and is better in scene generalization.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving vehicles and relates to a multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graphs. Background Art

[0002] Autonomous driving technology focuses on solving traffic challenges such as safety, efficiency, and energy, and therefore has received great attention from academia and industry. It achieves the autonomous driving capability of the car through modules such as environment perception, trajectory prediction, decision-making, and planning control. One of the efficient and accurate trajectory prediction methods can obtain the future state of itself and nearby traffic participants, and pass the information to the downstream decision-making module to ensure the safe driving of the car in a dynamic environment.

[0003] At present, trajectory prediction can be mainly divided into physics-based methods, classical machine learning-based methods, deep learning-based methods, and reinforcement learning-based methods. Most traditional prediction methods are only applicable to simple traffic scenarios or short-term predictions, while the actual traffic scenario is a highly complex and highly interactive task. Deep learning-based methods are data-driven, efficiently extract target features, and have powerful nonlinear modeling capabilities, and are receiving more and more attention. Deep learning-based methods can be further divided into specific methods based on recursive neural networks, convolutional neural networks, graph neural networks, and generative adversarial networks. In traffic tasks, the interactivity between traffic participants is a key link, and graph neural networks can efficiently express the interaction between node information and have strong migration applicability, which is very suitable for solving trajectory prediction problems based on interaction-related factors.

[0004] However, in the trajectory prediction task, in addition to focusing on interactive factors, the characteristics of traffic participants themselves are also important factors affecting the trajectory prediction results, and the scene map and traffic rules have continuous constraints on traffic. Therefore, there is an urgent need for a trajectory prediction method that comprehensively considers multi-source information to achieve multi-modal prediction of trajectories and provide information support for downstream safe and efficient decision-making planning. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graphs.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graphs comprises the following steps:

[0008] S1, generating graph structure data based on traffic scene trajectory dataset;

[0009] S2, construct an interactive graph considering multiple traffic factors based on graph structure data;

[0010] S3, when the interaction graph updates its features, the temporal coding module is used to capture the changes in the interaction graph node features in the time series;

[0011] S4, completing multimodal prediction by constructing trajectory decoding module and score decoding module;

[0012] S5. Design the prediction model loss objective function and perform training.

[0013] Further, in step S1, the graph structure data is represented as Which is used to build graph structure data The data sources include the historical motion status of the intelligent body. And high-definition map information

[0014]

[0015]

[0016]

[0017] Among them, i represents the number of agents that consider interactive prediction within a certain range, Represents the coordinate information of the agent at time t and heading angle

[0018] Further, in step S2, the following steps are included:

[0019] S21. Based on historical movement status Extract historical vehicle node features; convert historical motion states The vehicle node features are output through the ModuleList model list composed of the fully connected layer, regularization layer, activation layer, GRU encoding layer and residual connection module:

[0020]

[0021]

[0022] in, is the historical characteristics of an agent at time t; Indicates encoding through a list of sequential models of a specific agent type, the superscript at indicates a specific type of agent (agent-type), and the subscript hist indicates historical information; N t The historical features of all agents as output of this module.

[0023] S22. Based on global high-definition map information Extract lane line node features; for high-definition map information Use lanelet2 to create a lane object and extract the lane centerline, determine the lane node, and calculate the distance between the agent and the lane node:

[0024]

[0025]

[0026] in It represents the starting and ending indexes of the centerline of each lane whose distance from agent i is within the threshold at time t.

[0027] S23, establish the interaction between vehicle agents and the interaction between agents and lanes; wherein, for the interaction between vehicle agents, the target agent node feature Other agents within the specified range are embedded as neighbor nodes into their own node features, and directed edges from neighbor nodes to target nodes are constructed. The adjacency matrix is ​​used to represent the connection with neighbor agents. At the same time, the relative position information between nodes is calculated and embedded in the edge attribute variables constructed in the PyTorch_Geometric library to represent the interaction between agent nodes, that is:

[0028]

[0029]

[0030] in, As the set of edge attributes of nodes i and j, the relative position relationship between nodes is embedded, E t It is a collection of edge indexes, edge attributes, and edge types.

[0031] For the implicit interaction between the vehicle agent and the lane line, the interaction between the agent and the lane is represented by nodes and edges in the graph structure. And edge features In the case of , filter the index of the target lane line node according to the distance The distances between the corresponding agent nodes and lane line nodes are embedded into the edge attribute variables constructed in the PyTorch_Geometric library to capture the implicit interaction between the agent and the lane.

[0032] S24. By applying the multi-head self-attention mechanism to update the node features with attention weighted features, the construction and update of the interaction graph considering multi-source information can be realized:

[0033] MH(Q,K,V)=[head1,head2,head3]W

[0034]

[0035] In the multi-head self-attention mechanism, three attention heads are set, Q, K, and V are query vectors, key vectors, and value vectors respectively. is the weight matrix; scaled dot product attention is used to calculate the attention score α:

[0036]

[0037] Among them, soft max(·) represents the normalization layer, (·) T represents the matrix transpose, d k Indicates the dimension of the key vector.

[0038] Further, in step S3, the following steps are included:

[0039] S31, the target node completes its own feature matrix N by aggregating neighbor information t When updating the aggregation, the self-attention mechanism between the connected nodes is used to focus on the dynamic update results of the features over time. It is expressed as:

[0040]

[0041] in, is the linear transformation of the feature matrix of the previous time step, As the attention-weighted feature matrix of the target agent,

[0042] S32. For the attention weighted feature matrix, the temporal information is explicitly encoded through the temporal encoding module to obtain the final feature matrix of the target agent at the current time step:

[0043]

[0044] Among them, g0 and g1 are composed of a two-layer MLP network. It is a time series feature fusion module. The time series feature fusion module encodes the timestamp through a specific linear layer and encodes the time information into a high-dimensional vector. Then the time encoding is repeated to the same dimension as the feature, added to the hidden state and the feature mask is set. Then the hidden state is updated through the residual module, and the final output is the node time series feature of the current time step.

[0045] Furthermore, the state characteristics of the corresponding time step are stored through the data playback module, and the historical sequence information is provided to the time series encoding module; wherein at least a data playback module is provided for tracking and storing the target agent characteristics and the scene characteristics respectively.

[0046] Further, in step S32, for the data playback module used to track and store the target agent features, the LSTM is used to sequentially track the changes in the representation of the target agent's temporal features and hidden state at time t, and the formula is expressed as:

[0047]

[0048] where h t is the hidden state in LSTM, h t-1 is the input target node feature, Represents the feature matrix of the target agent at time t; in interactive prediction, the target agent is alternating. When the target agent of interest is confirmed, the remaining traffic participants participate in the calculation of the interactive timing graph as neighbor agents. Each agent making a prediction stores the node features in the data playback module as a target agent.

[0049] Further, in step S32, when the scene feature data playback module is constructed, at time step t, the feature matrix N t Apply a linear layer to initialize the module, and then build a multi-layer self-attention operation network to store scene features:

[0050]

[0051]

[0052] Where g2 is the linear layer for extracting feature matrix, l represents the number of layers, and the storage of scene features can also ensure that the map features are fully considered when the subsequent decoding module outputs the trajectory;

[0053] For each self-attention layer built, layer normalization is performed. After the last layer L, the node features are aggregated through the maximum pooling operation to summarize the scene-related information. The LSTM module is also used to associate information through time to complete the data storage of scene features:

[0054]

[0055] Among them, h t ' represents the hidden state of the map tracked by the module at time t; h t ' -1 Indicates the hidden state of the map at time t-1.

[0056] Furthermore, in step S4, a trajectory decoding module and a score decoding module are constructed to model the probability distribution of multimodal trajectory prediction, which specifically includes the following steps:

[0057] S41. Taking into account the agent's own characteristics, interaction characteristics and map characteristics, the task of trajectory decoding is completed through the GRU network structure, and the coordinates of the trajectory at each time step are finally output:

[0058]

[0059] in, is the agent’s multimodal future decoder, is the concatenation of features in the traffic task, is the predicted future trajectory of the agent’s kth mode;

[0060] S42, the score decoder follows the same structure as the trajectory decoder, and finally outputs the confidence score of the proposed trajectory, which is further passed through the softmax layer to produce the generation probability distribution of the multimodal trajectory:

[0061]

[0062] Among them, c i represents the trajectory confidence score, Represents the confidence range.

[0063] Further, in step S5, a loss function is designed according to the trajectory decoding module and the score decoding module to perform model training.

[0064] For trajectory loss, smoothing is used across all predicted time steps. Loss, the trajectory regression loss is defined as:

[0065]

[0066] in is the actual trajectory position at time t, is the predicted trajectory closest to the true trajectory position; k * is the index in the multimodal predicted trajectory that is closest to the true trajectory position, which is calculated by The end point of the real trajectory The distance between them is:

[0067]

[0068] Design a cross entropy loss function for the loss function of the predicted score Calculating the True Score The loss between the predicted probability distribution P and the predicted score loss is defined as:

[0069]

[0070] Define the score distribution corresponding to the true trajectory:

[0071]

[0072] The total loss is the weighted sum of trajectory regression loss and score loss:

[0073]

[0074] Where δ is a parameter set to balance the two learning objectives.

[0075] The beneficial effects of the present invention are:

[0076] The present invention performs spatiotemporal decoupling encoding of driving tasks in vehicle trajectory prediction, better learns the temporal representation in dynamic traffic scenes, and realizes interactive trajectory prediction considering multiple sources. Compared with traditional vehicle motion prediction methods, the present invention can complete multi-agent interactive multimodal trajectory prediction in a more refined manner, and has stronger generalization for scenes.

[0077] (1) The present invention designs a trajectory prediction model encoding module based on graph neural network and self-attention mechanism, which comprehensively considers the vehicle historical characteristics, vehicle interaction characteristics and traffic scene characteristics. The constructed interaction graph can more accurately and effectively express the traffic task of the current time frame, and effectively represents the interaction characteristics between traffic factors.

[0078] (2) The present invention designs a time series coding module based on target information data playback. By receiving the updated interaction graph node features from upstream, the time series coding module is constructed to capture the features in the time series and update them into the vehicle status features, thereby better decoupling the time and space features of the observed traffic tasks.

[0079] (3) The present invention constructs a multimodal trajectory prediction trajectory decoding module and a score decoding module, applies a GRU module to track proxy features and map features, and models the probability distribution of multimodal trajectories based on this, so that the decoded output can comprehensively consider map factors and proxy features. The designed trajectory smoothing loss and cross entropy loss fusion objective function can comprehensively and efficiently train the model.

[0080] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0082] Figure 1 It is the overall flow chart of the method proposed by the present invention;

[0083] Figure 2 A flowchart for constructing interactive graph representations of traffic tasks based on graph neural networks;

[0084] Figure 3 This is the workflow diagram of the timing encoding module and the data playback module. DETAILED DESCRIPTION

[0085] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0086] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0087] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0088] See also Figure 1 to Figure 3, the present invention relates to an interactive multimodal trajectory prediction method based on a dynamic time-series graph neural network. First, an interactive graph that considers the motion characteristics, interaction characteristics and map attributes of traffic agents is constructed through a graph attention network. Secondly, a data playback module is constructed to store key information of past frames, and a time-series encoding module is used to capture the changes in the constructed interaction graph features in the time series; finally, a trajectory decoding module and a score decoding module are constructed to model the probability distribution of multimodal trajectory prediction, and a loss function is designed accordingly for model training. The present invention combines the influence of map factors on motion prediction, and fully considers the interactive characteristics in traffic scenes. Compared with traditional vehicle motion prediction methods, the present invention can complete multi-agent interactive multimodal trajectory prediction with higher credibility, higher accuracy, and a wider range of usage scenarios. The method specifically includes the following steps:

[0089] S1, generating graph structure data based on traffic scene trajectory dataset;

[0090] S2, construct an interactive graph considering multiple traffic factors based on graph structure data;

[0091] S3, when the interaction graph updates its features, the temporal coding module is used to capture the changes in the interaction graph node features in the time series;

[0092] S4, complete multimodal prediction by constructing trajectory and score decoding module;

[0093] S5. Design the prediction model loss objective function and perform training.

[0094] Example

[0095] In step S1, this embodiment constructs graph structure data through the historical trajectory characteristics of the vehicle. Specifically, the historical trajectory characteristics of the vehicle include the historical motion status of the vehicle intelligent body and high-definition map information, and the constructed graph structure data is used for subsequent training, verification and testing of the prediction model.

[0096] Specifically, in step S1, the graph structure data is represented as Which is used to build graph structure data The data sources include the historical motion status of the intelligent body. And high-definition map information

[0097]

[0098]

[0099]

[0100] Among them, i represents the number of agents that consider interactive prediction within a certain range, Represents the coordinate information of the agent at time t and heading angle In order to ensure the translation invariance and rotation invariance of the predicted intelligent agent, the historical trajectory data needs to be transformed relative to the position and heading angle of the target agent in the current frame before feature extraction.

[0101] In step S2, according to the graph structure data The historical state of motion And high-definition map information The historical vehicle node features and lane line node features are extracted respectively, and then based on the graph structure data composed of vehicle nodes, lane line nodes and connecting edges, the multi-head self-attention mechanism is applied to perform attention-weighted feature update on the node features. Specifically, it includes the following steps:

[0102] S21. Based on historical movement status Extract historical vehicle node features: The historical trajectory input received by the prediction model passes through the fully connected layer, regularization layer, activation layer, GRU encoding layer, and finally the residual connection module in the ModuleList. The results obtained through the above steps are used as the proxy graph node features of the network, as shown in the formula:

[0103]

[0104]

[0105] in, is the historical features of an agent at time t, encoded as a list of sequential models of a specific agent type: N t The historical features of all agents as output of this module.

[0106] S22. Based on global high-definition map information Extract lane line node features, create lane objects and extract lane center lines through lanelet2 for high-definition map data, determine lane line nodes, and calculate the distance between the intelligent agent and the lane line nodes.

[0107]

[0108]

[0109] in It represents the starting and ending indexes of the centerline of each lane whose distance from agent i is within the threshold at time t.

[0110] S23, the interaction diagram mainly includes the interaction between agents (Agent-Agent) and the implicit interaction between agents and lanes (Agent-Lane). When establishing AA interaction, As the target agent node feature, all other agents within the specified range are embedded in their own node features as neighbor nodes, and a directed edge pointing from the neighbor node to the target node is constructed, and the connection with the neighbor agent is represented by the adjacency matrix. At the same time, the relative position information between nodes is calculated and embedded in the edge attribute variable constructed in the PyTorch_Geometric library to represent the interaction between agent nodes, that is:

[0111]

[0112]

[0113] in, As the set of edge attributes of nodes i and j, the relative position relationship between nodes is embedded, E t It is a collection of edge indexes, edge attributes, and edge types.

[0114] For the implicit interaction between the agent and the lane, that is, AL interaction, the interaction between the agent and the lane is also represented by nodes and edges in the graph structure. And edge features In the case of , filter the index of the target lane line node according to the distance The distances between the corresponding agent nodes and lane line nodes are embedded into the edge attribute variables constructed in the PyTorch_Geometric library to capture the implicit interaction between the agent and the lane.

[0115] S24. According to the above steps, the graph structure data consisting of vehicle nodes, lane line nodes and connection edges is obtained. The node features are updated with attention weighted features by applying the multi-head self-attention mechanism to complete the construction and update of the interaction graph that comprehensively considers multi-source information:

[0116] MH(Q,K,V)=[head1,head2,head3]W

[0117]

[0118] In the multi-head self-attention mechanism, three attention heads are set, Q, K, and V are the query vector, key vector, and value vector respectively. Is a trainable weight matrix. The scaled dot product attention is used to calculate the attention score α:

[0119]

[0120] In step S3, this embodiment constructs a data playback module to store key information of past frames, and captures the changes in the characteristics of the constructed interaction graph nodes in the time series through the time series coding module, which specifically includes the following steps:

[0121] S31, the interaction graph receives the incoming graph structure data and updates the node by aggregating neighbor information (including the interaction between agents and the interaction between agents and map lanes) to complete the self-feature matrix N t During the aggregation update, this embodiment uses the self-attention mechanism between the connected nodes to focus on the dynamic update results of the features over time. It is expressed as:

[0122]

[0123] in, is the linear transformation of the feature matrix of the previous time step, As the attention-weighted feature matrix of the target agent,

[0124] S32. Next, for the obtained feature matrix with attention score weighting, the temporal information is explicitly encoded through the temporal encoding module to obtain the final feature matrix of the target agent at the current time step:

[0125]

[0126] Among them, g0 and g1 are composed of a two-layer MLP network. It is a time series feature fusion module, which encodes the timestamp through a specific linear layer and encodes the time information into a high-dimensional vector. Then the time encoding is repeated to the same dimension as the feature, added to the hidden state and set the feature mask, and then the hidden state is updated through the residual module, and the final output is the node time series feature of the current time step;

[0127] When performing timing diagram encoding in step S32, a key module is the data playback module, which is used to store the state features of the corresponding time step and provide accurate historical sequence information to the timing encoding module. In the present invention, two specific modules are built to track and store the features of the target agent and the scene features respectively.

[0128] S321. In the data playback module for tracking and storing target agent features, for the time features and hidden state of the target agent at time t, there are non-learnable solutions such as taking the latest information or selecting average information. In this embodiment, a learnable solution is selected: LSTM is used to sequentially track the changes in its representation, and the formula is expressed as:

[0129]

[0130] where h t is the hidden state in LSTM, and the input is the feature of the target node. Here, this embodiment only builds a sequential data playback module for the target agent of interest. In interactive prediction, the target agent is alternating. When the target agent of interest is confirmed, the remaining traffic participants participate in the calculation of the interactive timing graph as neighbor agents. Each agent making predictions stores the node features in the data playback module as a target agent.

[0131] S322, in the construction of the scene feature data playback module, at time step t, by performing a feature matrix N t Apply a linear layer to initialize the module, and then build a multi-layer self-attention operation network to store scene features:

[0132]

[0133]

[0134] Among them, g2 is the linear layer for extracting the feature matrix, and l represents the number of layers. The storage of scene features can also ensure that the map features are fully considered when the subsequent decoding module outputs the trajectory.

[0135] For each self-attention layer built in S322, this embodiment performs layer normalization. After the last layer L, the node features are aggregated through the maximum pooling operation to summarize the scene related information The LSTM module is also used to associate information through time to complete the data storage of scene features:

[0136]

[0137] Among them, h t ' represents the hidden state of the map tracked by the module at time t; h t ' -1 Indicates the hidden state of the map at time t-1.

[0138] Step S323: The node state and time code are merged through the time feature fusion module in the time coding module. The module encodes the timestamp through a specific linear layer and encodes the time information into a high-dimensional vector. The time code is then repeated to the same dimension as the feature, and then added to the hidden state and the feature mask is set. The hidden state is updated through the residual module, and finally output as the node time feature of the current time step.

[0139] In step S4, a trajectory decoding module and a score decoding module are constructed to model the probability distribution of multimodal trajectory prediction, which specifically includes the following steps:

[0140] S41. Taking into account the agent's own characteristics, interaction characteristics and map characteristics, the task of trajectory decoding is completed through the GRU network structure, and the coordinates of the trajectory at each time step are finally output:

[0141]

[0142] in, is the agent’s multimodal future decoder, is the concatenation of features in the traffic task, is the predicted future trajectory of the agent’s k-th mode.

[0143] S42, the score decoder follows the same structure as the trajectory decoder, and finally outputs the confidence score of the proposed trajectory, which is further passed through the softmax layer to produce the generation probability distribution of the multimodal trajectory:

[0144]

[0145] In step S5, a loss function is designed based on the trajectory decoding module and the score decoding module to perform model training. For trajectory loss, this embodiment uses a smoothing function on all predicted time steps. Loss. The trajectory regression loss is defined as:

[0146]

[0147] in is the actual trajectory position at time t. This embodiment only calculates the loss between the actual trajectory and the closest predicted trajectory. * is the index in the multimodal predicted trajectory that is closest to the true trajectory position, k * The end point of the trajectory is predicted by calculating the multimodal The end point of the real trajectory The distance between them is:

[0148]

[0149] For the loss function of the predicted score, this embodiment designs a cross entropy loss function to calculate the true score (probability distribution) P gt The loss between the predicted probability distribution P and the predicted score loss is defined as:

[0150]

[0151] Here, this embodiment also needs to define the score distribution corresponding to the real trajectory:

[0152]

[0153] The total loss is the weighted sum of trajectory regression loss and score loss:

[0154]

[0155] Among them, δ is a parameter set to balance the two learning objectives, and this is used as the objective function for model training.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graph, characterized by: It includes the following steps: S1, generating graph structure data based on traffic scene trajectory dataset; S2, construct an interactive graph considering multiple traffic factors based on graph structure data; S3, when the interaction graph updates its features, the temporal coding module is used to capture the changes in the interaction graph node features in the time series; S4, completing multimodal prediction by constructing trajectory decoding module and score decoding module; S5. Design the prediction model loss objective function and perform training.

2. The multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graph according to claim 1 is characterized by: In step S1, the graph structure data is represented as Which is used to build graph structure data The data sources include the historical motion status of the intelligent body. And high-definition map information Among them, i represents the number of agents that consider interactive prediction within a certain range, Represents the coordinate information of the agent at time t and heading angle 3. The multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graph according to claim 1 is characterized by: In step S2, the following steps are included: S21. Based on historical movement status Extract historical vehicle node features; convert historical motion states The vehicle node features are output through the ModuleList model list composed of the fully connected layer, regularization layer, activation layer, GRU encoding layer and residual connection module: in, is the historical characteristics of an agent at time t; Indicates encoding through a list of sequential models of a specific agent type, the superscript at indicates a specific type of agent, and the subscript hist indicates historical information; N t The historical features of all agents as output of this module; S22. Based on global high-definition map information Extract lane line node features; for high-definition map information Use lanelet2 to create a lane object and extract the lane centerline, determine the lane node, and calculate the distance between the agent and the lane node: in It represents the starting and ending indexes of the centerline of each lane whose distance from agent i is within the threshold at time t; S23, establish the interaction between vehicle agents and the interaction between agents and lanes; wherein, for the interaction between vehicle agents, the target agent node feature Other agents within the specified range are embedded as neighbor nodes into their own node features, and directed edges from neighbor nodes to target nodes are constructed. The adjacency matrix is ​​used to represent the connection with neighbor agents. At the same time, the relative position information between nodes is calculated and embedded in the edge attribute variables constructed in the PyTorch_Geometric library to represent the interaction between agent nodes, that is: in, As the set of edge attributes of nodes i and j, the relative position relationship between nodes is embedded, E t It is a collection of edge index, edge attribute and edge type; For the implicit interaction between the vehicle agent and the lane line, the interaction between the agent and the lane is represented by nodes and edges in the graph structure. And edge features In the case of , filter the index of the target lane line node according to the distance The distance between the corresponding agent node and the lane line node is embedded into the edge attribute variable constructed in the PyTorch_Geometric library to capture the implicit interaction between the agent and the lane; S24. By applying the multi-head self-attention mechanism to update the node features with attention weighted features, the construction and update of the interaction graph considering multi-source information can be realized: MH(Q,K,V)=[head1,head2,head3]W head i =α(QW i Q ,KW i K ,VW i V ) In the multi-head self-attention mechanism, three attention heads are set, Q, K, and V are query vectors, key vectors, and value vectors respectively. W,W i Q ,W i K ,W i V is the weight matrix; scaled dot product attention is used to calculate the attention score α: Among them, softmax(·) represents the normalization layer, (·) T represents the matrix transpose, d k Indicates the dimension of the key vector.

4. The multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graph according to claim 1 is characterized by: In step S3, the following steps are included: S31, the target node completes its own feature matrix N by aggregating neighbor information t When updating the aggregation, the self-attention mechanism between the connected nodes is used to focus on the dynamic update results of the features over time. It is expressed as: in, is the linear transformation of the feature matrix of the previous time step, As the attention-weighted feature matrix of the target agent, S32. For the attention weighted feature matrix, the time information is explicitly encoded through the temporal encoding module to obtain the final feature matrix of the target agent at the current time step: Among them, g0 and g1 are composed of a two-layer MLP network. It is a temporal feature fusion module. The temporal feature fusion module encodes the timestamp through a specific linear layer and encodes the time information into a high-dimensional vector. Then the time encoding is repeated to the same as the feature dimension, added to the hidden state and the feature mask is set. Then the hidden state is updated through the residual module, and finally the node temporal features of the current time step are output. Furthermore, the state characteristics of the corresponding time step are stored through the data playback module, and the historical sequence information is provided to the time series encoding module; wherein at least a data playback module is provided for tracking and storing the target agent characteristics and the scene characteristics respectively.

5. The multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graph according to claim 4 is characterized by: In step S32, for the data playback module used to track and store the characteristics of the target agent, the LSTM is used to sequentially track the changes in the representation of the time characteristics and hidden state of the target agent at time t, and the formula is expressed as: where h t is the hidden state of the agent in LSTM, h t-1 is the input target node feature, Represents the feature matrix of the target agent at time t; in interactive prediction, the target agent is alternating. When the target agent of interest is confirmed, the remaining traffic participants participate in the calculation of the interactive timing graph as neighbor agents. Each agent making a prediction stores the node features in the data playback module as a target agent.

6. The multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graph according to claim 5 is characterized by: In step S32, when the scene feature data playback module is built, at time step t, the feature matrix N t Apply a linear layer to initialize the module, and then build a multi-layer self-attention operation network to store scene features: Where g2 is the linear layer for extracting feature matrix, l represents the number of layers, and the storage of scene features can also ensure that the map features are fully considered when the subsequent decoding module outputs the trajectory; For each self-attention layer built, layer normalization is performed. After the last layer L, the node features are aggregated through the maximum pooling operation to summarize the scene-related information. The LSTM module is also used to associate information through time to complete the data storage of scene features: Among them, h t ' represents the hidden state of the map tracked by the module at time t; h t ' -1 Indicates the hidden state of the map at time t-1.

7. The multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graph according to claim 1 is characterized by: In step S4, a trajectory decoding module and a score decoding module are constructed to model the probability distribution of multimodal trajectory prediction, which specifically includes the following steps: S41. Taking into account the agent's own characteristics, interaction characteristics and map characteristics, the task of trajectory decoding is completed through the GRU network structure, and the coordinates of the trajectory at each time step are finally output: in, is the agent’s multimodal future decoder, is the concatenation of features in the traffic task, is the predicted future trajectory of the agent’s kth mode; S42, the score decoder follows the same structure as the trajectory decoder, and finally outputs the confidence score of the proposed trajectory, which is further passed through the softmax layer to produce the generation probability distribution of the multimodal trajectory: Among them, c i represents the trajectory confidence score, Represents the confidence range.

8. The multimodal trajectory prediction method based on spatiotemporal decoupling coding of vehicle interaction graph according to claim 1 is characterized by: In step S5, a loss function is designed according to the trajectory decoding module and the score decoding module to perform model training; For trajectory loss, smoothing is used across all predicted time steps. Loss, the trajectory regression loss is defined as: in is the actual trajectory position at time t, is the predicted trajectory closest to the true trajectory position; k * is the index in the multimodal predicted trajectory that is closest to the true trajectory position, which is calculated by The end point of the real trajectory The distance between them is: Design a cross entropy loss function for the loss function of the predicted score Calculate the true score P gt The loss between the predicted probability distribution P and the predicted score loss is defined as: Define the score distribution corresponding to the true trajectory: The total loss is the weighted sum of trajectory regression loss and score loss: Where δ is a parameter set to balance the two learning objectives.

Images