A method for multi-entity trajectory prediction based on space-time graph and attention mechanism

Abstract

The application provides a kind of method for multi-entity trajectory prediction based on space-time graph and attention mechanism, it is related to trajectory prediction technical field.Utilize the historical trajectory of ground entity as input to deduce the future trajectory of entity in scene;First, the historical trajectory position is encoded into feature space by trajectory embedding, so as to obtain input graph representation, then it is sent into model processing, finally deduces the future trajectory of all entities in scene.The application solves the problem of trajectory prediction of multiple entities in the same scene, uses the historical trajectory of entities in the scene as input, deduces the future trajectory of entities from the two dimensions of time and space, analyzes the trajectory of multiple entities by using graph and attention mechanism, extracts effective interaction vector, and realizes accurate prediction of the future trajectory of multiple entities.

Description

Technical Field

[0001] This invention relates to the field of trajectory prediction technology, and in particular to a method for multi-entity trajectory prediction based on spatiotemporal graphs and attention mechanisms. Background Technology

[0002] Trajectory prediction refers to predicting possible future movement trajectories based on past movement trajectory information. Spatiotemporal data extraction is a crucial step in this problem; it involves extracting useful spatiotemporal information from known trajectory data to facilitate trajectory prediction. Time dependence and spatial interaction are two important concepts, representing the temporal and spatial correlations in trajectory prediction, respectively. Understanding these two concepts is essential for achieving accurate trajectory prediction models. The two dimensions of information—time and space—are inherent characteristics of trajectory prediction problems. On the one hand, as a typical time series problem, a realistic trajectory conforming to social norms cannot be separated from historical trajectories; that is, historical trajectories contain the characteristics of entity movement. On the other hand, spatial interactions between different entities genuinely influence each other's future trajectories. Although such spatial interactions are difficult to model, this modeling process is indispensable for accurate prediction models.

[0003] In real-world scenarios, time dependence and spatial interactivity often coexist. For example, in pedestrian trajectory prediction, algorithms need to consider the historical trajectory data of pedestrians, as well as the positions and speeds of other pedestrians in the vicinity. This is precisely the challenge of trajectory prediction. To address this issue, current research generally adopts deep learning-based trajectory prediction methods, such as Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs). RNNs are a type of recurrent neural network that takes sequential data as input, recursively processes data in the direction of sequence evolution, and connects all nodes (recurrent units) in a chain-like manner. Long Short-Term Memory (LSTM), a variant of RNN, processes input data in a gating manner. Its built-in storage units can temporarily store previous input information. Trajectories at different time steps are fed into the LSTM sequentially, and the model integrates previous trajectory positions when processing the current input data. This characteristic is crucial for trajectory prediction. Convolutional Neural Networks treat all entities in a scene as vertices in a graph, and the relationships between entities as edges. Numerous scholars both domestically and internationally have explored the application of graph neural networks and their various variants in trajectory prediction. Some existing methods can effectively capture the temporal dependence and spatial interaction in trajectory data and have achieved good trajectory prediction results.

[0004] Trajectory prediction algorithms are crucial for fields such as autonomous driving and autonomous robot navigation. Accurate trajectory prediction algorithms are essential for the normal and stable operation of these systems. From a macroscopic perspective, the trajectory prediction problem can be understood as a time series problem: analyzing input data, extracting its features, and then producing output data that conforms to a certain pattern, aiming to reflect the development trend of the time series task as accurately as possible. This requires the model to fully learn the distribution of the data itself, achieving both effectiveness and accuracy—these two aspects are the two major challenges of trajectory prediction.

[0005] Most existing methods comprehensively consider spatial interactions and temporal dependencies in the prediction, and the methods themselves vary. Trajectory prediction methods based on recurrent neural networks have a large number of parameters, and due to inherent problems such as gradient explosion and gradient vanishing, the training process converges slowly. Trajectory prediction methods based on convolutional neural networks focus on modeling the mutual influence of entities in the scene, using operations such as pooling to characterize this, resulting in high computational cost and limited model inference speed. Furthermore, because entities have potential intentions, their future trajectories are uncertain, and current methods rarely perform in-depth analysis of these potential intentions, leading to less accurate predicted trajectories. Summary of the Invention

[0006] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a method for multi-entity trajectory prediction based on spatiotemporal graphs and attention mechanisms. This method solves the problem of trajectory prediction for multiple entities in the same scene. It takes the historical trajectories of entities in the scene as input, infers the future trajectories of entities from both time and space dimensions, and uses graphs and attention mechanisms to analyze the trajectories of multiple entities, extracts effective interaction vectors, and achieves accurate prediction of the future trajectories of multiple entities.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A multi-entity trajectory prediction method based on spatiotemporal graphs and attention mechanisms uses the historical trajectories of ground entities as input to infer the future trajectories of entities within a scene. First, the positions of historical trajectories are encoded into a feature space through trajectory embedding to obtain an input graph representation. This representation is then fed into a model for processing, ultimately inferring the future trajectories of all entities in the scene. The method includes the following steps:

[0009] Step 1: Embed the historical trajectory of the target entity to obtain the embedding vector;

[0010] Step 2: Construct a spatial relationship connection diagram;

[0011] Step 3: Obtain the weighted adjacency matrix of the graph based on the spatial relationship connection graph;

[0012] Step 4: Use Graph Convolutional Network (GCN) to extract spatial interaction features and obtain spatial interaction feature vectors;

[0013] Step 5: Feed the spatial interaction feature set into the multi-head self-attention mechanism to output the fused interaction features at different times;

[0014] Step 6: Embed the historical trajectory information of the target entity at this moment and the spatial interaction influence of the surrounding entities at the current moment into a single vector;

[0015] Step 7: Input the vector obtained in Step 6 into the Temporal Convolutional Network (TCN) to obtain motion features at future time steps;

[0016] Step 8: Calculate the two-dimensional Gaussian distribution satisfied by the entity position based on the motion characteristics;

[0017] Step 9: Sample the two-dimensional Gaussian distribution to obtain the future trajectory of the entity in the scene.

[0018] Further, in step 4, in GCN, the feature vector of each node is updated by convolving with the feature vectors of its neighboring nodes; the essence of this convolution operation is to perform a weighted summation of the features of the node's neighboring nodes, where the weights represent the importance of the neighboring nodes to the target node; the convolution operation in GCN is expressed as:

[0019]

[0020] Among them, H (l) This represents the feature vector of the node in the l-th layer. This represents the adjacency matrix with its self-loops normalized. The degree matrix is ​​represented by the normalization of self-loops, W. (l) Let represent the weight matrix of the l-th layer, and σ(·) represent the activation function.

[0021] Furthermore, the activation function is PreLU, and its functional form is shown in formula (2):

[0022]

[0023] Furthermore, in step 5, the self-attention mechanism is used to distinguish the different degrees of influence of each surrounding entity, analyze the complex spatial interactions between entities, selectively capture useful interaction features while ignoring useless interaction information, and thus obtain the comprehensive influence of surrounding entities on the target entity.

[0024] Furthermore, in step 5, different spatial interaction features are selectively distinguished. For the set of spatial interaction features, Query, Key, and Value are adaptively calculated through a self-attention mechanism. The attention weight is regarded as a weighted average of each position in the input sequence, accurately capturing the importance of different positions in the sequence and the relationship between them, filtering important interaction information at different times, and outputting the fused interaction features at different times.

[0025] Furthermore, the specific method of step 5 is as follows:

[0026] Step 5.1: Calculate Query, Key, and Value based on the input, as shown in Equations (3), (4), and (5) respectively;

[0027]

[0028]

[0029]

[0030] Among them, I t This represents the obtained set of spatial interaction features. and These are learnable weight matrices used to map the input sequence to the query vector, key vector, and value vector of the i-th head;

[0031] Step 5.2: Introduce softmax to convert the attention score into a probability distribution, as shown in Equation (6), so that the weight of each position is between 0 and 1, and the total weight is 1. The attention weight is regarded as a weighted average of each position in the input sequence.

[0032]

[0033] Where, d k The dimension of the key vector. This is to scale the attention score;

[0034] Step 5.3: Connect the output vectors of all heads obtained in Step 5.2 along the last dimension to obtain the final output vector of the multi-head self-attention mechanism.

[0035] Furthermore, in step 7, the future trajectories of entities in the scene are inferred by using the causal convolution and dilated convolution of the temporal convolutional network TCN to process the various feature inputs.

[0036] The beneficial effects of adopting the above technical solution are as follows: The multi-entity trajectory prediction method based on spatiotemporal graphs and attention mechanisms provided by this invention uses graph convolution and attention mechanisms to obtain the interaction features between multiple entities and their own interaction features. This can capture long-distance spatiotemporal relationships, improving the accuracy and robustness of prediction. This method achieves the best average prediction error result and performs best on most datasets. On the average results of the five subsets of the ETH / UCY dataset, compared with the lowest error achieved by the current conventional trajectory prediction algorithms STGAT, Stgcnn, and S-LSTM, this method reduces the error by 14.65% on ADE and 9.33% on FDE. Attached Figure Description

[0037] Figure 1 The main flowchart provided for embodiments of the present invention;

[0038] Figure 2 This is an overall structural diagram provided for an embodiment of the present invention;

[0039] Figure 3 A process diagram of GCN aggregating neighbor node features provided in an embodiment of the present invention;

[0040] Figure 4 This is an overall structural diagram of the entity spatial interaction feature fusion part based on the self-attention mechanism provided in an embodiment of the present invention. Detailed Implementation

[0041] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0042] like Figure 1 As shown, the method of this embodiment is described below.

[0043] A multi-entity trajectory prediction method based on spatiotemporal graphs and attention mechanisms uses the historical trajectories of ground entities as input to infer the future trajectories of entities within a scene. First, the positions of historical trajectories are encoded into a feature space through trajectory embedding to obtain an input graph representation. This representation is then fed into a model for processing, ultimately inferring the future trajectories of all entities in the scene. The method includes the following steps:

[0044] Step 1: Embed the historical trajectory of the target entity to obtain the embedding vector;

[0045] Step 2: Construct a spatial relationship connection diagram;

[0046] Step 3: Obtain the weighted adjacency matrix of the graph based on the spatial relationship connection graph;

[0047] Step 4: Use Graph Convolutional Network (GCN) to extract spatial interaction features and obtain spatial interaction feature vectors;

[0048] Step 5: Feed the spatial interaction feature set into the multi-head self-attention mechanism to output the fused interaction features at different times;

[0049] Step 6: Embed the historical trajectory information of the target entity at this moment and the spatial interaction influence of the surrounding entities at the current moment into a single vector;

[0050] Step 7: Input the vector obtained in Step 6 into the TCN to obtain the motion features at future time steps;

[0051] Step 8: Calculate the two-dimensional Gaussian distribution satisfied by the entity position based on the motion characteristics;

[0052] Step 9: Sample the two-dimensional Gaussian distribution to obtain the future trajectories of entities in the scene.

[0053] The overall model structure diagram of the method is as follows: Figure 2 As shown below, the multi-entity trajectory prediction method of this embodiment will be described in further detail from the three main parts of the method.

[0054] (1) Entity interaction feature extraction based on graph convolution.

[0055] Based on the spatial location of entities in the scene, a spatial relationship graph is constructed, and the adjacency relationship between entities at different times is constructed. The location encoding information of entities is propagated using a graph convolutional network (GCN) to mine the spatial interaction between entities at the current time, i.e., the spatial interaction feature vector.

[0056] Graph Convolutional Networks (GCNs) are deep learning algorithms used to process graph-structured data. Unlike traditional Convolutional Neural Networks (CNNs), which process grid-structured data (such as images), GCNs can handle data with arbitrary topological structures, such as social networks, protein molecular structures, and traffic flow. GCNs are neural networks that perform convolution operations on graph-structured data. Unlike traditional CNNs, where convolution operations are defined in Euclidean space and therefore can only be applied to data with regular structures like images, GCNs can perform convolution operations. For example, in social networks, GCNs can identify community structure and node importance, thereby recommending friends or promoting products; in protein molecular structures, GCNs can predict protein properties and functions, thus accelerating new drug development; and in traffic flow, GCNs can predict congestion and optimize traffic planning, thereby improving urban traffic efficiency. In GCN, the convolutional kernels are not fixed like those in traditional CNNs, but are dynamically generated based on the adjacency matrix. In fact, GCN achieves message propagation between first-order neighbor nodes through a propagation formula. That is, from a spatial domain perspective, the essence of GCN is aggregating neighbor information. For any node in the graph, each update of the node's features aggregates information from higher-order neighbor nodes. Therefore, multi-layer GCNs can achieve message propagation between higher-order neighbors. The method in this embodiment uses GCN to process the obtained spatial relationship connection graph to extract spatial interaction features between entities.

[0057] In GCN, each node's feature vector is updated by convolving it with the feature vectors of its neighbors. This convolution operation essentially performs a weighted summation of the features from the node's neighbors, where the weights represent the importance of the neighboring nodes to the target node. The process of GCN aggregating neighbor features is as follows: Figure 3 As shown.

[0058] The convolution operation in GCN is represented as:

[0059]

[0060] Among them, H (l) This represents the feature vector of the node in the l-th layer. This represents the adjacency matrix with its self-loops normalized. The degree matrix is ​​represented by the normalization of self-loops, W. (l) Let represent the weight matrix of the l-th layer, and σ(·) represent the activation function.

[0061] Activation functions play a crucial role in deep learning. They enable neural networks to handle nonlinear problems, improve model expressiveness, alleviate the vanishing gradient problem, and limit the output range. Activation functions are diverse and adjustable, allowing neurons to output nonlinear activation values, enabling the network to learn more complex features. Furthermore, activation functions can address the vanishing gradient problem, a common issue in deep neural networks. Using activation functions like ReLU can effectively alleviate this problem, allowing deeper networks to be trained more effectively. PReLU (Parametric Rectified Linear Unit) is an improved ReLU activation function that introduces a learnable parameter on the negative half-axis, further enhancing model performance. PReLU has more parameters, which can, to some extent, avoid the "death" of neurons in ReLU when the input is negative, preventing gradient updates and effectively improving model expressiveness. Moreover, the parameter α of PReLU is learnable and can be dynamically adjusted based on data, further improving the model's generalization ability. Therefore, this embodiment uses PReLU as the activation function in the graph convolution process. The functional form of PReLU is shown in formula (2).

[0062]

[0063] (2) The fusion of entity spatial interaction features based on self-attention mechanism.

[0064] By combining the spatial interaction feature vectors between entities obtained by the feature extraction module, a self-attention mechanism is used to capture the different degrees of influence of different surrounding entities on the motion of the target entity, thereby filtering out information useful for future trajectory inference.

[0065] Self-attention mechanisms calculate a weight vector by comparing the similarity of each position to all other positions, and then use this weight vector as a weighted sum of the input sequences. The similarity calculation is typically implemented using dot products or other forms of matrix multiplication. The core idea of ​​self-attention is that when calculating the weight of each position, only the relationship between that position and other positions is considered, regardless of the positional order in the sequence, thus achieving position-independent feature extraction.

[0066] In real-world scenarios, an entity's trajectory is influenced by its surrounding entities. However, the influence of each surrounding entity is not uniform; entities at different distances, speeds, and directions have varying impacts on the target entity's motion. In other words, the interaction effects between entities differ. Therefore, to obtain more interpretable and accurate future trajectory inferences, this embodiment utilizes a self-attention mechanism to differentiate between these varying degrees of influence. It attempts to analyze complex spatial interactions between entities through this mechanism, allowing the model to selectively capture useful interaction features while ignoring useless interaction information, thereby obtaining the comprehensive influence of surrounding entities on the target entity.

[0067] For the spatial interaction feature set obtained using graph convolution, a self-attention mechanism adaptively filters important interaction information at different time steps, outputting fused interaction features from different time steps. Different spatial interaction features are selectively distinguished. For the previously obtained spatial interaction feature set, a self-attention mechanism adaptively filters important interaction information at different time steps, outputting fused interaction features from different time steps. For example... Figure 4 As shown, the calculation process is divided into three stages:

[0068] Step 5.1: Calculate the query, key, and value based on the input, as shown in equations (3), (4), and (5) respectively.

[0069]

[0070]

[0071]

[0072] Among them, I t This represents the obtained set of spatial interaction features. and These are learnable weight matrices used to map the input sequence to the query vector, key vector, and value vector of the i-th head.

[0073] Step 5.2: Introduce softmax to convert the attention score into a probability distribution, as shown in Equation (6), so that the weight of each position is between 0 and 1, and the sum is 1. In this way, the attention weight can be regarded as a weighted average of each position in the input sequence, thereby more accurately capturing the importance of different positions in the sequence and the relationship between them.

[0074]

[0075] Where, d k The dimension of the key vector. This is to scale the size of the attention score.

[0076] Step 5.3: Connect the output vectors of all heads obtained in Step 5.2 along the last dimension to obtain the final output vector of the multi-head self-attention mechanism. This vector is the fusion interaction feature at the current moment, which integrates the different spatial interaction effects at all observation moments and more accurately reflects the common spatial interaction influence of all surrounding entities on the target entity at the current moment.

[0077] (3) Future trajectory inference based on temporal convolutional network TCN.

[0078] The module takes the entity motion state information obtained at different times and the historical trajectory of the target entity as input, infers the future trajectory of the target entity from the time dimension, and generates the final discrete position.

[0079] Temporal Convolutional Networks (TCNs) are deep learning models used for sequence modeling. Their core idea is to process time series data using a series of convolutional layers. TCNs employ a fully convolutional architecture, with each layer being a one-dimensional convolutional layer, and dilated convolutions are used between layers to increase the receptive field. Within each convolutional layer, TCNs utilize residual connections to improve model performance and batch normalization (BN) to accelerate training. Furthermore, TCNs introduce a causal convolutional module to ensure that the model's predictions of future data rely only on past data, thus avoiding information leakage. In this embodiment, TCNs combine the fused interaction features from each observation time point with the historical trajectory of the target entity to infer the future trajectory of the target entity, completing the prediction task.

[0080] Trajectory prediction problems typically involve two dimensions: time and space. In the time dimension, trajectory prediction requires predicting the future position of a target, usually considering information from multiple time steps. In the spatial dimension, it requires predicting the target's position at a future point in time, typically considering the target's current position, as well as surrounding environmental and spatial interaction information. This part models the temporal dependencies of trajectory prediction, combining the fused interaction features from various observation moments with the target entity's historical trajectory to infer the target entity's future trajectory and complete the prediction task.

[0081] The performance of the proposed method was validated on the ETH / UCY dataset and compared with current conventional trajectory prediction algorithms such as STGAT, Stgcnn, and S-LSTM. Each method observed the historical trajectory of an entity for 8 time steps (3.2 seconds) and predicted the future trajectory for 12 time steps (4.8 seconds). The prediction results are shown in Table 1.

[0082] The data in Table 1 are in meters and represent the results of the ADE / FDE evaluation index; the smaller the value, the better. On the average results across the five subsets of the ETH / UCY dataset, the method in this embodiment achieves a 14.65% reduction in ADE and a 9.33% reduction in FDE compared to the lowest error achieved by current conventional trajectory prediction algorithms such as STGAT, Stgcnn, and S-LSTM.

[0083] Table 1 Comparison of prediction results of different methods on the ETH / UCY dataset.

[0084]

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for multi-entity trajectory prediction based on spatiotemporal graphs and attention mechanisms, characterized in that: The method uses the historical trajectories of ground entities as input to infer the future trajectories of entities within the scene. First, the positions of historical trajectories are encoded into a feature space through trajectory embedding to obtain an input graph representation. This representation is then fed into a model for processing, ultimately inferring the future trajectories of all entities in the scene. Specifically, the method includes the following steps: Step 1: Embed the historical trajectory of the target entity to obtain the embedding vector; Step 2: Construct a spatial relationship connection diagram; Step 3: Obtain the weighted adjacency matrix of the graph based on the spatial relationship connection graph; Step 4: Use Graph Convolutional Network (GCN) to extract spatial interaction features and obtain spatial interaction feature vectors; Step 5: Feed the spatial interaction feature set into the multi-head self-attention mechanism to output the fused interaction features at different times; By utilizing a self-attention mechanism to differentiate the varying degrees of influence of each surrounding entity, complex spatial interactions between entities are analyzed. For a set of spatial interaction features, the query, key, and value are adaptively calculated through the self-attention mechanism. The attention weight is regarded as a weighted average of each position in the input sequence, accurately capturing the importance of different positions in the sequence and their interrelationships. Different spatial interaction features are selectively differentiated, important interaction information at different times is filtered out, useless interaction information is ignored, and useful interaction features are selectively captured. This yields the comprehensive influence of surrounding entities on the target entity and outputs the fused interaction features at different times. Step 6: Embed the historical trajectory information of the target entity at this moment and the spatial interaction influence of the surrounding entities at the current moment into a single vector; Step 7: Input the vector obtained in Step 6 into the Temporal Convolutional Network (TCN) to obtain motion features at future time steps; Step 8: Calculate the two-dimensional Gaussian distribution satisfied by the entity position based on the motion characteristics; Step 9: Sample the two-dimensional Gaussian distribution to obtain the future trajectory of the entity in the scene.

2. The method for multi-entity trajectory prediction based on spatiotemporal graphs and attention mechanisms according to claim 1, characterized in that: In step 4, in GCN, the feature vector of each node is updated by convolving with the feature vectors of its neighbors. This convolution operation essentially involves a weighted sum of the features of the node's neighbors, where the weights represent the importance of the neighboring nodes to the target node. The convolution operation in GCN is represented as follows: (1) in, Indicates the first Layer node feature vectors This represents the adjacency matrix with its self-loops normalized. The degree matrix is ​​represented by the normalization of self-loops. Indicates the first The weight matrix of the layer, This represents the activation function.

3. The method for multi-entity trajectory prediction based on spatiotemporal graphs and attention mechanisms according to claim 2, characterized in that: The activation function is PreLU, and its function form is shown in formula (2): 。 4. The method for multi-entity trajectory prediction based on spatiotemporal graphs and attention mechanisms according to claim 1, characterized in that: The specific method for step 5 is as follows: Step 5.1: Calculate Query, Key, and Value based on the input, as shown in Equations (3), (4), and (5) respectively; (3) (4) (5) in, This represents the obtained set of spatial interaction features. , and These are learnable weight matrices used to map the input sequence to the query vector, key vector, and value vector of the i-th head; Step 5.2: Introduce softmax to convert the attention score into a probability distribution, as shown in Equation (6), so that the weight of each position is between 0 and 1, and the total weight is 1. The attention weight is regarded as a weighted average of each position in the input sequence. (6) in, The dimension of the key vector. This is to scale the attention score; Step 5.3: Connect the output vectors of all heads obtained in Step 5.2 along the last dimension to obtain the final output vector of the multi-head self-attention mechanism.

5. The method for multi-entity trajectory prediction based on spatiotemporal graphs and attention mechanisms according to claim 1, characterized in that: In step 7, the causal convolution and dilated convolution of the temporal convolutional network TCN are used to process the feature inputs of each part and infer the future trajectory of the entity in the scene.

Images