A self-supervised pedestrian trajectory prediction method based on velocity endpoint guidance

Abstract

This invention discloses a self-supervised pedestrian trajectory prediction method based on velocity endpoint guidance, capable of predicting multiple possible future trajectories based on the historical trajectories of pedestrians and their neighbors. The method includes the following steps: extracting spatial interaction features and temporal features from past and future trajectories using a graph attention network and a Transformer, respectively; feeding the obtained spatiotemporal features into a conditional variational autoencoder to predict multiple "coarse" endpoints and velocities; clustering these "coarse" endpoints using a clustering algorithm to obtain "refined" endpoints; and feeding the past spatiotemporal features, the "refined" endpoints, and the velocities into a decoder to obtain the final multiple predicted trajectories. This invention fully considers the interactive features of pedestrian trajectories, using a bi-branch feature extraction network to extract spatiotemporal features, and simultaneously predicting multiple endpoints and velocities through a self-supervised task without adding additional annotations, which are then used to guide the final trajectory generation. Furthermore, we cluster the predicted multiple endpoints, preserving multiple potential possibilities while increasing trajectory diversity.

Description

Technical Field

[0001] This invention relates to the fields of intelligent trajectory monitoring, autonomous driving, and intelligent transportation, and particularly to a method for predicting pedestrian trajectories. It can be used as a prediction module in video surveillance, autonomous driving, robotics, and intelligent transportation. Background Technology

[0002] In recent years, with the rapid development of autonomous driving, robotics, and other fields, people's lives are moving towards greater intelligence. Trajectory prediction is of great significance for intelligent systems to understand human behavior. Trajectory prediction is further divided into vehicle trajectory prediction and pedestrian trajectory prediction. Since vehicles travel on fixed tracks, are less affected by environmental interference, and have relatively simple interactions with each other, vehicle trajectory prediction has gradually matured. However, pedestrians, as more common intelligent agents in real-world scenarios, present significant challenges in trajectory prediction due to their susceptibility to environmental influences, high degrees of freedom, complex interactions, and strong randomness. This invention aims to establish a more interpretable and efficient model to obtain richer pedestrian trajectories that better reflect real-world scenarios, facilitating subsequent planning and control in fields such as autonomous driving.

[0003] As highly autonomous agents, pedestrians' trajectories are easily influenced by the environment and other pedestrians, making interaction information crucial for predicting their paths. Traditionally, optimization-based methods, such as Kalman filtering and Gaussian mixture models, have provided reliable solutions for pedestrian trajectory prediction. However, the complexity of manually designed rules and features, along with poor adaptability, has forced a shift in research focus to learning-based methods. Learning-based social interaction modeling methods mainly include Social-Pool, attention mechanisms, and graph neural networks. The Social-Pool method models interactions by aggregating the hidden states of pedestrians within a certain radius; attention mechanisms fuse interaction features by calculating attention scores between pedestrians; and graph neural networks simulate human-to-human interactions through graph structures. While these methods have achieved good results on multiple datasets, they still have some shortcomings. Social-Pool indiscriminately fuses neighbor information without considering weighting; attention mechanisms are insensitive to location perception; and graph neural networks have poor adaptability, with fully connected graphs considering useless interactions, affecting prediction performance. Therefore, designing a more reasonable and efficient social interaction method is a pressing issue that needs to be addressed. Furthermore, due to the randomness of pedestrian movement, there may be multiple potential future trajectories for a pedestrian. Capturing this potential randomness helps machines understand human behavior and better reflect real-world complex scenarios. Current mainstream multi-trajectory generation methods include Generative Adversarial Networks (GANs), Inverse Reinforcement Learning (IRL), and Conditional Variational Autoencoders (CVAs). GANs improve trajectory generation quality through adversarial loss between the generator and discriminator, but their training is unstable and prone to pattern collapse. IRL methods infer reward functions and then sample trajectories, but they are difficult to train and often require many iterations to generate trajectories that match real-world scenarios. CVAs map samples from a simple latent distribution to a trajectory output by incorporating real predicted trajectories into the training; however, because they often use random sampling, the generated trajectories tend to concentrate in the same area, failing to provide sufficient diversity. Therefore, how to generate richer trajectories is an urgent problem to be solved. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a self-supervised pedestrian trajectory prediction method based on velocity endpoint guidance. The technical solution employs a graph attention network and a Transformer to extract the spatial and temporal features of pedestrians, respectively. The graph attention network generates an adjacency matrix in a learnable manner, differentially fusions different interactions through attention, and extracts high-dimensional interaction features, which are more consistent with real-world scenarios and more interpretable. The Transformer overcomes the long-term dependency problem of Long Short-Term Memory networks, making temporal information extraction more efficient. To generate richer and more feasible trajectories, two self-supervised auxiliary tasks are designed based on the conditional variational autoencoder to predict multiple "coarse" endpoints and velocities, which are then used to guide the final trajectory prediction. Furthermore, the predicted multiple "coarse" endpoints are clustered to obtain more refined endpoints. The clustered endpoints not only retain multiple potential possibilities but also exhibit greater diversity. Finally, the endpoint information, velocity information, and historical trajectory information are fed into the decoder to obtain the final predicted trajectory.

[0005] A pedestrian trajectory prediction method based on information interaction, the method comprising the following steps:

[0006] Step (1) Obtain pedestrian trajectory, preprocess the trajectory to obtain past trajectory and future pedestrian trajectories , where x t , y t ∈R 2 , representing the pedestrian's 2D spatial coordinates at time t, T p T represents the length of the observed trajectory. f Indicates the length of the predicted trajectory;

[0007] Step (2) Encodes the spatiotemporal information of past trajectories using a graph attention network and a Transformer to obtain F. past By using graph attention networks and Transformers to encode the spatiotemporal information of future trajectories, F is obtained. future ;

[0008] Step (3) The spatiotemporal features F obtained in step (2) are... past and F future The fusion process yields fused information F, which is then input into a conditional variational autoencoder for encoding to obtain the posterior probability Q(Z|X). t ,Y t ), F past The input is fed into the encoder to obtain the prior probability P(Z|X). t ), the prior probability P(Z|X) t ) and posterior probability Q(Z|X t ,Yt Calculate the KL error to make the two probability distributions closer and obtain the latent distribution Z;

[0009] Step (4) Sample the latent distribution Z from step (3) and sum it with F. past After fusion, the decoder predicts L "coarse" endpoints G. pred and speed V pred ;

[0010] Step (5) converts the L “rough” endpoints G obtained in step (4) into... pred The endpoint refinement module yields M "refined" endpoints G. fine Specifically, the K-means clustering algorithm is used to cluster G. pred By dividing the possibilities, the number of endpoints is increased while ensuring multiple potential outcomes.

[0011] Step (6) obtains feature F from steps (2), (4), and (5). past Speed ​​V pred And the endpoint of "refinement" G fine The input is fed into the trajectory decoder to obtain the final trajectory Y. pred ;

[0012] Step (7) Obtain the prior probability P(Z|X) from step (3). t ) and posterior probability Q(Z|X t ,Y t ) Calculate the error; calculate the velocity V predicted in step (4). pred And the actual speed V gt Calculate the error, where V gt The coordinates G of the "refined" endpoint, predicted in step (5), can be obtained by differentiating Y. fine and the true endpoint coordinates G gt Calculation error, G gt This represents the last coordinate of Y; the trajectory Y predicted in step (6) pred The error is calculated with Y; the probability error, velocity error, endpoint error and trajectory error are summed, and then the final trajectory prediction model is obtained by minimizing the sum of errors to update the parameters.

[0013] The network structure proposed in this invention fully considers the characteristics of pedestrian movement compared to current technologies, resulting in more comprehensive feature extraction and richer, more diverse generated trajectories while preserving prediction diversity. Furthermore, the proposed auxiliary task effectively guides the final trajectory generation without adding extra annotations, leading to more robust results. Attached Figure Description

[0014] Figure 1This is an overall framework diagram of the present invention.

[0015] Figure 2 This is a structural diagram of the spatiotemporal feature extraction module of the present invention.

[0016] Figure 3 This is a structural diagram of the auxiliary task "endpoint speed prediction" module in this invention.

[0017] Figure 4 This is a flowchart of the clustering algorithm of the present invention. Detailed Implementation

[0018] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. The following examples or drawings are used to illustrate the present invention, but are not intended to limit the scope of the invention.

[0019] like Figure 1 As shown, a pedestrian trajectory prediction method based on information interaction specifically includes:

[0020] (1) A two-branch spatiotemporal feature extractor extracts the spatiotemporal features of past and future trajectories respectively. The input data is the spatial coordinates of the pedestrian within a certain time period. , where x t T represents the 2D spatial coordinates of the pedestrian at time t. p Represents the length of the observed trajectory. Pedestrian neighbor coordinate set. ,in The coordinates of the l-th neighbor represent the spatial coordinates, and N represents the number of neighbors. Future pedestrian trajectory coordinates. , where y t ∈R 2 T represents the spatial coordinates of future pedestrians. f This indicates the predicted trajectory length.

[0021] Step (1) as follows Figure 2 As shown, it includes the following steps:

[0022] (1-1) Using a graph attention network to extract spatial interaction features, firstly based on the neighbor coordinate set X N Establish a spatial graph G=(V t U t ),in Represents a set of nodes. Represents the spatial coordinates of the node. Represent the edge set, in which Represents a node and nodes Whether nodes are connected is indicated by a value of 1 if connected and 0 if not. Since there is no prior information about whether nodes are connected, U... tInitialize the graph as an upper triangular matrix padded with 1s, meaning the current state is independent of future states. To increase the graph's sparsity, i.e., eliminate some useless interactions, we use a self-attention mechanism to compute an asymmetric attention score matrix. The specific steps are as follows:

[0023] ,

[0024] ,

[0025] ,

[0026] ,

[0027] in Let E represent the linear transformation, Q and K represent the query and key-value pairs in the self-attention mechanism, W represent the weights, and d represent the embedding dimension. To obtain high-level interaction features, we apply multiple asymmetric convolutions to the rows and columns of R to obtain features F. Then, for each element in F, we pass a threshold function to convert its value to 0 or 1, resulting in matrix M. To ensure node self-connection, we add M to the identity matrix I, and then multiply it by the corresponding elements of R to obtain a sparse adjacency matrix A. The specific process is as follows:

[0028] ,

[0029] ,

[0030] ,

[0031] ,

[0032] ,

[0033] in S represents the number of layers, and S represents the size of the convolutional kernel. This represents the activation function. This represents the Sigmoid activation function. Let f be the threshold hyperparameter. Then we use GCN to extract features from the graph G, obtaining the spatial features f. spa .

[0034] (1-2) Using Transformer to extract temporal features, Transformer provides a more comprehensive model of temporal information compared to Long Short-Term Memory (LSTM) networks. Transformer learns the relationship between pedestrian positions at different times through a self-attention mechanism, thus avoiding long-term dependency issues. Inputting the pedestrian's past trajectory X yields the temporal feature f. temThen, spatial and temporal features are fused to obtain feature F. past The same steps are used to extract the spatiotemporal features of Y to obtain F. future .

[0035] (2) An auxiliary task module that predicts multiple “rough” endpoints and speeds of pedestrian trajectories.

[0036] The specific steps of step (2) are as follows: Figure 2 As shown, it includes:

[0037] The conditional variational autoencoder consists of an encoder and a decoder. The encoder processes the features F obtained in step (1). past Encode the prior probability P(Z|X) to obtain the prior probability. t ), for F past and F future Fusion characteristics Encode to obtain the posterior probability Q(Z|X) t ,Y t Then, KL loss is calculated on the prior and posterior probabilities to make the prior probability P closer to the posterior probability Q. Then, the resulting latent distribution Z is sampled and summed with F. past The data are fed into the decoder to predict L "coarse" endpoints G. pred and speed V pred .

[0038] (3) An endpoint refinement module to refine the “coarse” endpoint.

[0039] The specific steps are as follows: for the "rough" endpoint G obtained in step (2) pred We applied the K-means clustering algorithm to obtain M "refined" endpoints G. fine While ensuring multiple potential possibilities, it increases the diversity of endpoints. The clustering algorithm flowchart is as follows: Figure 4 As shown.

[0040] (4) A trajectory generation module generates the final trajectory.

[0041] The specific steps are as follows: the past trajectory features F obtained in step (1) are... past The velocity V obtained in step (2) pred The endpoint G obtained in step (3) fine The two inputs are fed into the trajectory decoder to generate the final predicted trajectory Y. pred .

[0042] Finally, the loss is calculated, and the model parameters are adjusted using gradient descent. The loss function is as follows:

[0043]

[0044] Among them G gt and V gt G represents the actual destination and velocity, respectively. gt V can be obtained from Y. gt It can be obtained by differentiating Y.

[0045] The above describes the specific steps of the present invention during training. When put into use, the future trajectory encoding part can be removed, and the other parts can be retained.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications, equivalent substitutions, improvements, etc., all of which are included within the protection scope of the present invention.

Claims

1. A self-supervised pedestrian trajectory prediction method based on velocity endpoint guidance, characterized in that, Includes the following steps: Step (1) Obtain pedestrian trajectory, preprocess the trajectory to obtain past trajectory and future pedestrian trajectories , where x t , y t ∈R 2 , representing the pedestrian's 2D spatial coordinates at time t, T p T represents the length of the observed trajectory. f Indicates the length of the predicted trajectory; Step (2) Encodes the spatiotemporal information of past trajectories using a graph attention network and a Transformer to obtain F. past By using graph attention networks and Transformers to encode the spatiotemporal information of future trajectories, F is obtained. future ; Step (3) The spatiotemporal features F obtained in step (2) are... past and F future The fusion process yields fused information F, which is then input into a conditional variational autoencoder for encoding to obtain the posterior probability Q(Z|X). t ,Y t ), F past The input is fed into the encoder to obtain the prior probability P(Z|X). t ), the prior probability P(Z|X) t ) and posterior probability Q(Z|X t ,Y t Calculate the KL error to make the two probability distributions closer and obtain the latent distribution Z; Step (4) Sample the latent distribution Z from step (3) and sum it with F. past After fusion, the decoder predicts L "coarse" endpoints G. pred and speed V pred ; Step (5) converts the L "coarse" endpoints G obtained in step (4) into... pred The endpoint refinement module yields M "refined" endpoints G. fine Specifically, the K-means clustering algorithm is used to cluster G. pred By dividing the possibilities, the number of endpoints is increased while ensuring multiple potential outcomes. Step (6) obtains feature F from steps (2), (4), and (5). past Speed ​​V pred And the endpoint of "refinement" G fine The input is fed into the trajectory decoder to obtain the final trajectory Y. pred ; Step (7) Obtain the prior probability P(Z|X) from step (3). t ) and posterior probability Q(Z|X t ,Y t ) Calculate the error; calculate the velocity V predicted in step (4). pred And the actual speed V gt Calculate the error, where V gt The "refined" endpoint coordinates G, predicted in step (5), can be obtained by differentiating Y. fine and the true endpoint coordinates G gt Calculation error, G gt This represents the last coordinate of Y; the trajectory Y predicted in step (6) pred The error is calculated by summing the probability error, velocity error, endpoint error, and trajectory error. Then, the parameters are updated by minimizing the sum of errors to obtain the final trajectory prediction model.

Images