Pedestrian trajectory prediction method based on spatio-temporal attention and contrastive representation learning

Abstract

This invention relates to the field of pedestrian trajectory prediction technology, specifically disclosing a pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning. The method includes: Step 1: Inputting pedestrian trajectory information, acquiring and organizing pedestrian trajectory data to obtain a dataset; establishing a training set and a test set; the datasets are the benchmark datasets ETH and UCY used for pedestrian trajectory prediction; Step 2: Extracting temporal features and interaction features using a graph attention mechanism; Step 3: Automatically generating positive and negative samples through automatic annotation; Step 4: Performing contrastive representation learning based on the positive and negative samples generated in Step 3; Step 5: Outputting the predicted trajectory through an Info-GAN decoder using the fused pedestrian trajectory feature information; This invention achieves reasonable trajectory prediction, avoids trajectory prediction mode collapse, and reduces the collision rate of the output trajectory while improving the safety and logical consistency of the output trajectory.

Description

Technical Field

[0001] This invention relates to the field of pedestrian trajectory prediction technology, specifically to a pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning. Background Technology

[0002] Due to the disordered and complex nature of pedestrian trajectories, the main difficulties are: (1) Interactions between people are complex and difficult to capture; in particular, pedestrians need to predict the trajectories of other pedestrians in the scene during movement, so as to dynamically adjust their own routes, and this ability is difficult to achieve through modeling. (2) Pedestrian trajectory prediction is a multimodal problem; based on the large number of pedestrian trajectories, multiple possible future trajectories should be generated. Therefore, when outputting trajectories, it is necessary to consider both rationality and diversity. (3) Self-consistency of trajectory output; for trajectory prediction, if the output trajectory will collide, it will not only be logically inconsistent, but may also bring huge risks at the application level, which is usually unacceptable.

[0003] Currently, numerous scholars both domestically and internationally have conducted in-depth research on the pedestrian interaction problem in trajectory prediction. RNN-based methods capture pedestrian motion through their latent states and simulate human-to-human interactions by merging these latent states. Social LSTM is a typical application of recurrent neural networks in pedestrian trajectory prediction, modeling interactions between pedestrians through pooling layers. In recent years, Transformer networks have made groundbreaking progress in Natural Language Processing (NLP). Compared to RNNs, the main advantage of the Transformer architecture is that it significantly improves temporal modeling using self-attention, better capturing potential interactions between pedestrians. However, whether using RNN models or self-attention mechanisms for human-to-human interaction modeling, spatial structure information is often neglected. Graph neural networks, with their unique form, allow for intuitive and effective utilization of spatial structure information in pedestrian interactions. SGCN is a classic application of graph convolutional neural networks in pedestrian trajectory prediction, achieving significant improvements, but still suffers from insufficient extraction of temporal information.

[0004] In summary, the challenge of collision detection in pedestrian trajectory prediction remains. Collisions can lead to model consistency issues and even safety problems. Some collision avoidance methods impose manual constraints on the predictions, but these physical constraints can result in rigid predicted trajectories lacking in plausibility. Many deep learning models address collisions by learning to model pedestrian interactions. While this approach, driven by data-driven learning, can generate reasonable trajectories, it cannot correct collisions because the model is constantly learning correct social interactions and doesn't understand what constitutes an "incorrect situation." Therefore, reducing collisions hinges on addressing the lack of collision samples and selecting the right training method. Summary of the Invention

[0005] The purpose of this invention is to provide a pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning, and to solve the following technical problems:

[0006] (1) How to achieve reasonable trajectory prediction and avoid trajectory prediction mode collapse;

[0007] (2) How to reduce the collision rate of the output trajectory while improving the safety and logical consistency of the output trajectory.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] A pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning, the method comprising:

[0010] Step 1: Input pedestrian trajectory information, obtain and organize pedestrian trajectory data to obtain a dataset; establish training and testing sets; the datasets are the benchmark datasets ETH and UCY datasets used for pedestrian trajectory prediction.

[0011] Step 2: Use graph attention mechanism to extract temporal features and interaction features;

[0012] Step 3: Automatically generate positive and negative samples using automatic labeling methods;

[0013] Step 4: Based on the positive and negative samples generated in Step 3, perform contrastive representation learning;

[0014] Step 5: Output the predicted trajectory by using the Info-GAN decoder to combine the fused pedestrian trajectory feature information.

[0015] Preferably, the problem of predicting the trajectory is expressed as inferring a possible trajectory for a future period based on the trajectory over a past period:

[0016] Assume there are N pedestrians in the scene, and the past trajectories of the pedestrians are defined as follows: The predicted future trajectory of pedestrians is defined as The future true trajectory of a pedestrian is defined as ;

[0017] The problem can be described as follows: given pedestrian i and other pedestrian j in the scene, the historical time period from 1 to... trajectory , The goal is to predict his future time period. arrive Trajectory within a time period That is, it can be expressed as:

[0018]

[0019]

[0020]

[0021]

[0022] Preferably, step two specifically includes:

[0023] Time Feature Extraction: Obtaining Time Trajectory data of N people in the interior scene are extracted to form N time maps; a graph attention mechanism is applied to the trajectory points of pedestrian i.

[0024] Spatial feature extraction: Obtain N people in the scene at each time point t, and form a spatial feature based on the spatial information between the N people. A spatial diagram;

[0025] According to the above A spatial graph is used to draw a graph attention mechanism for the trajectory points of pedestrians i and j at time t.

[0026] Preferably, the method for automatically generating positive and negative samples in step three is as follows:

[0027] The positive and negative samples include positive samples and negative samples; and the positive samples are trajectories (collision-free samples) in ordinary datasets, while the negative samples are collision samples.

[0028] Formula for generating positive samples: ;

[0029] negative sample generation formula: ;

[0030] in, That is, pedestrian i in There is only one positive sample at any given time. For pedestrians i The actual location coordinates at that moment It is a constant, representing random noise;

[0031] That is, pedestrian i in Negative samples at time points; For pedestrian i's neighbor j in The actual location coordinates at that moment, also It is random noise.

[0032] Preferably, the content of the comparative learning in step four is:

[0033] By learning the difference in data distribution between data samples and noise samples, specifically by training a classifier to learn the difference between data samples and noise samples.

[0034] Preferably, after generating the positive and negative samples in step three, a contrastive learning loss function is introduced:

[0035]

[0036] Where q represents the query sample. Represents positive samples. Represents negative samples;

[0037] In the trajectory prediction task, these are the predicted past trajectory vector of pedestrian i, the predicted future true trajectory vector of pedestrian i, and the predicted future true trajectory vector of pedestrian i's neighbor j, respectively; s() is the metric function.

[0038] Preferably, the method for outputting the predicted trajectory in step five is as follows:

[0039] The generator input noise z is divided into two parts (z, c), where z is random noise and c is an interpretable latent variable.

[0040] The loss function for contrastive learning is divided into two parts:

[0041] (1) The loss of noise z is the same as that of GAN;

[0042] (2) Noise c introduces a new mutual information loss. The loss function of info-GAN can be written as:

[0043]

[0044] in, For the generator loss and adversarial loss in the trajectory prediction module, This results in mutual information loss.

[0045] The beneficial effects of this invention are:

[0046] (1) To address the problem of complex and abstract pedestrian interactions and difficulty in feature extraction, a trajectory prediction model based on graph attention mechanism is proposed. This model adopts a method of modeling time features and spatial features separately, and uses graph self-attention technology for trajectory prediction.

[0047] (2) For a long time, the metrics for trajectory prediction tasks have mostly focused on accuracy, that is, the matching rate between the predicted trajectory and the actual trajectory. However, the collision rate is also a particularly critical metric. To this end, this system incorporates a contrastive representation learning module to reduce the collision rate. Furthermore, to address the issue of the scarcity of collision scenarios in real-world datasets, a method that eliminates the need for manual labeling is used to automatically generate positive and negative samples.

[0048] (3) By introducing mutual information loss into the adversarial generative network, the problem of pattern collapse is avoided, which effectively avoids the problem of excessively similar generated trajectories when generating different trajectory prediction modules, and achieves diversification of predicted trajectories.

[0049] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning according to the present invention;

[0052] Figure 2 This is a flowchart illustrating the pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning of the present invention.

[0053] Figure 3 This is a schematic diagram of the spatiotemporal graph attention network structure of the present invention;

[0054] Figure 4 This is a schematic diagram illustrating the automatic acquisition of positive and negative sample pairs in this invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] To address technical issues such as collision problems arising from pedestrian trajectory prediction results, this invention provides a pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning. (See also...) Figure 1-2 As shown, the specific methods include:

[0057] Step 1: Input pedestrian trajectory information, obtain and organize pedestrian trajectory data to obtain a dataset; establish training and testing sets; the datasets are the benchmark datasets ETH and UCY datasets used for pedestrian trajectory prediction.

[0058] Step 2: Use graph attention mechanism to extract temporal features and interaction features;

[0059] Step 3: Automatically generate positive and negative samples using automatic labeling methods;

[0060] Step 4: Based on the positive and negative samples generated in Step 3, perform contrastive representation learning;

[0061] Step 5: Output the predicted trajectory by using the Info-GAN decoder to combine the fused pedestrian trajectory feature information.

[0062] The above technical solution involves five steps: pedestrian trajectory input, social interaction feature extraction, temporal feature extraction, positive and negative sample generation, contrastive representation learning, and trajectory generation. This invention models pedestrian interactions in a scene using a graph attention network and improves the safety and logical consistency of the output trajectory by automatically generating positive and negative samples and reducing the collision rate through contrastive learning. Furthermore, in terms of spatiotemporal attention, the pedestrian trajectory feature information within the scene is represented and preprocessed using a graph, i.e., through an Info-GAN decoder, to construct a graph attention network for the initial extraction of pedestrian trajectory feature information in space, serving as input for subsequent networks.

[0063] In the contrastive representation learning method, positive and negative sample pairs are automatically obtained without manual annotation and then learned through contrastive representation, which reduces the probability of collision between the model output trajectories. In order to simulate the inherent uncertainty and multimodal characteristics of pedestrian movement trajectory, the trajectory generation module of this invention mainly adopts the info-GAN structure and adds mutual information loss to avoid the problem of mode collapse. This effectively avoids the problem of excessively similar generated trajectories when generating different trajectory prediction modules, and achieves diversification of predicted trajectories.

[0064] It's important to note that in Step 1, the trajectory input stage specifically uses the benchmark datasets ETH and UCY for pedestrian trajectory prediction. These two datasets contain a large amount of social interaction data, including 1536 pedestrians and thousands of real-world trajectories. They encompass various real-world scenarios such as pedestrians avoiding obstacles, individual pedestrians walking towards crowds, and pedestrians turning at intersections. These are commonly used benchmark datasets for pedestrian trajectory prediction. Each dataset contains five different crowd datasets: ETH includes the ETH and HOTEL datasets, and UCY includes the ZARA1, ZARA2, and UNIV datasets. During the training or evaluation phase, the input is the trajectory from the past 3.2 seconds, and the output is the trajectory from the next 4.8 seconds.

[0065] As one embodiment of the present invention, the problem of predicting trajectories is expressed as inferring possible trajectories for a future period based on trajectories over a past period:

[0066] Assume there are N pedestrians in the scene, and the past trajectories of the pedestrians are defined as follows: The predicted future trajectory of pedestrians is defined as The future true trajectory of a pedestrian is defined as ;

[0067] The problem can be described as follows: given pedestrian i and other pedestrian j in the scene, the historical time period from 1 to... trajectory , The goal is to predict his future time period. arrive Trajectory within a time period That is, it can be expressed as:

[0068]

[0069]

[0070]

[0071]

[0072] Using the above technical solution, for the coordinates of the i-th pedestrian at time t, we have the following formula: .

[0073] With the above definition, the general statement of this problem is: for each known set of observed pedestrian trajectory sequences: ; and the trajectory sequences of its neighbors: ;

[0074] The network framework extracts and models pedestrian trajectory characteristics from input data to obtain appropriate trajectory feature information and provides reasonable trajectory prediction output within the scene.

[0075] .

[0076] As one embodiment of the present invention, please refer to Figure 2-3 As shown, step two specifically involves:

[0077] Time Feature Extraction: Obtaining Time Trajectory data of N people in the interior scene are extracted to form N time maps; a graph attention mechanism is applied to the trajectory points of pedestrian i.

[0078] Spatial feature extraction: Obtain N people in the scene at each time point t, and form a spatial feature based on the spatial information between the N people. A spatial diagram;

[0079] according to A spatial graph is used to draw a graph attention mechanism for the trajectory points of pedestrians i and j at time t.

[0080] The above technical solution involves a pedestrian interaction feature extraction module that includes social interaction feature extraction and temporal interaction feature extraction. The module uses trajectory data as input and is divided into two parts: temporal feature extraction and interaction feature extraction. These two decoupled spatiotemporal vectors are then fused, and finally, a predicted trajectory is output. The extraction of pedestrian interaction features is divided into two modules: a temporal module and a spatial module. The temporal module considers the past trajectory of pedestrian i, which is a typical time series problem.

[0081] Specifically, time feature extraction: For trajectory data, this includes time... Trajectory data of all N people in the interior scene; extract the trajectory data of these N people to form N time-series graphs.

[0082] in yes The vertex represents pedestrian i during the observation time. The set of all trajectory points per second.

[0083] yes The edges represent the relationship between the trajectory points of pedestrian i at times m and n.

[0084] With this graph, we can implement a graph attention mechanism for the trajectory points of pedestrian i. The specific formula is as follows:

[0085]

[0086] in,

[0087]

[0088] Spatial Feature Extraction: The spatial map stores the interaction information between people in a scene at a certain time t. For trajectory data, there will be N people in the scene at each time point t, and the spatial information between these N people forms... A spatial diagram: ,in , representing all people in the scene at time t. The vertex, Let t represent the connections between these people. The edge.

[0089] With this graph, we can implement a graph attention mechanism for the trajectory points of pedestrians i and j at time t. The specific formula is as follows:

[0090]

[0091] in,

[0092] After applying the self-attention mechanism to the time graph and spatial graph, the nodes in both graphs can be updated separately, using the following formula:

[0093]

[0094] After obtaining the updated temporal and spatial graph node information, they are connected through a fully connected layer to obtain new features that incorporate pedestrian spatiotemporal interaction.

[0095] As one embodiment of the present invention, please refer to Figure 4 As shown, the method for automatically generating positive and negative samples in step three is as follows:

[0096] Positive and negative samples include both positive and negative samples; positive samples are trajectories (collision-free samples) in ordinary datasets, while negative samples are collision samples.

[0097] Formula for generating positive samples: ;

[0098] negative sample generation formula: ;

[0099] in, That is, pedestrian i in There is only one positive sample at any given time. For pedestrians i The actual location coordinates at that moment It is a constant, representing random noise;

[0100] That is, pedestrian i in Negative samples at time points; For pedestrian i's neighbor j in The actual location coordinates at that moment, also It is random noise.

[0101] The above technical solution involves including the positive and negative sample generation part in the trajectory prediction task.

[0102] Positive samples are trajectories (collision-free samples) in ordinary datasets, while negative samples are collision samples. However, since negative samples are rare and manual annotation is too costly, this invention adopts a method for automatically generating positive and negative samples, specifically using the following formula:

[0103] Positive sample generation:

[0104] Negative sample generation:

[0105] in, For pedestrians i There is only one positive sample at any given time. For pedestrians i The actual location coordinates at that moment It is a very small constant, representing random noise.

[0106] in, For pedestrians i There are multiple negative samples at each time step; For pedestrian i's neighbor j in The actual location coordinates at that moment, also It is random noise; This is a range that can be specified; here it is set to 0.5m. 2 .

[0107] In one embodiment of the present invention, the content of the comparative learning in step four is as follows:

[0108] By learning the difference in data distribution between data samples and noise samples, specifically by training a classifier to learn the difference between data samples and noise samples.

[0109] The above technical solution can be understood as a binary classification problem: learning the difference in data distribution between data samples and noise samples, and training a classifier to learn the difference between data samples and noise samples.

[0110] As one embodiment of the present invention, after generating the positive and negative samples in step three, a loss function for contrastive learning is introduced:

[0111]

[0112] Where q represents the query sample. Represents positive samples. Represents negative samples;

[0113] In the trajectory prediction task, these are the predicted past trajectory vector of pedestrian i, the predicted future true trajectory vector of pedestrian i, and the predicted future true trajectory vector of pedestrian i's neighbor j, respectively; s() is the metric function.

[0114] Using the above technical solution, the loss function for contrastive representation learning is as follows:

[0115]

[0116] Where q represents the query sample. Represents positive samples. Representing negative samples. In this invention, these are the predicted past trajectory vector of pedestrian i, the predicted future true trajectory vector of pedestrian i, and the predicted future true trajectory vector of pedestrian i's neighbor j, respectively; s() is the metric function, which is set here as cosine similarity.

[0117] As one embodiment of the present invention, the method for outputting the predicted trajectory in step five is specifically as follows:

[0118] The generator input noise z is divided into two parts (z, c), where z is random noise and c is an interpretable latent variable.

[0119] The loss function for contrastive learning is divided into two parts:

[0120] (1) The loss of noise z is the same as that of GAN;

[0121] (2) Noise c introduces a new mutual information loss. The loss function of info-GAN can be written as:

[0122]

[0123] in, For the generator loss and adversarial loss in the trajectory prediction module, This results in mutual information loss.

[0124] Through the above technical solution, the trajectory generation part of the present invention avoids the problem of pattern collapse by using mutual information loss, effectively avoiding the problem of excessively similar generated trajectories when generating different trajectory prediction modules, and realizing the diversification of predicted trajectories.

[0125] The specific loss function is as follows:

[0126]

[0127] in, For the generator loss and adversarial loss in the trajectory prediction module, This results in mutual information loss.

[0128] Finally, to verify the accuracy of trajectory prediction, this embodiment uses Final Displacement Error (FDE) and Average Displacement Error (ADE). For safety reasons, Collision Rate (CR), the collision rate between predicted trajectories of pedestrians, is also introduced. The formulas are as follows:

[0129]

[0130] Evaluating the accuracy of trajectory prediction based on test data is a crucial part of trajectory prediction systems, and the accuracy obtained from the test data will serve as the result of model evaluation.

[0131] The above description is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined in the claims, they should all fall within the protection scope of the present invention.

Claims

1. A pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning, characterized in that, The method includes: Step 1: Input pedestrian trajectory information, obtain and organize pedestrian trajectory data to obtain a dataset; establish training and testing sets; the datasets are the benchmark datasets ETH and UCY datasets used for pedestrian trajectory prediction. Step 2: Use graph attention mechanism to extract temporal features and interaction features; Step 3: Automatically generate positive and negative samples using automatic labeling methods; Step 4: Based on the positive and negative samples generated in Step 3, perform contrastive representation learning; Step 5: Output the predicted trajectory by passing the fused pedestrian trajectory feature information through the Info-GAN decoder. The positive and negative samples include positive samples and negative samples; and the positive samples are trajectories (collision-free samples) in ordinary datasets, while the negative samples are collision samples. Formula for generating positive samples: ; negative sample generation formula: ; in, That is, pedestrian i in There is only one positive sample at any given time. For pedestrians i The actual location coordinates at that moment It is a constant, representing random noise; That is, pedestrian i in Negative samples at time points; For pedestrian i's neighbor j in The actual location coordinates at that moment, also It is random noise; The content of the comparative learning in step four is as follows: By learning the difference in data distribution between data samples and noise samples, specifically by training a classifier to learn the difference between data samples and noise samples; After generating the positive and negative samples in step three, a loss function for contrastive learning is introduced: Where q represents the query sample. Represents positive samples. Represents negative samples; Furthermore, in the trajectory prediction task, these represent the predicted past trajectory vector of pedestrian i, the predicted future true trajectory vector of pedestrian i, and the predicted future true trajectory vector of pedestrian i's neighbor j; s() is the metric function; The specific method for outputting the predicted trajectory in step five is as follows: The generator input noise z is divided into two parts (z, c), where z is random noise and c is an interpretable latent variable. The loss function for contrastive learning is divided into two parts: (1) The loss of noise z is the same as that of GAN; (2) Noise c introduces a new mutual information loss. The loss function of info-GAN can be written as: in, For the generator loss and adversarial loss in the trajectory prediction module, This results in mutual information loss.

2. The pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning according to claim 1, characterized in that, The problem of predicting trajectories can be expressed as inferring possible trajectories for a future period based on trajectories over a past period: Assume there are N pedestrians in the scene, and the past trajectories of the pedestrians are defined as follows: The predicted future trajectory of pedestrians is defined as The future true trajectory of a pedestrian is defined as ; The problem can be described as follows: given pedestrian i and other pedestrian j in the scene, the historical time period from 1 to... trajectory , The goal is to predict his future time period. arrive Trajectory within a time period That is, it can be expressed as: ， 。 3. The pedestrian trajectory prediction method based on spatiotemporal attention and contrastive representation learning according to claim 2, characterized in that, Step two specifically involves: Time Feature Extraction: Obtaining Time Trajectory data of N people in the interior scene are extracted to form N time maps; a graph attention mechanism is applied to the trajectory points of pedestrian i. Spatial feature extraction: Obtain N people in the scene at each time point t, and form a spatial feature based on the spatial information between the N people. A spatial diagram; According to the above A spatial graph is used to draw a graph attention mechanism for the trajectory points of pedestrians i and j at time t.

Images