A pedestrian trajectory prediction method based on a spatio-temporal graph attention network

Abstract

The application provides a pedestrian trajectory prediction method based on a space-time graph attention network, and steps include: collecting pedestrian trajectory data to construct a data set; data preprocessing to extract trajectory features; time attention mechanism to calculate time state features; space attention mechanism to calculate space-time state features; predicting future trajectories and training a prediction model; the method extracts time features of each pedestrian by using the time attention mechanism, considers the influence of different historical time information of each pedestrian on the current prediction result, and effectively improves the accuracy of the prediction result; on the other hand, the space attention mechanism takes the time state features of all pedestrians in the scene extracted by the time attention mechanism in the previous step as input, uses a graph attention neural network to reasonably distribute attention coefficients of adjacent pedestrians of each pedestrian, thereby fusing the feature information of the adjacent pedestrians, simulates the space interaction between pedestrians containing social factors, and ensures the rationality of the prediction result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic driving, and relates to a pedestrian trajectory prediction model establishment method, more particularly to a pedestrian trajectory prediction method based on a space-time graph attention network. BACKGROUND

[0002] In recent years, with the continuous development of artificial intelligence technology, the research in the field of automatic driving is also continuously deepening. The automatic driving system is mainly divided into three modules of environment perception, decision planning and motion control. Through the sensor, the pose information of the road user in the traffic scene is obtained to accurately predict the future motion trajectory, which can improve the rationality and accuracy of the decision planning system. Ensuring the safety of the road user in the traffic scene is a prerequisite for the popularization and application of the automatic driving car, and the pedestrian as a weak road user, through the accurate prediction of the future motion position of the target pedestrian, the occurrence of vehicle and pedestrian collision accidents can be reduced, and the driving safety of the automatic driving car can be improved. In addition, according to the future action trajectory of the pedestrian, the automatic driving system can help to develop a more reasonable driving strategy to solve the problem of traffic congestion. Therefore, the research on the pedestrian trajectory prediction problem has important practical significance for the development of automatic driving technology.

[0003] The task of pedestrian trajectory prediction is to predict the motion position coordinates of the pedestrian at a future time according to the trajectory of the pedestrian in the past period of time. The challenge of pedestrian trajectory prediction lies in the complex and flexible motion of the pedestrian, which is difficult to establish a reasonable dynamic model, and the motion of the pedestrian is affected by the diversity of external environment factors. The existing trajectory prediction methods are mainly divided into two categories according to the modeling method: one category is a model-based method, which depends on a hand function and cannot simulate the interaction of complex scenarios and has poor generalization ability; the other category is a rapidly developing prediction method based on deep learning in recent years. Benefiting from the wide application of neural networks, its complete knowledge system and rich network model provide necessary conditions for improving the accuracy and rationality of pedestrian trajectory prediction. At present, most of the pedestrian trajectory prediction methods only consider the spatial constraints and interactions, and ignore the time continuity, so the prediction accuracy is not enough. SUMMARY

[0004] In view of the problems existing in the prior art, in order to improve the pedestrian trajectory prediction accuracy, the present application provides a pedestrian trajectory prediction method based on a space-time graph attention network.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] A pedestrian trajectory prediction method based on a space-time graph attention network is applied to the field of autonomous driving, analyzes and predicts the behavior of pedestrians crossing the road, and collects pedestrian information using a vehicle-mounted camera equipped with an autonomous vehicle perception system, characterized by the following specific steps:

[0007] Step one, collect pedestrian trajectory data to build a dataset:

[0008] Collect road video data in front of and on both sides of the vehicle during driving using a vehicle-mounted camera equipped with an autonomous vehicle perception system, and use semantic segmentation, image classification, and data labeling and conversion techniques to extract pedestrian information, including the coordinate values of each pedestrian in the image coordinate system in each frame of road video, to build a pedestrian trajectory dataset, which is further divided into training and testing datasets;

[0009] Step two, data preprocessing and trajectory feature extraction:

[0010] Preprocess the pedestrian trajectory training dataset, reasonably sample the input data to reduce computational complexity while ensuring prediction accuracy, extract the historical observation coordinates of each pedestrian, and define the observation coordinates of each pedestrian at historical time t as:

[0011]

[0012] Where i represents the i-th pedestrian in the scene, t represents the historical time, T obs represents the observation time domain length, N represents the total number of pedestrians in the scene, represents the observation coordinate value of the i-th pedestrian in the image coordinate system at historical time t along the x-axis direction and the observation coordinate value along the y-axis direction.

[0013] Define the true coordinates of each pedestrian at prediction time t p as:

[0014]

[0015] Where i represents the i-th pedestrian in the scene, t p represents the prediction time, T pre represents the prediction time domain length, N represents the total number of pedestrians in the scene, represents the true coordinate value of the i-th pedestrian in the image coordinate system at prediction time t p along the x-axis direction and the true coordinate value along the y-axis direction.

[0016] Calculate the historical relative position of each pedestrian at historical time t and the previous historical time t-1 ​

[0017]

[0018]

[0019] where, denotes the historical relative value of the i-th pedestrian along the x-axis direction and the historical relative value along the y-axis direction at the historical time t in the image coordinate system;

[0020] The historical relative position of each pedestrian is embedded by using the embedding function φ(·) to obtain the embedding vector of each pedestrian at the historical time t

[0021]

[0022] where, denotes the embedding vector of the i-th pedestrian at the historical time t and has a dimension of 16, φ(·) denotes the embedding function, and the embedding function is composed of a fully connected layer, W e denotes the learnable fully connected network parameter, the input feature dimension of the network is 2, the output feature dimension is 16, and the number of layers is 1;

[0023] Then, the embedding vector of each pedestrian at the historical time t is input into the long short-term memory network LSTM to obtain the hidden state feature of each pedestrian at the historical time t

[0024]

[0025] where, denotes the hidden state feature of the i-th pedestrian at the historical time t and has a dimension of 32, LSTM(·) is composed of a long short-term memory network unit, W en is a learnable long short-term memory network weight parameter, the input feature dimension of the network is 16, the output feature dimension is 32, the hidden feature dimension is 32, and the number of layers is 1;

[0026] Step three, time attention mechanism is used to calculate the time state feature:

[0027] The time state feature containing time correlation is calculated by the time attention mechanism The time attention coefficient of the hidden state feature of each pedestrian at other historical time r, r∈{1,...,t} to the hidden state feature at the historical time t is calculated by using the time attention mechanism, and the time correlation of the historical trajectory of each pedestrian is extracted, and the specific process is as follows:

[0028] First, the hidden state feature of each pedestrian is input ​​the hidden state feature of the i-th pedestrian at other historical time r the hidden state feature of the i-th pedestrian at other historical time r the time attention coefficient of the i-th pedestrian at other historical time r The calculation process is as follows:

[0029]

[0030]

[0031] wherein f(·) represents a cosine similarity function used to calculate a similarity value, the hidden state feature of the i-th pedestrian at other historical time r, and softmax(·) represents a normalized exponential function, the time attention coefficient of the i-th pedestrian at other historical time r;

[0032] Then, the time attention coefficient of the i-th pedestrian at other historical time r the time state feature of the i-th pedestrian at historical time t The calculation process is as follows:

[0033]

[0034] wherein the time state feature of the i-th pedestrian at historical time t and has a dimension of 32;

[0035] Step four, the spatial attention mechanism calculates the space-time state feature:

[0036] The spatial attention mechanism inputs the time state feature of each pedestrian into a graph attention network, all pedestrians in the scene correspond to nodes in the graph structure, and the interaction between pedestrians corresponds to edges in the graph structure. Based on the graph attention network, the trajectory interaction feature of the i-th pedestrian at historical time t and adjacent pedestrians is fused to obtain a space-time state feature which contains both time correlation and spatial interaction. The specific process is as follows:

[0037] First, define the set of adjacent pedestrians of the i-th pedestrian in the graph structure as N i Input the time state features of all pedestrians into the graph attention network to calculate the spatial attention coefficient of the j-th pedestrian to the i-th pedestrian in the same scene at historical time t

[0038]

[0039] wherein j∈{1,...,N} and j∈N i ​denotes the spatial attention coefficient of the jth pedestrian to the ith pedestrian in the same scene at the historical time t, N i denotes the set of neighboring pedestrians of the ith pedestrian, denotes the temporal state feature of the jth pedestrian at the historical time t, denotes the temporal state feature of any neighboring mth pedestrian of the ith pedestrian at the historical time t, m e {1,...,N} and m e N i , LeakyRelu(·) denotes a nonlinear activation function, a denotes a learnable model parameter, W denotes a learnable node feature transformation weight parameter, and || denotes a concatenation operation;

[0040] Secondly, after obtaining the spatial attention coefficient of the jth pedestrian to the ith pedestrian at the historical time t , the graph attention network is used to calculate the spatio-temporal state feature of the ith pedestrian at the historical time t

[0041]

[0042] wherein, denotes the spatio-temporal state feature of the ith pedestrian at the historical time t and has a dimension of 32, and σ(·) denotes a nonlinear function;

[0043] Step five, predicting the future trajectory, training the prediction model:

[0044] The spatio-temporal state feature and the hidden state feature of the ith pedestrian at the historical time T obs are concatenated, a noise vector z subject to a normal distribution is added to simulate the uncertainty of pedestrian movement in a real scene, and the trajectory decoding feature of the ith pedestrian at the historical time T obs is obtained. The calculation process is as follows:

[0045]

[0046] wherein, z denotes a noise vector and has a dimension of 16, denotes the trajectory decoding feature of the ith pedestrian at the historical time T obs and has a dimension of 80, and || denotes a concatenation operation;

[0047] A decoder Decoder composed of a long short-term memory network is used to calculate the future predicted relative position, the trajectory decoding feature of the ith pedestrian at the historical time T obs is taken as the input of the decoder Decoder, and the trajectory decoding feature of the ith pedestrian at the predicted time T obs+1 trajectory decoding features The calculation process is as follows:

[0048]

[0049] in, This indicates that the i-th pedestrian is at the predicted time T. obs +1 trajectory decoding features with a dimension of 80 This indicates that the i-th pedestrian was at historical time T. obs The embedding vector, W d This represents the learnable network weight parameters;

[0050] The calculation will determine the i-th pedestrian at the predicted time T. obs +1 trajectory decoding features Dimensionality reduction is performed using a fully connected layer to obtain the i-th pedestrian with dimension 2 at prediction time T. obs +1 Predicted relative position

[0051]

[0052] in, This indicates that the i-th pedestrian is at the predicted time T. obs +1 represents the predicted relative position with a dimension of 2, δ(·) represents the fully connected layer network, and W d This represents the learnable network parameters. The network has an input feature dimension of 80, an output feature dimension of 2, and a layer count of 1.

[0053] The i-th pedestrian will be at the predicted time T. obs +1 Predicted relative position With historical moment T obs Observation coordinates Adding them together gives the value of the i-th pedestrian at the predicted time T. obs +1 predicted coordinates The calculation process is as follows:

[0054]

[0055] in, This indicates that the i-th pedestrian is at the predicted time T. obs +1 predicted coordinates The i-th pedestrian is in the image coordinate system at prediction time T. obs +1 Predicted coordinate values ​​along the x-axis and along the y-axis;

[0056] When the i-th pedestrian is obtained at the predicted time T obs After the predicted coordinates of +1, the next predicted time T obsThe predicted coordinates of +2 are calculated by the same method, and the predicted coordinates of each prediction time are calculated by iterative calculation

[0057]

[0058] After obtaining the predicted coordinates of each prediction time of each pedestrian, considering that the generated reasonable pedestrian prediction trajectory may not be only one, the real coordinates of each prediction time of each pedestrian in step two are combined A diversity loss function L is constructed variety A plurality of trajectory samples are generated by sampling, and the sample with the minimum Euclidean distance is calculated as the best prediction trajectory, and the calculation method is as follows:

[0059]

[0060] Wherein, k is a hyperparameter initially set to 20, representing the number of samples generated by random sampling, The predicted coordinates of the i-th pedestrian are represented as The trajectory sample generated by random sampling is represented as L variety The diversity loss function is represented as L.

[0061] The model training of the application is carried out under the pytorch deep learning framework, the Adam optimizer is used for optimization, the learning rate is set to 0.01, the batch size is set to 64, the weight parameters and model parameters of various networks involved in the method are trained by using the training data set, and the diversity loss function L is calculated variety The best prediction trajectory is selected, the corresponding various weight parameters and model parameters are saved, the trained prediction model is obtained, and then the test data set is used to execute the above steps two to five to predict the future trajectory of the pedestrian.

[0062] Compared with the prior art, the application has the following beneficial effects:

[0063] The application discloses a pedestrian trajectory prediction method based on a space-time graph attention network, which models pedestrian interaction and considers time correlation and space interaction, and on the one hand, the existing trajectory prediction method often ignores the time correlation of pedestrians, so that the time attention mechanism is used to extract the time features of each pedestrian, the influence of different historical time information of each pedestrian on the current prediction result is considered, and the accuracy of the prediction result is effectively improved.

[0064] The spatial attention mechanism in another aspect takes the time state features of all pedestrians in the scene extracted by the time attention mechanism in the last step as input, uses a graph attention neural network to fuse the feature information of adjacent pedestrians by giving each pedestrian a reasonable attention coefficient, simulates the spatial interaction between pedestrians containing social factors, and ensures the rationality of the prediction result.

[0065] The application considers the continuity of the spatial interaction of pedestrians, effectively fuses the space-time interaction information, can maximize the use of effective information in the pedestrian trajectory data, and improves the accuracy and rationality of the pedestrian trajectory prediction result. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 A flowchart of the pedestrian trajectory prediction method based on the space-time graph attention network is shown in the figure.

[0067] Figure 2 A principle diagram of the time attention mechanism in step three of the method is shown in the figure. DETAILED DESCRIPTION

[0068] The application will be described in detail below with reference to the accompanying drawings:

[0069] The application provides a pedestrian trajectory prediction method based on a space-time graph attention network, as shown in the figure. Figure 1 The flowchart of the application is shown in the figure, and the specific method steps are as follows:

[0070] Step one, collecting pedestrian trajectory data to construct a data set:

[0071] The vehicle-mounted camera equipped with the automatic driving car perception system collects the road video data in front of and on both sides of the vehicle during driving, and uses semantic segmentation, image classification and data labeling and conversion technical means to extract pedestrian information, including the coordinate value of each pedestrian in each frame of road video in the image coordinate system, constructs a pedestrian trajectory data set, and further divides it into a training data set and a test data set.

[0072] Step two, data preprocessing and trajectory feature extraction:

[0073] The pedestrian trajectory training data set is preprocessed, reasonable sampling is performed on the input data to reduce the calculation amount on the premise of ensuring the prediction accuracy, the historical observation coordinates of each pedestrian are extracted, and the observation coordinates of each pedestrian at the historical time t are defined as :

[0074]

[0075] Wherein, i represents the i-th pedestrian in the scene, t represents the historical time, T obsdenotes the observation time domain length, N denotes the total number of pedestrians in the scene, denotes the observed coordinate value of the i-th pedestrian along the x-axis direction and the observed coordinate value along the y-axis direction at the historical time t in the image coordinate system;

[0076] The true coordinate of each pedestrian at the predicted time t p is defined as:

[0077]

[0078] where i denotes the i-th pedestrian in the scene, t p denotes the predicted time, T pre denotes the predicted time domain length, N denotes the total number of pedestrians in the scene, denotes the true coordinate value of the i-th pedestrian along the x-axis direction and the true coordinate value along the y-axis direction at the predicted time t p in the image coordinate system;

[0079] The frequency of the sampled data frame is 2.5fps, that is, the duration of each frame is 0.4s, the number of observation time domain frames is set to 8 frames, and the number of prediction time domain frames is set to 12 frames, that is, the trajectory information of 3.2s in the past is used to predict the trajectory information of 4.8s in the future;

[0080] The historical relative position of each pedestrian at the historical time t and the previous historical time t-1

[0081]

[0082]

[0083] where, denotes the historical relative value of the i-th pedestrian along the x-axis direction and the historical relative value along the y-axis direction at the historical time t in the image coordinate system;

[0084] The historical relative position is upgraded using an embedding function φ(·) to obtain the embedding vector of each pedestrian at the historical time t

[0085]

[0086] where, denotes the embedding vector of the i-th pedestrian at the historical time t and has a dimension of 16, φ(·) denotes the embedding function, and the embedding function is composed of a fully connected layer, W e denotes the learnable fully connected network parameter, the input feature dimension of the network is 2, the output feature dimension is 16, and the number of layers is 1;

[0087] ​​Then the embedding vector of each pedestrian at the historical time t is input into the long short-term memory network LSTM, and the hidden state feature of each pedestrian at the historical time t is calculated

[0088]

[0089] wherein, represents the hidden state feature of the i-th pedestrian at the historical time t, and the dimension is 32, LSTM(·) is composed of long short-term memory network units, W en is a learnable long short-term memory network weight parameter, the input feature dimension of the network is 16, the output feature dimension is 32, the hidden feature dimension is 32, and the number of layers is 1;

[0090] Step three, time attention mechanism calculates the time state feature:

[0091] The time state feature containing time correlation is calculated by the time attention mechanism The principle diagram of the time attention mechanism is shown in Figure 2 The time attention mechanism is used to calculate the time attention coefficient of the hidden state feature of each pedestrian at other historical time r, r∈{1,...,t} to the hidden state feature at the historical time t, and the time correlation of the historical trajectory of each pedestrian is extracted. The specific process is as follows:

[0092] First, the hidden state feature of each pedestrian is input, and the hidden state feature of the i-th pedestrian at other historical time r is calculated The time attention coefficient of the hidden state feature of the i-th pedestrian at the historical time t is calculated The calculation process is as follows:

[0093]

[0094]

[0095] wherein, f(·) represents a cosine similarity function, which is used to calculate a similarity value, represents the hidden state feature of the i-th pedestrian at other historical time r, and softmax(·) represents a normalized exponential function, represents the time attention coefficient of the i-th pedestrian at other historical time r;

[0096] Then, the time attention coefficient of the i-th pedestrian at other historical time r is used to calculate the time state feature of the i-th pedestrian at the historical time t The calculation process is as follows:

[0097]

[0098] wherein, represents the time state feature of the i-th pedestrian at the historical time t and has a dimension of 32;

[0099] Step four, the spatial attention mechanism calculates the space-time state feature:

[0100] The spatial attention mechanism inputs the time state feature of each pedestrian into the graph attention network, all pedestrians in the scene correspond to nodes in the graph structure, and the interaction between pedestrians corresponds to edges in the graph structure. Based on the graph attention network, the space-time state feature of the i-th pedestrian at the historical time t is obtained by fusing the trajectory interaction features of adjacent pedestrians, which contains both time correlation and spatial interaction. The specific process is as follows:

[0101] Firstly, define the set of adjacent pedestrians of the i-th pedestrian in the graph structure as N i Input the time state features of all pedestrians into the graph attention network to calculate the spatial attention coefficient of the j-th pedestrian to the i-th pedestrian in the same scene at the historical time t

[0102]

[0103] wherein, j∈{1,...,N} and j∈N i , represents the spatial attention coefficient of the j-th pedestrian to the i-th pedestrian in the same scene at the historical time t, N i represents the set of adjacent pedestrians of the i-th pedestrian, represents the time state feature of the j-th pedestrian at the historical time t, represents the time state feature of the m-th pedestrian adjacent to the i-th pedestrian at the historical time t, m∈{1,...,N} and m∈N i , LeakyRelu(·) represents a nonlinear activation function, a represents a learnable model parameter, W represents a learnable node feature transformation weight parameter, and || represents a splicing operation;

[0104] Secondly, after calculating the spatial attention coefficient of the j-th pedestrian to the i-th pedestrian at the historical time t , the graph attention network is used to calculate the space-time state feature of the i-th pedestrian at the historical time t by fusing the spatial interaction features of adjacent pedestrians

[0105]

[0106] wherein, Let represent the spatiotemporal state characteristics of the i-th pedestrian at historical time t with a dimension of 32, and σ(·) represent a nonlinear function;

[0107] Step 5: Predict future trajectories and train the prediction model:

[0108] The i-th pedestrian at historical time T obs spatiotemporal state characteristics and hidden state features To simulate the uncertainty of pedestrian movement in real-world scenarios, a noise vector z following a normal distribution is added during splicing to obtain the data for the i-th pedestrian at historical time T. obs Trajectory decoding features The calculation process is as follows:

[0109]

[0110] Where z represents a noise vector with a dimension of 16. This indicates that the i-th pedestrian was at historical time T. obs The trajectory decoding features are 80 in dimension, and || represents the concatenation operation;

[0111] The decoder, composed of a long short-term memory network, is used to calculate the predicted relative position of the i-th pedestrian at historical time T. obs Trajectory decoding features As input to the decoder, the value of the i-th pedestrian at prediction time T is calculated. obs +1 trajectory decoding features The calculation process is as follows:

[0112]

[0113] in, This indicates that the i-th pedestrian is at the predicted time T. obs +1 trajectory decoding features with a dimension of 80 This indicates that the i-th pedestrian was at historical time T. obs The embedding vector, W d This represents the learnable network weight parameters;

[0114] The calculation will determine the i-th pedestrian at the predicted time T. obs +1 trajectory decoding features Dimensionality reduction is performed using a fully connected layer to obtain the i-th pedestrian with dimension 2 at prediction time T. obs +1 Predicted relative position

[0115]

[0116] in, represents the predicted relative position of the i-th pedestrian at the prediction time T obs +1, and the dimension is 2, δ(·) represents a fully connected layer network, W d represents the learnable network parameters, the input feature dimension of the network is 80, the output feature dimension is 2, and the number of layers is 1;

[0117] the predicted relative position of the i-th pedestrian at the prediction time T obs +1 is added to the observed coordinates of the i-th pedestrian at the historical time T obs the predicted coordinates of the i-th pedestrian at the prediction time T obs +1 are obtained. The calculation process is as follows:

[0118]

[0119] wherein, represents the predicted coordinates of the i-th pedestrian at the prediction time T obs +1, the predicted coordinate value of the i-th pedestrian in the image coordinate system at the prediction time T obs +1 along the x-axis direction and the predicted coordinate value along the y-axis direction;

[0120] After obtaining the predicted coordinates of the i-th pedestrian at the prediction time T obs +1, the predicted coordinates of the next prediction time T obs +2 are calculated by the same method, and the predicted coordinates of each prediction time are calculated by iteration.

[0121]

[0122] After obtaining the predicted coordinates of each pedestrian at each prediction time, considering that there may be more than one reasonable pedestrian prediction trajectory generated, the real coordinates of each pedestrian at each prediction time in step two are combined to construct a diversity loss function L variety , a plurality of trajectory samples are generated by sampling, the sample with the minimum Euclidean distance is calculated as the best prediction trajectory, and the calculation method is as follows:

[0123]

[0124] wherein, k is a hyperparameter initially set to 20, representing the number of samples generated by random sampling, represents the trajectory sample generated by random sampling according to the predicted coordinates of the i-th pedestrian , L variety represents the diversity loss function; ​​

[0125] The model training of the application is carried out under the pytorch deep learning framework, the Adam optimizer is used for optimization, the learning rate is set to 0.01, the batch size is set to 64, the weight parameters and model parameters of various networks involved in the method are trained by using the training data set, and the multi-loss function L variety The best prediction trajectory is selected, the corresponding various weight parameters and model parameters are saved, a trained prediction model is obtained, and then the above steps two to five are executed to predict the future trajectory of the pedestrian.

Claims

1. A pedestrian trajectory prediction method based on a spatiotemporal graph attention network, applied to the field of autonomous driving, analyzes and predicts pedestrian behavior in scenarios of pedestrians crossing the road, and utilizes onboard cameras equipped with the perception system of autonomous vehicles to collect pedestrian information, characterized in that... The specific steps are as follows: Step 1: Collect pedestrian trajectory data to build a dataset: The vehicle-mounted cameras equipped with the autonomous vehicle perception system collect road video data in front of and to both sides of the vehicle during the vehicle's driving process. The pedestrian information is extracted using semantic segmentation, image classification, and data annotation and conversion techniques. This includes the coordinate values ​​of each pedestrian in the image coordinate system in each frame of the road video. A pedestrian trajectory dataset is constructed and further divided into a training dataset and a test dataset. Step 2: Data Preprocessing and Trajectory Extraction feature: The pedestrian trajectory training dataset is preprocessed. To reduce computational load while maintaining prediction accuracy, the input data is sampled appropriately, and the historical observation coordinates of each pedestrian are extracted. The observation coordinates of each pedestrian at historical time t are defined. for: Where i represents the i-th pedestrian in the scene, t represents the historical moment, and T obs The time domain length represents the observation period, and N represents the total number of pedestrians in the scene. This represents the observed coordinates of the i-th pedestrian at historical time t along the x-axis and along the y-axis in the image coordinate system. Define each pedestrian at prediction time t p true coordinates for: Where i represents the i-th pedestrian in the scene, t p Indicates the predicted time, T pre This indicates the prediction time domain length, and N represents the total number of pedestrians in the scene. This indicates that the i-th pedestrian is at prediction time t in the image coordinate system. p The true coordinates along the x-axis and the true coordinates along the y-axis; Calculate the historical relative position of each pedestrian at historical time t and the previous historical time t-1. in, This represents the historical relative value of the i-th pedestrian at historical time t along the x-axis and the historical relative value along the y-axis in the image coordinate system. Using the embedding function φ(·) to determine historical relative positions By increasing the dimensionality, we obtain the embedding vector of each pedestrian at historical time t. in, Let W represent the embedding vector of the i-th pedestrian at historical time t with dimension 16, φ(·) represent the embedding function, which consists of fully connected layers. e This represents the learnable parameters of a fully connected network with an input feature dimension of 2, an output feature dimension of 16, and a layer count of 1. Next, the embedding vector of each pedestrian at historical time t is... The input is fed into a Long Short-Term Memory (LSTM) network to compute the hidden state features of each pedestrian at historical time t. in, The LSTM(·) represents the hidden state features of the i-th pedestrian at historical time t, with a dimension of 32. It is composed of Long Short-Term Memory network units, W en The weight parameters of the learnable Long Short-Term Memory network are: input feature dimension 16, output feature dimension 32, hidden feature dimension 32, and number of layers 1. Step 3: Calculate temporal state features using the temporal attention mechanism: Calculate time-dependent time state features using a time attention mechanism. The temporal attention mechanism is used to calculate the temporal attention coefficient of the hidden state features at other historical moments r, r∈{1,...,t} for each pedestrian at historical moment t, thereby extracting the temporal correlation of each pedestrian's historical trajectory. The specific process is as follows: First, input the hidden state features of each pedestrian. Calculate the hidden state features of the i-th pedestrian at other historical times r. Hidden state features at historical time t Time Attention Coefficient The calculation process is as follows: Where f(·) represents the cosine similarity function, used to calculate the similarity value. The softmax function represents the hidden state features of the i-th pedestrian at other historical times r, and the softmax function represents the normalized exponential function. This represents the temporal attention coefficient of the i-th pedestrian at other historical moments r; Next, the temporal attention coefficients of the i-th pedestrian at other historical moments r are used. Calculate the temporal state features of the i-th pedestrian at historical time t. The calculation process is as follows: in, Let represent the temporal state feature of the i-th pedestrian at historical time t, and its dimension is 32; Step 4: Spatial attention mechanism for calculating spatiotemporal state features: The spatial attention mechanism inputs the temporal state features of each pedestrian into a graph attention network. All pedestrians in the scene correspond to nodes in the graph structure, and the interactions between pedestrians correspond to edges in the graph structure. Based on the graph attention network, the trajectory interaction features of the i-th pedestrian with neighboring pedestrians at historical time t are fused. The resulting spatiotemporal state features include both temporal correlation and spatial interactivity. The specific process is as follows: First, in a graph structure, the set of neighboring pedestrians of the i-th pedestrian is defined as N. i The temporal state features of all pedestrians are input into a graph attention network to calculate the spatial attention coefficient between the j-th pedestrian and the i-th pedestrian in the same scene at historical time t. Where j∈{1,...,N} and j∈N i , N represents the spatial attention coefficient of the j-th pedestrian to the i-th pedestrian in the same scene at historical time t. i Let i represent the set of neighboring pedestrians of the i-th pedestrian. This represents the temporal state characteristics of the j-th pedestrian at historical time t. Let m represent the temporal state characteristics of any adjacent m-th pedestrian of the i-th pedestrian at historical time t, where m∈{1,...,N} and m∈N i LeakyRelu(·) represents a non-linear activation function, a represents learnable model parameters, W represents learnable node feature transformation weight parameters, and || represents the concatenation operation. Secondly, the spatial attention coefficients of the j-th pedestrian to the i-th pedestrian at historical time t are calculated. Then, a graph attention network is used to calculate the spatiotemporal state features of the i-th pedestrian at historical time t, which incorporates the spatial interaction features of neighboring pedestrians. in, Let represent the spatiotemporal state characteristics of the i-th pedestrian at historical time t with a dimension of 32, and σ(·) represent a nonlinear function; Step 5: Predict future trajectories and train the prediction model: The i-th pedestrian at historical time T obs spatiotemporal state characteristics and hidden state features To simulate the uncertainty of pedestrian movement in real-world scenarios, a noise vector z following a normal distribution is added during splicing to obtain the data for the i-th pedestrian at historical time T. obs Trajectory decoding features The calculation process is as follows: Where z represents a noise vector with a dimension of 16. This indicates that the i-th pedestrian was at historical time T. obs The trajectory decoding features are 80 in dimension, and || represents the concatenation operation; The decoder, composed of a long short-term memory network, is used to calculate the predicted relative position of the i-th pedestrian at historical time T. obs Trajectory decoding features As input to the decoder, the value of the i-th pedestrian at prediction time T is calculated. obs +1 trajectory decoding features The calculation process is as follows: in, This indicates that the i-th pedestrian is at the predicted time T. obs +1 trajectory decoding features with a dimension of 80 This indicates that the i-th pedestrian was at historical time T. obs The embedding vector, W d This represents the learnable network weight parameters; The calculation will determine the i-th pedestrian at the predicted time T. obs +1 trajectory decoding features Dimensionality reduction is performed using a fully connected layer to obtain the i-th pedestrian with dimension 2 at prediction time T. obs +1 Predicted relative position in, This indicates that the i-th pedestrian is at the predicted time T. obs +1 represents the predicted relative position with a dimension of 2, δ(·) represents the fully connected layer network, and W d This represents the learnable network parameters. The network has an input feature dimension of 80, an output feature dimension of 2, and a layer count of 1. The i-th pedestrian will be at the predicted time T. obs +1 Predicted relative position With historical moment T obs Observation coordinates Adding them together gives the value of the i-th pedestrian at the predicted time T. obs +1 predicted coordinates The calculation process is as follows: in, This indicates that the i-th pedestrian is at the predicted time T. obs +1 predicted coordinates The i-th pedestrian is in the image coordinate system at prediction time T. obs +1 Predicted coordinate values ​​along the x-axis and along the y-axis; When the i-th pedestrian is obtained at the predicted time T obs After the predicted coordinates of +1, the next predicted time T obs The predicted coordinates at +2 are calculated using the same method. By iterating in this way, the predicted coordinates at each prediction time can be calculated. After obtaining the predicted coordinates of each pedestrian at each predicted time, considering that there may be more than one reasonable pedestrian prediction trajectory, the actual coordinates of each pedestrian at each predicted time in step two are combined. Constructing diverse loss functions L variety Multiple trajectory samples are generated through sampling, and the sample with the smallest Euclidean distance is calculated as the best predicted trajectory. The calculation method is as follows: Where k is a hyperparameter initially set to 20, representing the number of samples generated by random sampling. This indicates the predicted coordinates based on the i-th pedestrian. Trajectory samples generated by random sampling, L variety Representing diverse loss functions; The model training for this method is performed within the PyTorch deep learning framework, using the Adam optimizer with a learning rate of 0.01 and a batch size of 64. The training dataset is used to train the weight parameters and model parameters of various networks involved in the method, and diverse loss functions L are calculated. variety Select the best predicted trajectory, save the corresponding weight parameters and model parameters to obtain the trained prediction model, and then use the test dataset to perform steps two to five above to predict the future trajectory of pedestrians.

Images