Pedestrian trajectory prediction method and device based on graph attention mechanism

Abstract

The invention provides a pedestrian trajectory prediction method and device based on a graph attention mechanism. The pedestrian trajectory prediction method comprises the steps of obtaining traffic data and executing data preprocessing to obtain pedestrian data; enabling the pedestrian data to sequentially pass through a self-attention layer and a long-short-term memory network to generate space-time variables; inputting the time-space variable into a graph attention network to generate and output pedestrian data with social information; generating spatial features with environmental information by using the pedestrian data; and generating a predicted pedestrian trajectory by using the pedestrian data with the social information and the spatial features with the environmental information.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and in particular to a method and device for predicting pedestrian trajectories based on a graph attention mechanism. Background Art

[0002] Autonomous driving technology covers the fields of computer science, pattern recognition and intelligent control. It can help vehicles avoid obstacles in time and drive safely by predicting the movement of pedestrians in advance.

[0003] Due to the randomness of pedestrian movement, it is challenging to accurately predict pedestrian movement trajectories. Many methods have been proposed for this purpose. The social force model can decompose and model the internal factors that affect pedestrian movement in various road environments, and predict the movement behavior of pedestrians under complex road conditions with traffic congestion. The Bayesian model generates the surrounding environmental conditions of pedestrians according to real-time road conditions, and applies the Bayesian theorem to give the possible probabilities of pedestrian movement directions. However, the final walking direction and trajectory of pedestrians will be affected by these probabilities.

[0004] As a pioneer of deep learning methods, Social-LSTM captures pedestrian movement patterns through LSTM units and introduces a social pooling mechanism to simulate crowd interactions. However, traditional Social-LSTM and its derivative models often ignore the importance of distance between pedestrians when dealing with social interactions between pedestrians, and assign the same weight to all adjacent pedestrians.

[0005] The attention mechanism performs well in capturing dependencies between elements in a scene, but it has limitations in modeling complex interactions and computational efficiency. Summary of the invention

[0006] The present invention proposes an improved model based on the attention mechanism, which can assign dynamic weights to different adjacent pedestrians according to the hidden state obtained by LSTM, thereby focusing on pedestrians that have an important impact on the prediction and effectively filtering out irrelevant information.

[0007] In order to further improve the prediction accuracy, the present invention proposes to use the Graph Attention Network (GAT) to address the limitations of processing directed graphs and dynamic graphs. GAT improves the model's ability to express and predict graph structured data by allowing different importance to be assigned to different nodes in the neighborhood.

[0008] In addition, the present invention adds GAT to the model to capture scene features and enhance the accuracy of trajectory prediction. GAT performs well in object classification and tracking, and when combined with LSTM, it can simultaneously capture the motion features of pedestrians and the contextual information of the scene. Through this multimodal approach, the present invention can not only simulate social interactions between pedestrians, but also consider the impact of the surrounding environment on trajectory prediction.

[0009] One aspect of the present invention provides a pedestrian trajectory prediction method based on a graph attention mechanism, the pedestrian trajectory prediction method comprising: acquiring traffic data and performing data preprocessing to obtain pedestrian data; passing the pedestrian data through a self-attention layer and a long short-term memory network in sequence to generate spatiotemporal variables; inputting the spatiotemporal variables into a graph attention network to generate and output pedestrian data with social information; using the pedestrian data to generate spatial features with environmental information; and using the pedestrian data with social information and the spatial features with environmental information to generate predicted pedestrian trajectories.

[0010] According to an embodiment, the step of performing data preprocessing to obtain pedestrian data includes: performing normalization processing on the traffic data; and converting the absolute position of the input sequence in the pedestrian data into the relative position of the pedestrian.

[0011] According to an embodiment, the step of sequentially passing the pedestrian data through a self-attention layer and a long short-term memory network to generate spatiotemporal variables includes: based on the relative positions of the pedestrians, assigning different weights to adjacent pedestrians that have different influences on the target pedestrian through the self-attention layer, and disentangling the temporal-spatial dynamic dependency existing in the output of the self-attention layer through the long short-term memory network to generate spatiotemporal variables.

[0012] According to an embodiment, a nonlinear layer is set between two linear layers in the graph attention network to form a dynamic attention network, wherein the step of inputting the spatiotemporal variables into the graph attention network to generate and output pedestrian data with social information includes: converting the spatiotemporal variables into node data through the dynamic attention network, calculating attention scores for the node data, performing weighted averaging on neighbor node features in the node data based on the calculated attention scores, and obtaining and outputting pedestrian data with social information.

[0013] According to an embodiment, the step of generating spatial features with environmental information using the pedestrian data includes: using a residual neural network to extract feature vectors from a semantic segmentation map in the pedestrian data; flattening the extracted feature vectors and resizing them through a multi-layer perceptron to obtain a physical environment feature vector; and converting the pedestrian data into a trajectory sequence in a semantic map pixel coordinate system to form a semantic map, and generating spatial features with environmental information using the spatial features of the semantic map.

[0014] According to an embodiment, the step of generating a predicted pedestrian trajectory using the pedestrian data with social information and the spatial features with environmental information includes: splicing the spatial features with environmental information and the pedestrian data with social information to form a mixed feature vector; and inputting the mixed feature vector into the conditional variational autoencoder to obtain a conditional probability distribution, sampling from the conditional probability distribution to obtain latent variables, and inputting the latent variables into another long short-term memory network to generate a predicted pedestrian trajectory.

[0015] According to an embodiment, the pedestrian trajectory prediction method may further include: in data preprocessing, dividing the acquired traffic data into a training set and a test set; and training a neural network for executing the pedestrian trajectory prediction method based on the graph attention mechanism through the training set, and correcting the error of the pedestrian trajectory prediction neural network through the test set.

[0016] Another aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the pedestrian trajectory prediction method based on the graph attention mechanism as described above.

[0017] Another aspect of the present invention provides a computer device, comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the pedestrian trajectory prediction method based on the graph attention mechanism as described above is implemented.

[0018] Another aspect of the present invention provides a pedestrian trajectory prediction device based on a graph attention mechanism, the pedestrian trajectory prediction device comprising: a data preprocessing module, which acquires traffic data and performs data preprocessing to obtain pedestrian data; a time encoder module, which sequentially inputs the pedestrian data into a self-attention layer and a long short-term memory network to generate spatiotemporal variables; a global interaction module, which inputs the spatiotemporal variables into a graph attention network to generate and output pedestrian data with social information; a semantic map module, which uses the pedestrian data to generate spatial features with environmental information; and a decoder module, which uses the pedestrian data with social information and the spatial features with environmental information to generate predicted pedestrian trajectories.

[0019] The present invention aims at the problem of pedestrian trajectory prediction in autonomous driving. It uses deep learning to implement it as a sequence generation problem, establishes social interactions between different pedestrians and captures the influence of the surrounding environment on pedestrian trajectories, thereby achieving accurate prediction of pedestrian trajectories with good robustness, scalability and generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a pedestrian trajectory prediction method based on a graph attention mechanism according to an embodiment is shown.

[0021] Figure 2 is a schematic diagram of a time encoder module according to an embodiment.

[0022] Figure 3 is a schematic diagram of a global interaction module according to an embodiment.

[0023] Figure 4 A pedestrian trajectory prediction device based on a graph attention mechanism according to an embodiment is shown. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described herein. However, it should be understood that the disclosed embodiments are examples only, and other embodiments may take various forms and alternative forms. The specific structural and functional details disclosed herein should not be interpreted as limitations, but only as a representative basis for teaching those skilled in the art to utilize the present invention in various forms. As will be understood by those of ordinary skill in the art, each feature shown and described with reference to any of the accompanying drawings may be combined with the features shown in one or more other accompanying drawings to produce an embodiment that is not explicitly shown or described. The combination of features shown provides representative embodiments for typical applications. However, various combinations and variations of features consistent with the teachings of the present invention may be desired for specific applications or embodiments.

[0025] The present invention provides a pedestrian trajectory prediction method and device based on a graph attention mechanism, which uses a time encoder composed of an attention mechanism to capture the spatiotemporal dynamics of pedestrians. The encoder can learn the motion patterns and subtle movements of pedestrians in crowded scenes. The model models the physical environment and pedestrian behavior information in the scene as a semantic graph, and the graph attention network (GAT) is used in many models to capture social interactions. The pedestrian trajectory prediction problem is modeled as a sequence generation problem, and the attention aggregation of shared information and the collection of scene information are considered and improved on the basis of the classic pedestrian trajectory prediction framework. It has good generalization performance and is helpful to the field of autonomous driving to solve the reasonable travel problem of vehicle trajectory prediction considering pedestrian factors.

[0026] The technical solutions in the embodiments will be described clearly and completely below in conjunction with the drawings in the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0027] Figure 1 A flowchart of a pedestrian trajectory prediction method based on a graph attention mechanism according to an embodiment is shown.

[0028] Reference Figure 1According to an embodiment, a pedestrian trajectory prediction method based on a graph attention mechanism may include: in step 101, traffic data is acquired and data preprocessing is performed to obtain pedestrian data; in step 102, the pedestrian data is sequentially passed through a self-attention layer and a long short-term memory network to generate spatiotemporal variables; in step 103, the spatiotemporal variables are input into a graph attention network to generate and output pedestrian data with social information; in step 104, spatial features with environmental information are generated using pedestrian data; and in step 105, predicted pedestrian trajectories are generated using pedestrian data with social information and spatial features with environmental information.

[0029] The pedestrian trajectory prediction method based on the graph attention mechanism according to the embodiment can establish social interactions between different pedestrians and capture the influence of the surrounding environment on the pedestrian trajectory, thereby achieving accurate prediction of the pedestrian trajectory.

[0030] In the above step 101, the step of performing data preprocessing to obtain pedestrian data may include: performing normalization processing on the traffic data; and converting the absolute position of the input sequence in the pedestrian data into the relative position of the pedestrian. The step of acquiring traffic data and performing data preprocessing to obtain pedestrian data may include data acquisition and data preprocessing.

[0031] Data acquisition can obtain basic pedestrian data including pedestrian trajectories by acquiring traffic data. Pedestrian movement trajectory data can be recorded by traffic data collection devices (e.g., global positioning system (GPS) devices, video detectors, and cameras, etc.). The acquired data format may include information such as pedestrian number, location (longitude, latitude), movement speed, date and time. In addition, environmental factors that affect pedestrian movement (e.g., buildings, walls, and doors, etc.) can also be marked. The integrity and detail of the acquired data can ensure the accuracy of the model.

[0032] After obtaining the traffic data, data preprocessing can be performed to better input the model for training.

[0033] In the step of data preprocessing, the pedestrian trajectory data can be resampled into coordinate data and standardized. Specifically, data preprocessing may include resampling the trajectory data into standardized coordinate data according to a predetermined format, such as formatting the acquired pedestrian identification code (ID), location, speed and other data information according to a predetermined format. At the same time, groups of pedestrians can be marked, assuming that they have the same destination. These preprocessing steps can achieve the optimization of subsequent time coding and interactive learning. Then, data enhancement operations are performed on these coordinate data, such as random rotation, mirroring, adding Gaussian noise, etc. These operations can increase the diversity of the data and avoid model overfitting, thereby improving the generalization ability of the model and the reliability of prediction. Data preprocessing may also include standardization and data expansion techniques for the acquired pedestrian trajectory data.

[0034] According to one embodiment, data preprocessing may include: resampling the ID number, three-dimensional position and speed in the pedestrian data, for example, organizing the pedestrian data in the format of [frame_number, ped_ID, pos_x, pos_z, pos_y, v_x, v_z, v_y]; recording people walking in groups, each line containing a list of IDs, and these IDs constitute a group; assuming that all moving pedestrians have their own destinations, and obtaining the assumed destinations of all moving objects in the scene; taking the origin in the first observation point as the coordinate origin, and then taking each point in the sequence as the origin of the first position, and calculating the distance to other points; and applying random rotation to each trajectory and applying Gaussian noise with a mean value of 0 to each point.

[0035] In addition, data preprocessing may also include normalizing the traffic data. Specifically, all variables may be mapped to a uniform interval, and the minimum value v of each variable may be selected. min and the maximum value v max , set the variable in the value range [v min , v max ] are mapped to the interval [0,1]. In addition, the input and output data and weight parameters of the temporal encoder are normalized in the same way to improve the stability and efficiency of training.

[0036] In addition, data preprocessing may also include converting the absolute positions in the pedestrian data into the relative positions of the pedestrians. Specifically, the entire space can be divided into grids of fixed size, and each pedestrian can be assigned to a corresponding grid cell according to the position of each pedestrian. A binary mask is used to represent the occupancy of each pedestrian in the grid of another pedestrian. Traverse all frames in the sequence and obtain grid masks for each frame respectively. Input a series of pedestrian positions, spatial dimensions, a list of pedestrians for each frame, neighborhood and grid size, and a flag for calculating the occupancy map. Returns a list of grid masks for each frame in the sequence. Finally, a list containing the grid masks for each frame in the input sequence is returned. The tensor of the input sequence, the pedestrian list, and the pedestrian ID are mapped as input data to a dictionary of corresponding indices in the tensor, and the data converted to relative positions is used as output data. The origin is moved to the last observed time step of each pedestrian for data normalization to ensure that the model is invariant to translation. The observed trajectory of pedestrian i is represented by relative position, expressed as in is the offset from (t-1) to the next time step. Get the relative position between pedestrians i and j, and the relative position is expressed as the following formula 1:

[0037]

[0038] where i≠j to preserve the relative position information between pedestrians, which is crucial for modeling their interactions.

[0039] In the above step 102, the step of sequentially passing the pedestrian data through a self-attention layer and a long short-term memory network (LSTM) to generate a spatiotemporal variable includes: the step of sequentially passing the pedestrian data through a self-attention layer and a long short-term memory network to generate a spatiotemporal variable includes:

[0040] Based on the relative position of the pedestrian, different weights are assigned to adjacent pedestrians that have different influences on the target pedestrian through the self-attention layer, and the temporal and spatial dynamic dependencies in the output of the self-attention layer are untangled through the long short-term memory network (LSTM) to generate spatiotemporal variables. In addition, the relative position of the pedestrian can be input into a one-dimensional convolutional layer (Conv1D) and a multi-layer perceptron (MLP), and then the obtained output is input into the self-attention layer for weight allocation. The one-dimensional convolutional layer uses a self-attention mechanism to cleverly distinguish temporal dynamics from the observed trajectory. In the above steps, the relative position can be input into the one-dimensional convolutional layer (Conv1D), the multi-layer perceptron (MLP), the self-attention layer, and the long short-term memory network in sequence.

[0041] Specifically, a deep learning network can be established and trained to identify key temporal features and temporal dependencies through normalization, relative position representation, self-attention layer and LSTM processing.

[0042] By capturing the temporal characteristics of pedestrian motion, it helps the model better understand the dynamic changes of trajectories, identify key temporal features and temporal dependencies. This module combines normalization, relative position representation, self-attention layer and LSTM to achieve dynamic modeling in the temporal dimension.

[0043] Figure 2 is a schematic diagram of a time encoder module according to an embodiment.

[0044] Specifically, refer to Figure 2 , the step of generating spatiotemporal variables (hidden states) using pedestrian data may include: processing the relative positions through a one-dimensional convolutional layer Then, two layers of positional multilayer perceptrons are used. The Transformer encoder module in the self-attention mechanism is used to identify key temporal features, emphasizing the temporal sequence steps that increase attention through the self-attention layer. To generate spatiotemporal variables, the positional encoding inherent in the Transformer encoder module is integrated at the base of the encoder block, preserving the order of the data. The key of the Transformer encoder module (e.g., the Temporal Encoder module) involves the use of LSTM to disentangle the temporal dependencies that change over time. Each pedestrian's observation trajectory is processed independently, ensuring that their respective temporal dynamics are analyzed in parallel. The self-attention mechanism is introduced here to further enhance the expressive power of temporal features. The temporal dynamics in the trajectory are discerned through the Transformer encoder module in the self-attention mechanism. In addition, LSTM is used to disentangle the temporal dependencies and capture the motion patterns of pedestrians at different time steps, thereby better simulating the behavior dynamics of pedestrians.

[0045] The trajectory data is processed by normalization and relative position representation, and feature extraction is performed using one-dimensional convolutional layers and multi-layer perceptrons. Then, the Transformer encoder module is used to learn temporal dynamic features, emphasizing the importance of each time step in the trajectory. The self-attention mechanism can capture the interaction information between pedestrians by calculating the relative positions between them, retaining the sequential information of the trajectory. The self-attention mechanism allows the model to pay attention to temporal dynamics from multiple aspects at the same time, thereby improving the accuracy of trajectory prediction.

[0046] LSTM is used to process long-term dependencies and further encode temporal features into time series information to capture temporal dependencies in pedestrian trajectories. The recursive nature of LSTM can effectively learn the trend of pedestrian movement, enabling the model to better predict pedestrians' future trajectories. By combining normalization, self-attention layers, and LSTM, the temporal encoder module achieves effective capture and modeling of trajectory temporal dynamics.

[0047] Figure 3 is a schematic diagram of a global interaction module according to an embodiment.

[0048] Specifically, refer to Figure 3 According to an embodiment, a multilayer perceptron (MLP-a), a nonlinear layer (LeakyReLU), another multilayer perceptron (MLP-b), and a softmax function block may be sequentially arranged. The graph attention mechanism (GAT) may be used to capture the mutual relationships between pedestrians and improve the accuracy of trajectory prediction. In the global interaction module, nodes represent pedestrians, edges describe the relationships between pedestrians, and information interaction between nodes is achieved through attention weights. Specifically, the information of neighbor nodes is aggregated through the GAT model and the attention scores are calculated to derive the importance of the corresponding nodes.

[0049] In the above step 103, a nonlinear layer is set between two linear layers in the graph attention network to form a dynamic attention network, wherein the step of inputting the spatiotemporal variables into the graph attention network to generate and output pedestrian data with social information includes: converting the spatiotemporal variables into node data through the dynamic attention network, calculating the attention score for the node data, performing weighted averaging on the neighbor node features in the node data based on the calculated attention score, and obtaining and outputting the pedestrian data with social information. The weighted average result of the dynamic attention network can be used to generate pedestrian data with social information through residual connection and motion gate mechanism.

[0050] Specifically, the steps of inputting spatiotemporal variables into the graph attention network to generate and output pedestrian data with social information include:

[0051] In step 1031, the set of input node representations And the corresponding edge set E. The same parameter function f is used uniformly on each node θ and AGGREGATE, then the layer outputs a series of nodes represented as GAT calculation N i ={j∈V|(i, j)∈E} to aggregate the representation of neighbors in pedestrians. The importance of the features of neighborhood j to node i is calculated by the score function, as shown in Equation 2:

[0052] score(ri , r j )=LeakyReLU(a T ·[Wr i ||Wr j ]) Formula 2

[0053] Among them, Wr i and Wr j Both represent weights.

[0054] At step 1032, the attention score is calculated over all neighbors j∈N i The normalization on is achieved through the softmax function operation. The definition of the attention softmax function is as follows:

[0055] α ij =softmax(score(r i , r j )) Formula 3

[0056] Among them, α is the learned weight vector.

[0057] In step 1033, GAT calculates the weighted average of the conversion features associated with the adjacent nodes, as shown in Equation 4:

[0058]

[0059] In standard GAT, linear layers follow each other and the ranking of nodes is not restricted by the query node. To overcome this problem, the following improvements are proposed.

[0060] In step 1034, after calculating the attention scores for the spatiotemporal variables, the neighbor node features in the pedestrian are weighted averaged, the order of the nonlinear layer (LeakyReLU) in the dynamic attention is changed, and the nonlinear layer is placed between two linear layers to separate them and prevent them from collapsing into one linear layer, making the structure more expressive, as shown in Equation 5:

[0061] score(r i , r j )=aTLeakyReLU(Wr i ||Wr j ) Formula 5

[0062] In the global interaction module, the dynamic attention network is introduced to make the attention calculation process more flexible and effective, ensuring that each pedestrian node has appropriate attention to other pedestrians in its neighborhood. It can more accurately simulate the interaction between pedestrians, especially in crowded environments. By dynamically modeling global interactions, the uncertainty problem in trajectory prediction is effectively solved.

[0063] After the dynamic attention network completes message passing and updating, it generates pedestrian data with social information through residual connection and motion gate mechanism.

[0064] In step 1035, pedestrian data with social information is generated through residual connection and motion gate mechanism. For example, residual connection is performed as shown in the following equation 6:

[0065]

[0066] The motion gate is defined by using the relative position between pedestrians i and j and their respective hidden states as follows:

[0067]

[0068] In the model using conventional GAT, the neighbor's The impact of MLP Learning, as shown in Equation 8:

[0069]

[0070] DATraj A nonlinear layer and an MLP are added to calculate dynamic attention, as shown in Equation 9:

[0071]

[0072] The weights are normalized by the Softmax function through all neighboring pedestrians, as shown in Equation 10:

[0073]

[0074] Output the LSTM cell state, and then update the cell state through the message selection mechanism, as shown in Equation 11:

[0075]

[0076] in, represents the cell state of pedestrian i after message passing, ⊙ represents the element-wise product operation, is an MLP layer. Message passing may refer to passing messages of a graph including nodes and edges between the nodes.

[0077] The residual connection and motion gate mechanism ensure that features at different levels can be smoothly transmitted to better capture the interaction between nodes. Residual connections help alleviate the gradient vanishing problem in deep neural networks, thereby maintaining the training efficiency and effectiveness of the model. The motion gate is defined by relative position and hidden state, allowing the model to selectively update the interaction information, thereby improving the accuracy of trajectory prediction.

[0078] In the above step 104, the step of using pedestrian data to generate spatial features with environmental information includes: using a residual neural network to extract feature vectors from a semantic segmentation map in the pedestrian data; flattening the extracted feature vectors and resizing them through a multi-layer perceptron to obtain a physical environment feature vector; and converting the pedestrian data into a trajectory sequence in a pixel coordinate system of the semantic map to form a semantic map, and using the spatial features of the semantic map to generate spatial features with environmental information.

[0079] According to one embodiment, a semantic map module based on ResNet can be established, and semantic segmentation map features can be extracted through ResNet18, and physical environment feature vectors can be obtained through MLP processing, and these features can be used to predict trajectories in the semantic map pixel coordinate system. The spatial features of the scene are extracted from the semantic segmentation map of the environment.

[0080] Using pedestrian data to generate spatial features with environmental information may include:

[0081] In step 1041, a convolutional neural network (e.g., ResNet18) is used to extract features from the semantic segmentation map, as shown in Formula 12:

[0082]

[0083] in, represents the semantic segmentation map input at time step t, Features represent the feature vector extracted from the semantic segmentation map.

[0084] At step 1042, the feature vector Flatten layer is flattened in dimension and resized through a multi-layer perceptron (MLP) layer to obtain a physical environment feature vector As shown in formula 13:

[0085]

[0086] In step 1043, the observed trajectory of the pedestrian is converted into a trajectory sequence in the pixel coordinate system of the semantic map, and the entire prediction process is completed in this pixel coordinate system to use the spatial features of the semantic map for trajectory prediction.

[0087] In the above step 105, the step of generating predicted pedestrian trajectories using pedestrian data with social information and spatial features with environmental information includes: concatenating spatial features with environmental information and pedestrian data with social information to form a mixed feature vector; and inputting the mixed feature vector into a conditional variational autoencoder (CVAE) to obtain a conditional probability distribution, sampling from the conditional probability distribution to obtain latent variables, and inputting the latent variables into another long short-term memory network to generate predicted pedestrian trajectories.

[0088] The conditional variational autoencoder and LSTM can be used to predict the future pedestrian trajectory. The CVAE model combines semantic features with environmental features to generate more representative trajectory features.

[0089] Specifically, the step of generating a predicted pedestrian trajectory using pedestrian data with social information and spatial features with environmental information may include:

[0090] In step 1051, a conditional variational autoencoder is established to concatenate pedestrian data with social information and spatial features with environmental information to form a mixed feature vector as shown in Formula 14:

[0091]

[0092] in, represents pedestrian data with social information, Represents spatial features with environmental information, At time step t, the scene segmentation map The mixed feature vector is obtained by concatenating the feature vector extracted by ResNet with the context information.

[0093] In step 1052, the latent variable z is sampled from the latent space of the CVAE model as follows:

[0094]

[0095] The latent variable z is sampled from the conditional probability distribution through the conditional variational autoencoder. The introduction of the latent variable z enables the model to capture the uncertainty in the trajectory when generating future trajectories, thereby generating a more natural and realistic predicted trajectory.

[0096] In step 1053, the future trajectory is generated using the LSTM network as shown in Equation 16:

[0097]

[0098] Among them, W g is the weight of the LSTM network, using LSTM and latent variables z to predict the next time step T obs +1 to generate the future pedestrian trajectory. LSTM predicts the trajectory of the next time step based on the current state and latent variable z, as shown in Equation 17:

[0099]

[0100] Among them, W g Represents the weight parameters of LSTM, which are learned by back-propagation training of LSTM.

[0101] In step 1054, the position coordinates of the future trajectory are decoded as shown in Equation 18:

[0102]

[0103] The output of LSTM is decoded through the nonlinear activation function δ to obtain the predicted future trajectory position coordinates. Using the implicit state generated by LSTM, the future trajectory position is decoded through the nonlinear activation function as follows:

[0104]

[0105] In addition, the pedestrian trajectory prediction method may also include: in data preprocessing, dividing the acquired traffic data into a training set and a test set; and training a neural network for executing the pedestrian trajectory prediction method based on the graph attention mechanism through the training set, and correcting the error of the pedestrian trajectory prediction neural network through the test set. The above-mentioned neural network for executing the pedestrian trajectory prediction method based on the graph attention mechanism includes but is not limited to one or more self-attention layers, long short-term memory networks, graph attention networks, and residual neural networks in steps 101 to 105 above. At the same time, the original data can be organized into a pedestrian trajectory prediction sample data set according to a certain prediction step size.

[0106] In the validation phase of the model, the test data set is forward calculated, and the input position, grid mask, hidden and unit state, and pedestrian ID list of each frame in the sequence are used as input to verify its prediction effect, and the structure and hyperparameters of the model prediction neural network are adjusted. This is repeated for each deep network model, and its prediction error on the test set is calculated and recorded. The network structure and parameters corresponding to the minimum error are selected as the structure and parameters of the optimal model to ensure that the model can achieve the best performance.

[0107] The decoder module combines CVAE and LSTM to not only capture the complex dynamic features in the trajectory, but also introduce uncertainty in the trajectory through latent variables, making the generated trajectory more realistic. In addition, by sampling latent variables multiple times in the decoding stage, the uncertainty of the trajectory can be simulated in a diversified way, so that different trajectory predictions can be obtained in the testing stage, improving the adaptability of the model.

[0108] According to an exemplary embodiment of the present invention, a computer-readable storage medium storing a computer program is also provided. The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to execute a pedestrian trajectory prediction method based on a graph attention mechanism according to the present invention. The computer-readable recording medium is any data storage device that can store data read out by a computer system. Examples of computer-readable recording media include: read-only memory, random access memory, read-only optical disk, magnetic tape, floppy disk, optical data storage device, and carrier wave (such as data transmission over the Internet via a wired or wireless transmission path).

[0109] According to an exemplary embodiment of the present invention, a computer device is also provided. The computer device includes a processor and a memory. The memory is used to store a computer program. The computer program is executed by the processor so that the processor executes the computer program of the pedestrian trajectory prediction method based on the graph attention mechanism according to the present invention.

[0110] Figure 4 A pedestrian trajectory prediction device based on a graph attention mechanism according to an embodiment is shown.

[0111] Reference Figure 4 According to an embodiment of the present invention, a pedestrian trajectory prediction device based on a graph attention mechanism may include: a data preprocessing module 10, which acquires traffic data and performs data preprocessing to obtain pedestrian data; a time encoder module 20, which generates spatiotemporal variables using pedestrian data; a global interaction module 30, which inputs the spatiotemporal variables into a graph attention network to generate and output pedestrian data with social information; a semantic map module 40, which generates spatial features with environmental information using pedestrian data; and a decoder module 50, which generates predicted pedestrian trajectories using pedestrian data with social information and spatial features with environmental information.

[0112] According to the embodiment, the pedestrian trajectory prediction device based on the graph attention mechanism can establish social interactions between different pedestrians and capture the influence of the surrounding environment on the pedestrian trajectory, thereby achieving accurate prediction of the pedestrian trajectory.

[0113] The pedestrian trajectory prediction method based on the graph attention mechanism according to the embodiment can be performed by the pedestrian trajectory prediction device based on the graph attention mechanism described above, but is not limited thereto.

[0114] The embodiments of the present invention have been described in detail in conjunction with the accompanying drawings. It should be noted that the implementation methods not shown or described in the drawings or the body of the specification are all forms known to ordinary technicians in the relevant technical field and are not described in detail. In addition, the above definitions of each component are not limited to the various specific structures, shapes or methods mentioned in the embodiments, and ordinary technicians in the field can simply change or replace them.

[0115] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined or coupled in a variety of ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined or coupled in a variety of ways without departing from the spirit and teachings of the present invention. All of these combinations or couplings fall within the scope of the present invention.

[0116] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0117] The words used in the specification are descriptive words rather than restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present invention. As before, the features of each embodiment can be combined to form further embodiments of the present invention that may not be clearly described or shown. Although each embodiment may be described as providing advantages or being superior to other embodiments or the implementation of the prior art in one or more desired characteristics, it should be appreciated by those of ordinary skill in the art that one or more features or characteristics can be compromised, to achieve the desired overall system properties, according to specific application and implementation. These attributes may include but are not limited to cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. Therefore, the embodiments described as being less desirable than other embodiments or the implementation of the prior art in one or more characteristics are not outside the scope of the present invention, and can be expected to be used for specific applications.

Claims

1. A pedestrian trajectory prediction method based on graph attention mechanism, characterized in that: The pedestrian trajectory prediction method comprises: Acquire traffic data and perform data preprocessing to obtain pedestrian data; Passing the pedestrian data sequentially through a self-attention layer and a long short-term memory network to generate spatiotemporal variables; Inputting the spatiotemporal variables into a graph attention network to generate and output pedestrian data with social information; generating spatial features with environmental information using the pedestrian data; and The pedestrian data with social information and the spatial features with environmental information are used to generate predicted pedestrian trajectories.

2. The pedestrian trajectory prediction method according to claim 1, characterized in that: The steps to perform data preprocessing to obtain pedestrian data include: performing normalization processing on the traffic data; and Convert the absolute positions of the input sequence in pedestrian data to the relative positions of pedestrians.

3. The pedestrian trajectory prediction method according to claim 2, characterized in that: The steps of sequentially passing the pedestrian data through the self-attention layer and the long short-term memory network to generate spatiotemporal variables include: Based on the relative positions of the pedestrians, different weights are assigned to adjacent pedestrians that have different influences on the target pedestrian through the self-attention layer, and the temporal and spatial dynamic dependencies existing in the output of the self-attention layer are disentangled through the long short-term memory network to generate spatiotemporal variables.

4. The pedestrian trajectory prediction method according to claim 1, characterized in that: In the graph attention network, a nonlinear layer is set between two linear layers to form a dynamic attention network. The step of inputting the spatiotemporal variables into a graph attention network to generate and output pedestrian data with social information includes: Through the dynamic attention network, the spatiotemporal variables are converted into node data, the attention scores are calculated for the node data, and the neighbor node features in the node data are weighted averaged based on the calculated attention scores to obtain and output pedestrian data with social information.

5. The pedestrian trajectory prediction method according to claim 1, characterized in that: The step of generating a spatial feature with environmental information using the pedestrian data includes: Using a residual neural network to extract feature vectors from the semantic segmentation map in the pedestrian data; Flattening the extracted feature vector and resizing it through a multi-layer perceptron to obtain a physical environment feature vector; and The pedestrian data is converted into a trajectory sequence in the pixel coordinate system of the semantic map to form a semantic map, and the spatial features of the semantic map are used to generate spatial features with environmental information.

6. The pedestrian trajectory prediction method according to claim 1, characterized in that: The step of generating a predicted pedestrian trajectory using the pedestrian data with social information and the spatial features with environmental information includes: Concatenate the spatial features with environmental information and the pedestrian data with social information to form a hybrid feature vector; and The mixed feature vector is input into a conditional variational autoencoder to obtain a conditional probability distribution, latent variables are sampled from the conditional probability distribution, and the latent variables are input into another long short-term memory network to generate a predicted pedestrian trajectory.

7. The pedestrian trajectory prediction method according to claim 1, characterized in that: The pedestrian trajectory prediction method may further include: In data preprocessing, the acquired traffic data is divided into a training set and a test set; and A neural network for executing a pedestrian trajectory prediction method based on a graph attention mechanism is trained using the training set, and an error of the pedestrian trajectory prediction neural network is corrected using the test set.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the pedestrian trajectory prediction method based on the graph attention mechanism as described in any one of claims 1 to 7 is implemented.

9. A computer device, characterized in that: The computer device comprises: processor; A memory storing a computer program, which, when executed by a processor, implements the pedestrian trajectory prediction method based on the graph attention mechanism as described in any one of claims 1 to 7.

10. A pedestrian trajectory prediction device based on graph attention mechanism, characterized in that: The pedestrian trajectory prediction device comprises: A data preprocessing module, which acquires traffic data and performs data preprocessing to obtain pedestrian data; A temporal encoder module, which sequentially inputs the pedestrian data into a self-attention layer and a long short-term memory network to generate spatiotemporal variables; A global interaction module, which inputs the spatiotemporal variables into a graph attention network to generate and output pedestrian data with social information; A semantic map module, which generates spatial features with environmental information using the pedestrian data; and A decoder module generates a predicted pedestrian trajectory using the pedestrian data with social information and the spatial features with environmental information.

Images