A Pedestrian Trajectory Prediction Method Based on Graph Neural Network with Attention Mechanism

Abstract

The present invention discloses a method for predicting pedestrian trajectories based on a graph neural network with an attention mechanism. The method mainly includes: collecting pedestrian trajectory information, extracting trajectory motion features, and constructing an original node graph of pedestrian trajectories; fusing, discarding, and amplifying the original node graph of pedestrian trajectories to generate a final node graph of pedestrian trajectories; extracting the spatiotemporal features of the final node graph of pedestrian trajectories using a spatiotemporal graph convolutional neural network, constructing an original spatiotemporal feature graph of pedestrian trajectories based on the spatiotemporal features, and using a graph channel attention mechanism to screen out important spatiotemporal features to form a new spatiotemporal feature graph; inputting the new spatiotemporal feature graph into a predictor, which outputs a predicted pedestrian trajectory within a preset time; assigning weights to the predicted pedestrian trajectories, and taking the trajectory with the largest weight as the final prediction result. The method maximizes effective information by paying more attention to features that affect the final result, effectively improving the accuracy of pedestrian trajectory prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of trajectory planning, and in particular to a graph neural network pedestrian trajectory prediction method based on an attention mechanism. Background Art

[0002] Incorrect driving behavior will not only cause economic losses, but also seriously endanger the lives of oneself and others. Therefore, autonomous driving technology has considerable prospects due to its safety and intelligence. Pedestrian trajectory prediction is a key issue in autonomous driving technology. In autonomous driving, accurate prediction of pedestrian trajectories can allow decision-makers to plan routes in advance and reduce the possibility of traffic jams and accidents. However, due to the uncertainty of pedestrian behavior and the complex interaction with the environment, it is difficult to achieve sufficient accuracy in the prediction task.

[0003] In recent years, many methods have been proposed for trajectory prediction: The initial trajectory prediction task usually uses a uniform linear predictor, but this prediction method has the defect that it can only be used to predict scenes of uniform straight-line movement. If the pedestrian motion trajectory to be predicted is very complex, then this method cannot be applied. Later traditional methods often use social force models, which represent the attraction and repulsion of pedestrians to predict the interaction between pedestrians and the environment. However, this model relies on manually designed features. If the environment is complex, it is difficult for the model to express the implicit interaction behavior at this time. Some other prediction methods include Markov decision processes and Gaussian mixture models, but their performance is not satisfactory.

[0004] With the development of neural networks in recent years, trajectory prediction tasks are often completed by training neural networks. RNN and its variants LSTM and GRU are gradually applied to trajectory prediction because they can extract time series features, and have achieved good results. Although recurrent neural networks can achieve very good results in extracting Euclidean spatial data features, most of the data for trajectory prediction tasks are non-Euclidean data. Therefore, due to the lack of advanced spatiotemporal views, recurrent neural networks cannot handle non-Euclidean spatial data well. Graph structures can more intuitively and effectively express the interactions between pedestrians. Graphs are irregular structures that contain a variable number of nodes, and nodes may or may not interact with each other. Therefore, compared with traditional neural networks, graph neural networks can capture the relationships between instances. Current interaction technologies often focus more on spatial interactions and ignore temporal correlations. We can use spatiotemporal graph neural networks to pay more attention to spatiotemporal correlations and complete trajectory prediction tasks more efficiently.

[0005] GUPTA et al. proposed the Social GAN ​​model, which defines a spatial pool for motion prediction. They also proposed a new pooling mechanism that enables the network to generate trajectories that conform to social norms by learning past historical trajectory codes. However, neural networks based on the GAN model are prone to slow convergence.

[0006] In addition, there has been research on the Social-BIGAT model. The Social-BIGAT model introduces a graph attention network to improve the modeling of social interactions between pedestrians in the scene, allowing all pedestrians in the scene to interact, and encourages the generalization of multimodal distribution by constructing a reversible mapping between the output trajectory and the representation of pedestrian behavior in the scene. It can learn larger multimodal trajectory distributions, but the Social-BIGTA model lacks features, resulting in the pedestrian trajectory prediction effect not meeting expectations. Summary of the invention

[0007] In order to overcome the problem of low pedestrian trajectory prediction accuracy in autonomous driving technology, the present invention provides a graph neural network pedestrian trajectory prediction method based on an attention mechanism, which simultaneously focuses on spatial correlation and temporal correlation in interactive technology and maximizes effective information through the attention mechanism.

[0008] The technical solution proposed by the present invention is as follows:

[0009] S1: Collect pedestrian trajectory information, extract trajectory motion features, and construct a pedestrian trajectory original node graph, wherein the pedestrian trajectory original node graph contains spatial information and temporal information of the pedestrian trajectory;

[0010] S2: fusing, discarding and amplifying the original node graph of the pedestrian trajectory, filtering out the information that has a significant impact on the formation of the pedestrian trajectory, and generating a final node graph of the pedestrian trajectory, wherein the fusing, discarding and amplifying are completed by a graph channel attention mechanism;

[0011] S3: Use the spatiotemporal graph convolutional neural network to extract the spatiotemporal features of the final node graph of the pedestrian trajectory, construct the original spatiotemporal feature graph of the pedestrian trajectory based on the spatiotemporal features, and use the graph channel attention mechanism to screen out important spatiotemporal features to form a new spatiotemporal feature graph;

[0012] S4: inputting the new spatiotemporal feature map into a predictor, which outputs a predicted pedestrian trajectory within a preset time, wherein the predictor adopts a time extrapolation neural network, and the predicted pedestrian trajectory includes a variety of different results;

[0013] S5: Assign weights to the predicted pedestrian trajectories, and take the trajectory with the largest weight as the final prediction result, wherein the weight assignment is completed by a time channel attention mechanism.

[0014] Furthermore, the original node graph of the pedestrian trajectory is represented as:

[0015] Gt=(V t ,E t ), where V t Represents the relevant information of the nodes of the pedestrian trajectory graph of the tth frame, E t Represents the relevant information of the edges of the pedestrian trajectory graph of the tth frame, where the edges represent the interaction relationship between nodes.

[0016] Furthermore, when there is an edge between two nodes, a weight is assigned to the edge. The weight represents the intensity of the interaction between nodes and is generated by a kernel function, which is:

[0017]

[0018] in, Represents the Euclidean distance between node i and node j in the pedestrian trajectory graph of frame t.

[0019] Furthermore, the graph channel attention mechanism in S2 uses a graph convolution channel attention module to filter the structural information of the original node graph of the pedestrian trajectory, wherein the graph convolution channel attention module pays more attention to the relationship between channels, identifies and filters out important channel features through a learning mechanism, and suppresses unimportant channel features.

[0020] Furthermore, the spatiotemporal features of the final node graph of pedestrian trajectories extracted by using the spatiotemporal graph convolutional neural network described in S3 need to be converted into the adjacency matrix A t Normalization, that is, Where the adjacency matrix A t is the edge weight The collection of Λ t is a diagonal matrix.

[0021] Furthermore, the expression of the convolution operation defined by the spatiotemporal graph convolutional neural network in S3 on the plane graph or feature graph is as follows:

[0022]

[0023] Wherein, σ represents the activation function, k represents the kernel size, p represents the sampling function, and l represents the number of layers; based on the expression of the convolution operation of the spatiotemporal graph convolutional neural network on the plane graph or feature graph, the convolution operation expression of the final node graph of the pedestrian trajectory is as follows:

[0024]

[0025] in, is a normalized term, B(v i)={v i |d(v i ,v j )≤D} is the neighbor set of the node.

[0026] Furthermore, the predictor in S4 performs a convolution operation on the time dimension of the new spatiotemporal feature map.

[0027] Furthermore, the temporal channel attention mechanism described in S5 not only takes into account the channel itself, but also the adjacent channels of the channel, and completes the calculation of the channel weight through one-dimensional convolution, where the number of adjacent channels is related to the size of the channel, and an exponential family function is used to process the mapping problem between the number of adjacent channels and the size of the channel, that is, C = φ(k) ≈ 2^(γ*kb), where k represents the number of adjacent channels, and C represents the size of the channel, then the adaptive algorithm expression of k is as follows:

[0028]

[0029] Furthermore, in the plane space, assuming that the random variable of the probability distribution of the position of the pedestrian at a certain moment follows a bivariate normal distribution, the loss function of the graph neural network based on the attention mechanism is as follows:

[0030]

[0031] Among them, V contains all the trainable parameters in the graph neural network, represents the mean of the probability distribution of the possible positions of pedestrians at a certain moment, represents the variance of the distribution, represents the correlation of the distribution.

[0032] Beneficial Effects

[0033] The present invention proposes a graph neural network pedestrian trajectory prediction method based on attention mechanism, which focuses on spatial correlation and temporal correlation in interactive technology. On the one hand, it uses the graph channel attention mechanism to screen out important spatiotemporal features, and on the other hand, it assigns weights to all possible trajectories through the temporal channel attention mechanism. This method maximizes effective information by paying more attention to the features that affect the final result, thereby effectively improving the accuracy of pedestrian trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0035] Figure 1 It is a schematic diagram of the steps of a method for predicting pedestrian trajectories using a graph neural network based on an attention mechanism in an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of the structure of a graph neural network based on an attention mechanism in an embodiment of the present invention;

[0037] Figure 3 Schematic diagram of the working principle of the channel convolution attention mechanism (GCA) in the embodiment of the present invention;

[0038] Figure 4 Schematic diagram of the working principle of the temporal channel convolutional attention mechanism (TCA) in an embodiment of the present invention;

[0039] Figure 5 It is a schematic diagram of pedestrian trajectory prediction results based on a graph neural network with an attention mechanism when pedestrians are walking towards each other in an embodiment of the present invention;

[0040] Figure 6 It is a schematic diagram of pedestrian trajectory prediction results of a graph neural network based on an attention mechanism when pedestrians are walking in the same direction in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention, and the solution of the present invention will be better described through embodiments.

[0042] This embodiment provides a graph neural network pedestrian trajectory prediction method based on an attention mechanism, the method comprising the following steps:

[0043] The pedestrian trajectory information used in this embodiment comes from the public human trajectory datasets UCY and ETH. Specifically, UCY has three scenes, named UNIV, ZARA1, and ZARA2; ETH has two scenes, named ETH and HOTEL.

[0044] The average displacement error and final displacement error are used to evaluate the performance of the network.

[0045] Reference Figure 1 , the specific steps of a graph neural network pedestrian trajectory prediction method based on an attention mechanism in this embodiment are as follows:

[0046] S1: Use the long short-term memory neural network (LSTM) to extract motion features from the pedestrian trajectory sequence and generate the corresponding pedestrian trajectory original node graph;

[0047] Specifically, the training set of the human trajectory dataset is input into the LSTM neural network, and the vector output by the network is used as the motion feature. In the tth frame, the graph embedded with the spatial and temporal information of the pedestrian trajectory can be expressed as: Gt = (V t ,E t ), where V t Represents the relevant information of the nodes of the pedestrian trajectory graph of the tth frame, E t Represents the relevant information of the edge of the pedestrian trajectory graph of the tth frame. The edge represents the interaction relationship between nodes. When there is an edge between two nodes, a weight is assigned to the edge. The weight represents the intensity of the interaction between nodes and is generated by a kernel function, which is:

[0048]

[0049] in, Represents the Euclidean distance between node i and node j in the pedestrian trajectory graph of frame t.

[0050] S2: Using the graph channel convolutional attention mechanism (GCA) to fuse, discard and amplify the original node graph of the pedestrian trajectory, filter out the information that has a significant impact on the formation of the pedestrian trajectory, and generate the final node graph of the pedestrian trajectory;

[0051] Specifically, the graph channel attention mechanism uses a graph convolution channel attention module to filter the structural information of the original node graph of pedestrian trajectories. The graph convolution channel attention module pays more attention to the relationship between channels, identifies and filters important channel features through a learning mechanism, and suppresses unimportant channel features.

[0052] Reference Figure 3 ,The original node graph of the pedestrian trajectory is first subjected to global mean pooling, then passes through two layers of fully connected layers, and finally through one activation layer, where the activation function is the Sigmoid function for nonlinear activation, and the final node graph of the pedestrian trajectory is obtained.

[0053] S3: The spatiotemporal features of the final node graph of the pedestrian trajectory are extracted using the spatiotemporal graph convolutional neural network (STGCN), the original spatiotemporal feature graph of the pedestrian trajectory is constructed based on the spatiotemporal features, and the graph channel attention mechanism is used to screen out important spatiotemporal features to form a new spatiotemporal feature graph;

[0054] Specifically, the spatiotemporal graph convolutional neural network extracts the spatiotemporal features of the final node graph of pedestrian trajectories by transforming the adjacency matrix A t Normalization, that is, The adjacency matrix A t is the weight of the edge The collection of Λ t is a diagonal matrix;

[0055] The expression of the convolution operation defined by the spatiotemporal graph convolutional neural network on a planar graph or feature map is as follows:

[0056]

[0057] Wherein, σ represents the activation function, k represents the kernel size, p represents the sampling function, and l represents the number of layers; based on the expression of the convolution operation of the spatiotemporal graph convolutional neural network on the plane graph or feature graph, the convolution operation expression of the final node graph of the pedestrian trajectory is as follows:

[0058]

[0059] in, is a normalized term, B(v i )={v i |d(v i ,v j )≤D} is the neighbor set of the node.

[0060] S4: inputting the new spatiotemporal feature map into a predictor, which outputs a predicted pedestrian trajectory within a preset time, wherein the predictor adopts a time extrapolation neural network (TECNN), and the predicted pedestrian trajectory includes multiple possible results;

[0061] Specifically, the predictor performs a convolution operation on the time dimension of the new spatiotemporal feature map.

[0062] S5: The time channel attention mechanism assigns weights to the predicted pedestrian trajectories, and takes the trajectory with the largest weight as the final prediction result;

[0063] Specifically, the channel attention mechanism not only takes into account the channel itself, but also the adjacent channels of the channel, and completes the calculation of the channel weight through one-dimensional convolution, where the number of adjacent channels is related to the size of the channel, and the exponential family function is used to handle the mapping problem between the number of adjacent channels and the size of the channel, that is, C = φ (k) ≈ 2^ (γ * kb), where k represents the number of adjacent channels and C represents the size of the channel. The adaptive algorithm expression of k is as follows:

[0064]

[0065] Reference Figure 4 In this embodiment, five adjacent channels around the channel are selected, and a one-dimensional (1D) convolution is used instead of a fully connected layer. Finally, a nonlinear activation is performed through an activation layer, and the activation function is a Sigmoid function;

[0066] In the plane space, assuming that the random variable of the probability distribution of the pedestrian's position at time t is set to (xt, yt), and assuming that it follows a bivariate normal distribution, the loss function of the graph neural network based on the attention mechanism is as follows:

[0067]

[0068] Among them, V contains all the trainable parameters in the graph neural network, represents the mean of the probability distribution of the possible positions of pedestrians at a certain moment, represents the variance of the distribution, represents the correlation of the distribution.

[0069] S6: Set the batch size of pedestrian trajectory sequence to 256, the initial learning rate (lr) to 0.01, the epoch to 500, and the learning rate becomes 0.002 after 300 epochs.

[0070] S7: Select a trajectory every 0.4s in the human trajectory dataset and compare it with other models.

[0071] In order to intuitively demonstrate the prediction effect, according to the method of the present invention, this embodiment takes the prediction result of the UCY-Zara02 dataset in the human trajectory dataset as an example, and marks the real trajectory, observed trajectory and predicted trajectory in the figure. This embodiment selects two states of pedestrians in the dataset for trajectory prediction, referring to Figure 5 , the lines in the figure indicate the trajectories of pedestrians, the arrow directions of the lines indicate the walking directions of pedestrians, the pedestrians in the figure are walking towards each other, the trajectory 11 represented by the black solid line in the figure is the historical trajectory of the first pedestrian, the trajectory 12 represented by the black dotted line is the real trajectory of the first pedestrian, the trajectory 13 represented by the white dotted line is the predicted trajectory of the first pedestrian output by the graph neural network based on the attention mechanism of the present invention, the trajectory 21 represented by the black solid line is the historical trajectory of the second pedestrian, the trajectory 22 represented by the black dotted line is the real trajectory of the second pedestrian, the trajectory 23 represented by the white dotted line is the predicted trajectory of the second pedestrian output by the graph neural network based on the attention mechanism of the present invention, and so on; refer to Figure 6 , the pedestrians in the figure are walking in the same direction; by analyzing the prediction results of the pedestrians in different walking states, it can be intuitively seen that the method according to the present invention can accurately predict the future trajectory of the pedestrians.

[0072] In addition, an embodiment of the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, which, when executed, includes part or all of the steps of any graph neural network pedestrian trajectory prediction method based on attention mechanism recorded in the above method embodiment.

[0073] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0074] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0075] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0076] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation methods of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

Claims

1. A graph neural network pedestrian trajectory prediction method based on attention mechanism, characterized in that: The following steps are involved: S1: Collect pedestrian trajectory information, extract trajectory motion features, and construct a pedestrian trajectory original node graph, wherein the pedestrian trajectory original node graph contains spatial information and temporal information of the pedestrian trajectory; S2: fusing, discarding and amplifying the original node graph of the pedestrian trajectory, filtering out the information that has a significant impact on the formation of the pedestrian trajectory, and generating a final node graph of the pedestrian trajectory, wherein the fusing, discarding and amplifying are completed by a graph channel attention mechanism; S3: Use the spatiotemporal graph convolutional neural network to extract the spatiotemporal features of the final node graph of the pedestrian trajectory, construct the original spatiotemporal feature graph of the pedestrian trajectory based on the spatiotemporal features, and use the graph channel attention mechanism to screen out important spatiotemporal features to form a new spatiotemporal feature graph; S4: inputting the new spatiotemporal feature map into a predictor, which outputs a predicted pedestrian trajectory within a preset time, wherein the predictor adopts a time extrapolation neural network, and the predicted pedestrian trajectory includes a variety of different results; S5: assigning weights to the predicted pedestrian trajectories, and taking the trajectory with the largest weight as the final prediction result, wherein the weight assignment is completed by a time channel attention mechanism.

2. The method for predicting pedestrian trajectories using a graph neural network based on an attention mechanism according to claim 1, characterized in that: The original node graph of the pedestrian trajectory is represented as: Gt=(V t ,E t ), where V t Represents the relevant information of the nodes of the pedestrian trajectory graph of the tth frame, E t Represents the relevant information of the edges of the pedestrian trajectory graph of the tth frame, where the edges represent the interaction relationship between nodes.

3. The method for predicting pedestrian trajectories using a graph neural network based on an attention mechanism according to claim 2, characterized in that: When there is an edge between two nodes, a weight is assigned to the edge. The weight represents the intensity of the interaction between nodes and is generated by a kernel function, which is: in, Represents the Euclidean distance between node i and node j in the pedestrian trajectory graph of frame t.

4. The method for predicting pedestrian trajectories using a graph neural network based on an attention mechanism according to claim 1, characterized in that: The graph channel attention mechanism S2 adopts a graph convolution channel attention module to filter the structural information of the original node graph of the pedestrian trajectory, wherein the graph convolution channel attention module pays more attention to the relationship between channels, identifies and filters out important channel features through a learning mechanism, and suppresses unimportant channel features.

5. The method for predicting pedestrian trajectories using a graph neural network based on an attention mechanism according to claim 1, characterized in that: S3 uses the spatiotemporal graph convolutional neural network to extract the spatiotemporal features of the final node graph of pedestrian trajectories. It is necessary to transform the adjacency matrix A t Normalization, that is, Where the adjacency matrix A t is the edge weight The collection of Λ t is a diagonal matrix.

6. The method for predicting pedestrian trajectories using a graph neural network based on an attention mechanism according to claim 1, characterized in that: The expression of the convolution operation defined by the spatiotemporal graph convolutional neural network in S3 on a plane graph or feature graph is as follows: Wherein, σ represents the activation function, k represents the kernel size, p represents the sampling function, and l represents the number of layers; based on the expression of the convolution operation of the spatiotemporal graph convolutional neural network on the plane graph or feature graph, the convolution operation expression of the final node graph of the pedestrian trajectory is as follows: in, is a normalized term, B(v i )={v i |d(v i ,v j )≤D} is the neighbor set of the node.

7. The method for predicting pedestrian trajectories using a graph neural network based on an attention mechanism according to claim 1, characterized in that: S4: the predictor performs a convolution operation on the time dimension of the new spatiotemporal feature map.

8. The method for predicting pedestrian trajectories using a graph neural network based on an attention mechanism according to claim 1, characterized in that: The temporal channel attention mechanism described in S5 not only takes into account the channel itself, but also the adjacent channels of the channel, and completes the calculation of the channel weight through one-dimensional convolution, where the number of adjacent channels is related to the size of the channel, and the exponential family function is used to process the mapping problem between the number of adjacent channels and the size of the channel, that is, C = φ(k) ≈ 2^(γ*kb), where k represents the number of adjacent channels and C represents the size of the channel. The adaptive algorithm expression of k is as follows:

9. The method for pedestrian trajectory prediction based on graph neural network with attention mechanism according to claim 1, characterized in that: In the plane space, assuming that the random variable of the probability distribution of the location of a pedestrian at a certain moment follows a bivariate normal distribution, the loss function of the graph neural network based on the attention mechanism is as follows: Among them, V contains all the trainable parameters in the graph neural network, represents the mean of the probability distribution of the possible positions of pedestrians at a certain moment, represents the variance of the distribution, represents the correlation of the distribution.

10. A computer-readable storage medium, characterized in that: The storage medium contains the graph neural network pedestrian trajectory prediction method based on the attention mechanism as described in any one of claims 1 to 9.

Images