Pedestrian trajectory prediction method based on GCN interactive modeling

Abstract

The invention discloses a pedestrian trajectory prediction method based on GCN interactive modeling, and the method comprises the steps: defining each pedestrian as a node, and building a pedestrian space diagram according to the relative position of the pedestrian, so as to capture the space relation between the pedestrians; in the pedestrian space diagram, speed information is added for pedestrians, and a speed adjacency matrix is constructed; in the pedestrian space diagram, the categories of objects around pedestrians are used as labels to be embedded, and a label adjacent matrix is constructed; combining the speed adjacency matrix with the label adjacency matrix to generate a semantically guided adjacency graph; inputting the adjacent graph into a space-time graph convolution network, carrying out space-time convolution operation on the graph representation of the pedestrian trajectory, and respectively extracting a time feature and a space feature; and inputting the obtained features into an improved time extrapolation pyramid convolutional neural network, and jointly estimating two-dimensional Gaussian distribution parameters of a future trajectory through multi-layer prediction features so as to obtain a pedestrian trajectory. According to the method, the influence degrees of different objects are considered, different label classifications are designed for different objects, and the social attributes are fully utilized to calculate the weight of social interaction, so that a better pedestrian prediction result is realized.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving of pedestrian trajectory prediction, and in particular to a pedestrian trajectory prediction method based on GCN interactive modeling. Background Art

[0002] In the field of autonomous driving, accurately predicting pedestrian trajectories is one of the current research hotspots, because accurate pedestrian trajectory prediction helps autonomous vehicles better plan driving routes. In pedestrian trajectory prediction, it is crucial to effectively model pedestrian spatial interactions. In the process of pedestrian movement, considering that pedestrian trajectory prediction is usually carried out in complex scenes, it is necessary to model the environment, including road structure, obstacles, and interactions between pedestrians. At the same time, since pedestrian movement has a certain degree of uncertainty, it is necessary to consider the modeling and processing of uncertainty in trajectory prediction to improve the accuracy and robustness of the prediction.

[0003] In previous research results, Social-LSTM uses a pooling layer to learn temporal trajectory dependencies; SocialGAN increases the diversity of trajectories through the generative adversarial network (GAN) model, but the network training is unstable and prone to collapse. The scalability of models based on these two is not suitable for scenarios with multiple classes. As one of the mainstream methods for modeling pedestrian spatial interactions, graph convolutional neural networks are very effective for graph data processing. There is still a certain gap between the existing work on pedestrian spatial interaction modeling based on GCN and the actual situation. Although it successfully simulates the influence of surrounding neighbors based on relative distance, it ignores the influence of implicit correlations between different types of road users on the trajectory to be predicted. Nearby pedestrians and cars have different degrees of influence on the same pedestrian, so there will be deviations in the prediction of pedestrian trajectories. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a pedestrian trajectory prediction method based on GCN interaction modeling, which takes into account the influence degree of different objects, designs different label classifications for different objects, and makes full use of social attributes to calculate the weight of social interaction to achieve better pedestrian prediction results.

[0005] The present invention achieves the above technical objectives through the following technical means.

[0006] A pedestrian trajectory prediction method based on GCN interactive modeling includes the following steps:

[0007] Step 1: Define each pedestrian as a node and build a pedestrian spatial graph based on the relative positions of pedestrians to capture the spatial relationship between pedestrians;

[0008] Step 2: In the pedestrian space graph, add speed information for pedestrians and construct a speed adjacency matrix;

[0009] Step 3: In the pedestrian spatial graph, embed the categories of objects around the pedestrian as labels and construct a label adjacency matrix;

[0010] Step 4: Combine the speed adjacency matrix with the label adjacency matrix to generate a semantically guided adjacency graph;

[0011] Step 5: Input the adjacency graph into the spatiotemporal graph convolutional network, perform spatiotemporal convolution operations on the graphical representation of pedestrian trajectories, and extract temporal features and spatial features respectively;

[0012] Step 6: Input the temporal features and spatial features into the improved time extrapolation pyramid convolutional neural network (TXPCNN), and jointly estimate the two-dimensional Gaussian distribution parameters of the future trajectory through multiple layers of prediction features to obtain the pedestrian trajectory.

[0013] Furthermore, the method of establishing a pedestrian space graph is as follows: In the pedestrian space graph, the vertex represents each pedestrian, and the edge represents the interaction between pedestrians, so the pedestrian space graph G t Defined as:

[0014] G t =(V t ,E t ),

[0015] Among them, V t It is Figure G t The set of all vertices in is represented by For Figure G t The i-th vertex in the graph G t The number of vertices in E t It is Figure G t The set of edges is represented by G t The edge between the i-th vertex and the j-th vertex in G, N is the graph G t The number of vertices in .

[0016] Furthermore, step 2 calculates the speed based on the Euclidean distance per unit time of the pedestrian; then the speed of each object based on the relative position is expressed as:

[0017]

[0018] Among them, u t is the set of all pedestrian velocities in the spatial graph, represents the speed of the i-th pedestrian, is the x-coordinate of the ith pedestrian at the tth time step, is the y coordinate of the ith pedestrian at the tth time step.

[0019] Furthermore, by setting a distance threshold, if the Euclidean distance between pedestrians is greater than the set distance threshold, it is considered that the pedestrians are far apart, and pedestrians at a long distance are excluded.

[0020] Further, the process of step 3 is:

[0021] Step 3.1, set corresponding category labels for objects around pedestrians, and the category labels are represented by graphs;

[0022] Step 3.2, embed the category label into the label adjacency matrix;

[0023] Step 3.3: Use the attention mechanism to capture social influence from neighboring objects.

[0024] Furthermore, the method of combining matrices is as follows: two matrices are connected in the feature dimension, and the connected matrix is ​​passed to the fully connected layer to obtain a new adjacency graph in the same scenario.

[0025] Furthermore, the network structure of the spatiotemporal graph convolutional network is as follows: the adjacency graph is subjected to graph convolution and temporal convolution respectively to obtain the spatial graph and temporal graph representing the graph features, and the corresponding spatial and temporal matrices are generated by introducing the Embedding layer.

[0026] Furthermore, the improved time extrapolation pyramid convolutional neural network structure includes 5 Conv modules, among which 4 Conv modules are connected in series in sequence, and the Conv module at the head end is the input end, which concatenates the received time features and spatial features of the same dimension and inputs them, and the input time features and spatial features are sequentially processed by convolution of the 4 Conv modules; and the outputs of the 4 Conv modules are connected and input into the fifth Conv module for convolution operation, and finally the predicted features are output.

[0027] Furthermore, the label adjacency matrix is ​​constructed by creating an adjacency matrix with trainable parameters and embedding the category labels into the adjacency matrix.

[0028] Furthermore, the method for obtaining social influence in step 3.3 is as follows: analyzing the interaction between the position, speed, and direction of the target pedestrian and surrounding objects in the attention mechanism; obtaining the pedestrian social interaction weight according to the intensity of the mutual influence between the two nodes, and obtaining the degree of influence of one type of object on another type by embedding the supervised training class labels.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) In response to recent pedestrian trajectory prediction work, which simulates the influence of surrounding neighbors based on relative distance and ignores the impact of implicit correlations between different types of road users on the predicted trajectory, this method constructs a social interaction weight model based on pedestrian speed and position information and the impact of different road users on pedestrians, which can better weigh the impact between different pedestrians.

[0031] (2) Since the previous method only considered transmitting features to the multi-layer TXPCNN and only considered simple stacking, it could not effectively utilize the feature information. Therefore, this method uses the feature information of the previous layer and combines the feature information to jointly predict the parameters of the bivariate Gaussian distribution. It can make full use of the acquired pedestrian feature information and lay a good foundation for the next step of prediction.

[0032] (3) Pedestrian trajectory prediction is of great significance to the development and popularization of autonomous driving technology, and helps to improve driving safety, traffic efficiency and user experience. In terms of the effectiveness of predicting pedestrian trajectories, the GCN-based pedestrian social interaction modeling method for pedestrian trajectory prediction proposed in the present invention can effectively improve the accuracy of pedestrian trajectory prediction compared with the existing GCN-based modeling method.

[0033] (4) The present invention can better predict individual trajectories by passing the temporal and spatial dependency information of pedestrians into graph convolution and temporal convolution networks, thereby improving the effect of pedestrian trajectory prediction. Compared with the existing GCN-based pedestrian spatial interaction modeling method, the accuracy of pedestrian trajectory prediction can be significantly improved. For this purpose, a GCN-based pedestrian spatial interaction modeling method for pedestrian trajectories is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the flow of a pedestrian trajectory prediction algorithm based on GCN pedestrian spatial modeling according to an embodiment of the present invention;

[0035] Figure 2 Schematic diagram of the spatiotemporal graph convolutional network structure.

[0036] Figure 3 Schematic diagram of the improved temporal extrapolation pyramid convolutional neural network (TXPCNN) structure.

[0037] Figure 4 Compare the ADE results of different methods.

[0038] Figure 5 Compare the results of FDE for different methods. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a pedestrian trajectory prediction method based on GCN interactive modeling, which effectively predicts pedestrian trajectories based on the relative positions of pedestrians, combined with the speed of pedestrians and the labels of different road users.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] In the scene of pedestrian movement, there are a large number of pedestrians and their interactions are irregular, and the surrounding environment needs to be modeled in detail, including road signs and obstacles. Since pedestrian movement has certain uncertainty, the modeling and processing of uncertainty must be considered in trajectory prediction to improve the accuracy and robustness of the prediction. Graph convolutional neural networks are very effective for processing graph data when modeling pedestrian spatial interactions. However, the current pedestrian spatial interaction modeling based on graph convolutional neural networks has some deviations from the actual situation, does not fully utilize social attributes to calculate the weight of social interactions, and does not consider the differences in the impact of different road users on pedestrian trajectories. Although some recent pedestrian trajectory prediction works have successfully simulated the influence of surrounding neighbors based on relative distance, they perform poorly in multi-class trajectory prediction. This is because they ignore the implicit correlations between different types of road users, which have different degrees of influence on the predicted trajectory. For example, if a person sees a car running very close, the action will have a greater impact than when a person sees another person running very close.

[0043] In the process of pedestrian movement, since the number of pedestrians is uncertain and the interactions between different pedestrians are irregular, the use of graph structures can more clearly express pedestrian interaction behaviors. Compared with the RNN method, the graph convolutional neural network is more effective in processing graph data. It can efficiently process complex topological relationships in traffic scenes, intuitively reflect the role of nodes and edges, and present more effective results. Therefore, the GCN-based method is very suitable for modeling spatial interactions between pedestrians. Therefore, refer to Figure 1 and Figure 2 In an embodiment of the present invention, the present invention provides a pedestrian trajectory prediction method based on GCN interactive modeling, comprising the following steps:

[0044] Step 1: Use pedestrians as nodes and the lines between pedestrians as edges to build a pedestrian spatial graph to capture the spatial relationship between pedestrians. The method to build the pedestrian spatial graph is:

[0045] Based on pedestrian trajectories, a set of pedestrian spatial graphs G is constructed t , which is used to represent the relative position of pedestrians at each time step t in the scene. In the pedestrian space graph, the vertex represents each pedestrian, and the edge represents the interaction between pedestrians, so the pedestrian space graph G t Defined as:

[0046] G t =(V t ,E t ),

[0047] Among them, V t It is Figure G t The set of all vertices in is represented by For Figure G t The i-th vertex in the graph G t The number of vertices in E t It is Figure G t The set of edges is represented by G t The edge between the i-th vertex and the j-th vertex in G, N is the graph G t The number of vertices in .

[0048] Step 2: Take the speed of pedestrians as one of the attributes of the node, add information about speed in the pedestrian space graph, and construct a speed adjacency matrix to reflect the degree of speed correlation between pedestrians.

[0049] Assume that there are N pedestrians in a given scene, and the relative positions between pedestrians are represented by a spatial graph. The speed is calculated using the Euclidean distance of pedestrians per unit time. Then the speed of each object based on the relative position is expressed as:

[0050]

[0051] Among them, u t is the set of all pedestrian velocities in the spatial graph, represents the speed of the i-th pedestrian, is the x-coordinate of the ith pedestrian at the tth time step, is the y coordinate of the ith pedestrian at the tth time step.

[0052] In this application, pedestrians that are far away have less impact on the target pedestrian trajectory, so pedestrians at a long distance can be excluded. Therefore, by setting a distance threshold, if the Euclidean distance between pedestrians is greater than the set distance threshold, it is considered that the pedestrians are far away.

[0053] Step 3: Since pedestrian trajectories depend not only on the distance to other objects but also on the types of objects they interact with, object labels need to be taken into account during trajectory prediction. The specific process is as follows:

[0054] Step 3.1, set corresponding category labels for objects around pedestrians, and the category labels are represented by graphs;

[0055] Step 3.2, embed the category label into the label adjacency matrix;

[0056] Step 3.3, use the attention mechanism to obtain social influence from neighboring objects. The specific process is: analyze the interaction between the social factors such as the position, speed, and direction of the target pedestrian and the surrounding objects in the attention mechanism; obtain the pedestrian social interaction weight according to the intensity of the mutual influence between the two nodes, and obtain the degree of influence of one type of object on another by embedding the supervised training class label. For example, if a person sees a car approaching, the action will have a greater impact than seeing another person approaching. Therefore, modeling the label information of different types of objects can greatly affect the relationship between trajectories.

[0057] More specifically, objects around pedestrians include different types such as pedestrians, cyclists, etc., and corresponding category labels are set for these objects.

[0058] More specifically, the label adjacency matrix is ​​constructed by creating an adjacency matrix with trainable parameters and embedding the category labels into the adjacency matrix.

[0059] Step 4: Combine the speed adjacency matrix and the label adjacency matrix to generate a semantically guided adjacency graph. The present invention uses a semantically guided adjacency graph to represent a speed graph with label information, which can improve the accuracy of trajectory prediction.

[0060] The method of combining matrices is to connect the two matrices in the feature dimension and pass the connected matrix to the fully connected layer to obtain a new adjacency graph in the same scenario.

[0061] Step 5: Pass the adjacency graph obtained in step 4 into the spatiotemporal graph convolutional network, perform spatiotemporal convolution operations on the graphical representation of the pedestrian trajectory, and extract the temporal features and spatial features respectively.

[0062] The network structure of the spatiotemporal graph convolutional network is as follows Figure 2 As shown in the figure, the adjacency graph is subjected to graph convolution and time convolution respectively to obtain the spatial graph and time graph representing the graph features, and the corresponding spatial and time matrices are generated by introducing the Embedding layer.

[0063] Step 6: Input the temporal features and spatial features into the improved time extrapolation pyramid convolutional neural network (TXPCNN), and jointly estimate the two-dimensional Gaussian distribution parameters of the future trajectory through multiple layers of prediction features to obtain the pedestrian trajectory.

[0064] The improved time extrapolation pyramid convolutional neural network structure is as follows Figure 3 As shown, it includes 5 Conv modules, among which 4 Conv modules are connected in series in sequence. The Conv module at the head end is the input end, which concatenates the received time features and spatial features of the same dimension and inputs them. The input time features and spatial features are sequentially processed by convolution of the 4 Conv modules; and the outputs of the 4 Conv modules are connected and input into the fifth Conv module for convolution operation, and finally the predicted features are output.

[0065] In order to verify the effect of the pedestrian trajectory prediction method based on GCN interactive modeling proposed in the present invention, the method is verified and analyzed in combination with specific data as follows.

[0066] First, the public datasets ETH and UCY contain the movement trajectories of pedestrians in different environments, from which we can obtain the location information and movement direction of each pedestrian. Preprocess the data and convert it into a form acceptable to the neural network model. Divide the data into three parts: training set, test set, and validation set. The position and movement direction of each pedestrian at each time step are used as input features, and the position of the pedestrian at the next time step is used as the output label.

[0067] This method uses the ETH and UCY datasets. The ETH dataset contains two scenes, each with 750 different pedestrians. The UCY dataset contains 786 people. The dataset has three components: ZARA01, ZARA02, and UCY. Pedestrian trajectory prediction is performed on these five sets of data. This dataset represents the real world with multiple nonlinear trajectories and also covers challenging group behaviors such as couples walking together, groups crossing each other, and groups forming and dispersing in certain scenarios.

[0068] The experiment set a training batch size of 128 and used stochastic gradient descent (SGD) to train the model for 250 cycles. The initial learning rate was 0.01 and changed to 0.002 after 150 cycles. The ADE and FDE evaluation indicators are shown in Table 1 below:

[0069] Table 1 Preliminary experimental results

[0070] ETH HOTEL UNIV ZARA1 ZARA2 AVG↓ Linear 1.33 / 2.94 0.39 / 0.72 0.82 / 1.59 0.62 / 1.21 0.77 / 1.48 0.79 / 1.59 S-GAN-P 0.87 / 1.62 0.67 / 1.37 0.76 / 1.52 0.35 / 0.68 0.42 / 0.84 0.61 / 1.21 STSGN 0.75 / 1.63 0.63 / 1.01 0.48 / 1.08 0.30 / 0.65 0.26 / 0.57 0.48 / 0.99 OURS 0.67 / 1.27 0.41 / 0.64 0.48 / 0.86 0.34 / 0.56 0.32 / 0.48 0.44 / 0.76

[0071] The experimental results are as follows Figure 4 , 5 As shown by Figure 4 , 5 From the line graph, we can see that compared with other methods, the evaluation index values ​​obtained by this method are smaller, the overall effect is better, and the accuracy of predicting pedestrian trajectories is improved.

[0072] In summary, a pedestrian trajectory prediction method based on GCN interactive modeling can effectively improve the accuracy of pedestrian trajectory prediction. By accurately predicting the movement trajectory of pedestrians, the autonomous driving system can better avoid collisions with pedestrians, thereby improving driving safety. At the same time, predicting the route and dynamics of pedestrians can help autonomous driving vehicles better plan routes, avoid conflicts with pedestrians, and improve traffic efficiency.

[0073] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A pedestrian trajectory prediction method based on GCN interactive modeling, characterized in that: The steps include: Step 1: Define each pedestrian as a node and build a pedestrian spatial graph based on the relative positions of pedestrians to capture the spatial relationship between pedestrians; Step 2: In the pedestrian space graph, add speed information for pedestrians and construct a speed adjacency matrix; Step 3: In the pedestrian space graph, embed the categories of objects around the pedestrian as labels and construct a label adjacency matrix; Step 4: Combine the speed adjacency matrix with the label adjacency matrix to generate a semantically guided adjacency graph; Step 58: Input the adjacency graph into the spatiotemporal graph convolutional network, perform spatiotemporal convolution operations on the graphical representation of the pedestrian trajectory, and extract temporal features and spatial features respectively; Step 6: Input the temporal features and spatial features into the improved time extrapolation pyramid convolutional neural network, and jointly estimate the two-dimensional Gaussian distribution parameters of the future trajectory through multiple layers of prediction features to obtain the pedestrian trajectory.

2. According to the method of pedestrian trajectory prediction based on GCN interactive modeling according to claim 1, it is characterized in that: The method of establishing a pedestrian space graph is as follows: In the pedestrian space graph, vertices represent each pedestrian, and edges represent the interaction between pedestrians. Therefore, the pedestrian space graph G t Defined as: G t =(V t ,E t ), Among them, V t It is Figure G t The set of all vertices in is represented by For Figure G t The i-th vertex in the graph G t The number of vertices in E t It is Figure G t The set of edges is represented by G t The edge between the i-th vertex and the j-th vertex in G, N is the graph G t The number of vertices in .

3. The pedestrian trajectory prediction method based on GCN interactive modeling according to claim 1 is characterized in that: Step 2 calculates the speed based on the Euclidean distance per unit time of the pedestrian; then the speed of each object based on relative position is expressed as: Among them, u t is the set of all pedestrian velocities in the spatial graph, represents the speed of the i-th pedestrian, is the x-coordinate of the ith pedestrian at the tth time step, is the y coordinate of the ith pedestrian at the tth time step.

4. The pedestrian trajectory prediction method based on GCN interactive modeling according to claim 3 is characterized in that: By setting a distance threshold, if the Euclidean distance between pedestrians is greater than the set distance threshold, it is considered that the pedestrians are far apart and pedestrians at a long distance are excluded.

5. The pedestrian trajectory prediction method based on GCN interactive modeling according to claim 1 is characterized in that: The process of step 3 is: Step 3.1, set corresponding category labels for objects around pedestrians, and the category labels are represented by graphs; Step 3.2, embed the category label into the label adjacency matrix; Step 3.3: Use the attention mechanism to capture social influence from neighboring objects.

6. The pedestrian trajectory prediction method based on GCN interactive modeling according to claim 1 is characterized in that: The method of combining matrices is to connect two matrices in the feature dimension and pass the connected matrix to the fully connected layer to obtain a new adjacency graph in the same scenario.

7. The pedestrian trajectory prediction method based on GCN interactive modeling according to claim 6 is characterized in that: The network structure of the spatiotemporal graph convolutional network is as follows: the adjacency graph is subjected to graph convolution and temporal convolution respectively to obtain the spatial graph and temporal graph representing the graph features, and the corresponding spatial and temporal matrices are generated by introducing the Embedding layer.

8. The pedestrian trajectory prediction method based on GCN interactive modeling according to claim 1 is characterized in that: The improved time extrapolation pyramid convolutional neural network structure includes 5 Conv modules, among which 4 Conv modules are connected in series in sequence. The Conv module at the head end is the input end, which concatenates the received time features and spatial features of the same dimension and inputs them. The input time features and spatial features are processed by convolution of the 4 Conv modules in sequence; and the outputs of the 4 Conv modules are connected and input into the fifth Conv module for convolution operation, and finally the predicted features are output.

9. The pedestrian trajectory prediction method based on GCN interactive modeling according to claim 1 is characterized in that: The label adjacency matrix is ​​constructed by creating an adjacency matrix with trainable parameters and embedding the category labels into the adjacency matrix.

10. The pedestrian trajectory prediction method based on GCN interactive modeling according to claim 5 is characterized in that: The method of obtaining social influence in step 3.3 is as follows: analyze the interaction between the position, speed, and direction of the target pedestrian and the surrounding objects in the attention mechanism; obtain the pedestrian social interaction weight according to the intensity of the mutual influence between the two nodes, and obtain the degree of influence of one type of object on another type by embedding the supervised training class label.

Images