A spatial graph attention mechanism pedestrian trajectory prediction method combining adaptive weighted fusion mechanism and pedestrian walking behavior specification

Abstract

The application provides a pedestrian trajectory prediction method combining an adaptive weighted fusion mechanism and a spatial graph attention mechanism of pedestrian walking behavior norms. First, the time sequence characteristics of the pedestrian motion trajectory are accurately extracted through a BiLSTM network fused with an adaptive weighting mechanism, and the flexibility and accuracy of feature extraction are enhanced. Second, the spatial graph attention mechanism is used to construct the preliminary interaction characteristics between the target pedestrian and the surrounding pedestrians, and the spatial graph attention characteristics of the target pedestrian and the surrounding pedestrians are obtained through the mechanism, so that the dynamic interaction relationship between the pedestrians can be accurately captured. Then, under the constraint of the pedestrian walking behavior norms, the spatial attention interaction characteristics are subjected to secondary dynamic programming, so that the trajectory prediction result is more in line with the actual pedestrian behavior mode. Finally, through the fusion of various trajectory characteristics, the current state of the pedestrian is mapped to the future trajectory by using a sequence-to-sequence model (Seq2Seq), so that high-precision pedestrian trajectory prediction is realized.

Description

Technical fields:

[0001] This invention relates to the field of trajectory prediction technology, and more specifically to pedestrian trajectory prediction based on graph neural networks. Technical background:

[0002] Due to the uncertainty of pedestrian movement, the complexity of group movement, and the influence of environmental factors, the problem of predicting the trajectory of dynamic objects has always been complex. When mobile robots navigate through dense crowds, they need to consider the current movement trajectories of surrounding people and predict their future trajectories. This allows for the planning of the mobile robot's future path, ensuring safer movement and smoother planned paths.

[0003] Traditional pedestrian trajectory prediction methods often rely on building social models among pedestrians to predict trajectories. These models utilize the attraction between pedestrians and their destination, as well as the repulsion between pedestrians, to drive the prediction of their approximate direction. However, this method cannot predict situations where pedestrians are walking in groups. Subsequent methods employed recurrent neural networks, such as RNNs and LSTMs, to predict trajectories based on historical trajectory features. These methods, however, lack consideration of the interaction between pedestrians. Pedestrian movement is not an isolated activity but rather involves interaction with all surrounding objects. Furthermore, the interactions between dynamic groups of pedestrians in space vary. Considering the walking distance constraint, closer pedestrians within the walking range have a greater influence on the target pedestrian's future movement decisions. For example, the closer a person is to a target pedestrian, the greater the impact on their future trajectory. Therefore, pedestrian trajectory prediction can dynamically consider the influence of surrounding pedestrians on the target individual based on their walking range, thereby significantly improving the accuracy of trajectory prediction. Summary of the Invention:

[0004] This invention aims to overcome the problems of existing technologies that lack consideration of the spatial interaction between the target pedestrian and surrounding pedestrians, as well as the problem of pedestrian historical trajectory feature extraction. By combining the temporal feature extraction characteristics of a recurrent neural network module with an adaptive weighted fusion mechanism and the spatial interaction feature extraction characteristics of a graph spatial attention neural network, a novel pedestrian trajectory prediction method integrating recurrent neural network and graph spatial attention neural network mechanisms is proposed. This method utilizes the historical motion features of pedestrians and the attention features of surrounding pedestrians, while also combining the generated trajectory interaction features of surrounding pedestrians. By setting a weight factor to influence pedestrian walking norms, the motion trajectories and attention features of surrounding pedestrians are added to the trajectory prediction module to predict the trajectory of the target pedestrian, thereby improving the accuracy of the pedestrian trajectory prediction algorithm. Attached image description:

[0005] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0006] Figure 1 The trajectory prediction framework based on graph neural networks of the present invention is shown in the figure.

[0007] Figure 2 A schematic diagram illustrating the pedestrian trajectory prediction process for implementing the present invention; Detailed implementation method:

[0008] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0009] like Figure 1 As shown, a pedestrian trajectory prediction method combining an adaptive weighted fusion mechanism and a spatial graph attention mechanism based on pedestrian walking behavior norms includes the following steps:

[0010] S1. Obtain the temporal characteristics of the target population to be predicted, including the following steps:

[0011] S101. Calculate the relative position of the target pedestrian to surrounding obstacles, including the relative distance and direction of the target pedestrian to each obstacle. This information is represented in a Cartesian coordinate system to capture the spatial relationship between the target and obstacles. For example, for pedestrian i, we need to capture the pedestrian's position at observation time t. obs The persistent spatial coordinate position, where the spatial coordinate position of pedestrian i at a single time t is defined as follows: The pedestrian's observation time in the scene is t obs The spatial position coordinate sequence within is Its value is defined as:

[0012] S102. Obtain the velocity and acceleration information of the target pedestrian and obstacles. Preprocessing the motion states of the target pedestrian and surrounding obstacles will help predict their future interactions. Based on the position, velocity, and acceleration relationships between the target pedestrian and surrounding obstacles, a temporal feature vector of a single graph node is constructed. The definition of a single temporal feature vector is as follows: in This represents the velocity of person i on the x-axis at time t. Represents the velocity of person i on the y-axis at time t. Represents the acceleration of individual i on the x-axis at time t. Represents the acceleration on the y-axis of individual i at time t. Indicates the orientation of person i at time t, when hour, when hour,

[0013] S103. Traditional unidirectional LSTM networks can only extract features from one direction of the time series (usually past to present), causing the model to only utilize information from a single direction for feature extraction. However, since pedestrian trajectories have bidirectional time dependencies, ignoring information from the reverse time series will lead to a decrease in prediction accuracy.

[0014] S104. By simultaneously introducing forward and backward LSTMs, bidirectional dependencies in time series are captured. The forward LSTM in BiLSTM is responsible for extracting features from the past to the present of the time series, and the generated output features are represented as h. forward In BiLSTM, the backward LSTM is responsible for extracting features from the future to the past, and the generated output features are represented as h. backward .

[0015] S105. Introduce trainable weight parameters α and β, which correspond to the output features of the forward LSTM and backward LSTM, respectively. Through these weight parameters, the model can automatically learn the optimal fusion ratio of the forward and backward features during training. The final fused feature h... fused Calculated using the weighted summation formula: h fused =α·h forward +β·h backward (3) The weight parameters α and β are not static, but dynamically adjusted based on the features of the input data and the model's learning progress. This adaptive property allows the model to appropriately allocate the weights of forward and backward features at different time steps and under different input conditions.

[0016] S106. During model training, the weight parameters α and β, along with other model parameters, are optimized using gradient descent. The goal is to minimize the loss function of the downstream task, thereby finding the optimal weight configuration.

[0017] S107. By automatically adjusting the weights of forward and backward features, this mechanism can flexibly handle the extraction of temporal features in different scenarios and adapt to diverse data features. The fused features not only contain the historical information of the forward LSTM and the future information of the backward LSTM, but also find the best balance between the two.

[0018] S108. Finally, the historical motion trajectories of the target pedestrian and surrounding pedestrians at T time points will be collected. The improved adaptive fusion mechanism of the BiLSTM module will be used to extract the temporal feature vectors from the preprocessed motion history trajectories of each pedestrian in the collected group, thus obtaining the temporal features f of the target population. a.

[0019] S2. Obtain the interaction features f between the target pedestrian and surrounding pedestrians. b It includes the following steps:

[0020] S201. Represent pedestrians and surrounding obstacles in a crowd as graph nodes. Each node has a feature vector containing information about that node, such as position, velocity, and acceleration. This information can be obtained using sensor data, such as cameras or LiDAR.

[0021] S202. Construct a graph structure using the relationship between pedestrians and surrounding pedestrians, and establish a dynamic graph G over time using the Euclidean distance between the absolute positions of pedestrians. t G t =(V t E t ),in This represents the number of interactions between pedestrian i and surrounding pedestrians at the current time t, where n represents the number of pedestrians in the scene. This indicates whether there is interaction between pedestrians in the scene. Its value is defined as: In this graph structure, nodes represent the target pedestrian and surrounding pedestrians, and edges represent the interaction relationships between them. Let f represent the Euclidean distance between pedestrian i and pedestrian j at time t. ε is a manually set hyperparameter. When the distance between pedestrians is greater than the set ε, it is determined that there is no interaction between them. Based on this method, the basic interaction characteristics f between the target pedestrian and surrounding pedestrian nodes are obtained. b .

[0022] S203. Obtaining spatial graph attention features of the target pedestrian and surrounding pedestrians based on spatial attention mechanism. c It includes the following steps:

[0023] S204. A graph attention neural network (GAT) is applied to focus on important information in the graph. An attention mechanism is used to aggregate neighboring nodes, achieving adaptive allocation of weights for different neighbors. The output of the GAT is used to aggregate node features. This is achieved by weighted summation of node features, with weights provided by the GAT. The initial input is the fused features of the pedestrian node feature vectors, where is defined as... w c It is a weight matrix, the purpose of which is to combine the features of the target pedestrian nodes. The transformation proceeds to a new feature space with a new feature vector dimension of D′. || represents concatenating the transformed feature representations of the target pedestrian and its neighboring pedestrians to obtain the final feature vector. Its dimensions are

[0024] S205. An attention matrix between nodes is established based on node feature fusion. At time t, the attention of pedestrian graph node i to its surrounding pedestrian graph node j is... This indicates that its value is based on the spatial interaction of pedestrian i at time t. Corresponding node features Assign different weights to the neighboring pedestrians of the target pedestrian i. In the above formula, σ is a nonlinear activation function, and exp represents an exponential function used to perform softmax normalization. T It is a weight vector, the purpose of which is to... The matrix multiplication yields the attention coefficients of the surrounding neighbors i to the target pedestrian j at time t. This refers to pedestrian i whose surroundings are connected to him by edges in the graph, and who interact with him in space.

[0025] S206. For pedestrian i, the features after processing by the graph attention mechanism are: Finally, after multiple training iterations and backpropagation of the training parameters, the feature vector value of a single node i is output using a graph attention neural network (GAT) as follows:

[0026] S207. Extract the interaction features between the target pedestrian and surrounding obstacles from the aggregated features. This can include the association strength between the target pedestrian and other nodes, relative position, speed difference, etc. In the above formula, W... gat These are all the training parameters for the spatial graph attention neural network GAT.

[0027] S3. Integrate the spatial graph attention features of the target pedestrian's walking behavior constraints, and perform secondary dynamic programming on the spatial attention features of the target pedestrian, including the following steps:

[0028] S301. Here, the focus is on the circular area within one meter of the target pedestrian's walking distance; this area becomes the region of interest in the spatial graph. Based on pedestrian walking behavior norms, the focus is on a circular area with a radius of 1m around the target pedestrian. Among the surrounding pedestrians within the region of interest, a dynamically weighted assignment is used to update the adjacency matrix of interaction relationships. This indicates that the model dynamically adjusts the association strength between pedestrians based on their relative positions and other dynamic factors.

[0029] S302. Designate a circular area with a radius of 1m as the focus area for the target pedestrians. Within this focus area, create a pedestrian interaction matrix that includes the interaction relationships between pedestrians. The pedestrian interaction matrix values ​​within this area... The value is dynamically assigned based on the distance to the target pedestrian. The value represents the distance between pedestrian node i and pedestrian node j at time t, and is defined as: and This represents the distance between all pedestrian nodes in the space and pedestrian node i. The spatial graph attention interaction features, which are fused with the target pedestrian walking behavior constraints, are specifically defined as: Formula (9) assigns values ​​based on dynamic constraints within the walking distance according to the interaction characteristics between the target pedestrian and the surrounding interactive objects. When the target predicted object is close to the surrounding pedestrians, the weight is assigned by collecting the distance between all the surrounding pedestrians and the target pedestrian. The value ranges from 0 to 1. Since the closer pedestrians have a greater impact on the trajectory of the target predicted pedestrian, the interaction characteristics are finally obtained by subtracting their values.

[0030] S303. The values ​​in the pedestrian interaction matrix are dynamically assigned based on their distance from the target pedestrian. This value is calculated using a distance function rule to ensure that pedestrians closer to the target pedestrian have a greater weight in the interaction relationship. Pedestrians outside this range are assumed to have a smaller impact on the predicted target pedestrian's interaction, and the spatial graph attention interaction features are not updated in this case.

[0031] S304. When processing spatial graph data, the application of spatial attention mechanisms is crucial in graph attention network models. This allows the model to selectively focus on specific regions rather than processing the entire spatial graph. The model's focus is on a circular area within one meter of the target pedestrian. In this way, the model can concentrate its attention on the area most relevant to the target pedestrian, thereby improving processing efficiency and accuracy.

[0032] S305, the graph attention network model will pay special attention to a circular area with a radius of 1 meter around the target pedestrian. The selection of this area is based on the characteristics of pedestrian behavior, because during the walking process, the surrounding pedestrians may have a direct impact on the behavior of the target pedestrian.

[0033] S306. Within this key area of ​​focus, the graph attention network model dynamically updates the interaction relationships between pedestrians. This dynamic update is achieved through dynamic weighted assignment, taking into account the relative positions of pedestrians and other dynamic factors. This approach allows the model to adapt more flexibly to pedestrian interactions in different environments, improving its applicability and generalization ability.

[0034] S307. Furthermore, the graph attention network model dynamically adjusts the weights of interaction relationships based on the distance to the target pedestrian. This adjustment is calculated based on a distance function rule, ensuring that pedestrians closer to the target pedestrian have a greater influence in the interaction relationship. In this way, the model can more accurately capture the environmental conditions around the target pedestrian and make more precise predictions.

[0035] S308. In summary, by combining spatial graph attention features with the walking behavior constraints of the target pedestrians, the graph attention network model can better understand the interaction relationships between pedestrians, thereby improving the accuracy and robustness of pedestrian prediction.

[0036] S4. The pedestrian trajectory features obtained in steps S1, S2, and S3 are fused, and a sequence-to-sequence (Seq2Seq) model is used to map the pedestrian's state over a certain period to their trajectory over a future period. This method includes the following steps:

[0037] S401, an adaptive BiLSTM (Bidirectional Long Short-Term Memory) network is used to extract pedestrian trajectory features more accurately.

[0038] S402 and BiLSTM, by simultaneously processing forward and backward information of the trajectory, enable the model to capture more comprehensive temporal dependencies. Furthermore, an adaptive mechanism enhances BiLSTM's capabilities, allowing it to dynamically adjust network parameters and weights based on the characteristics of different pedestrian trajectories, thereby achieving optimal feature extraction for various trajectory information. This method can more accurately extract temporal features from pedestrian trajectories, improving the accuracy of trajectory prediction and the model's generalization performance.

[0039] S403. Use a graph attention neural network (GAT) to process the different trajectories of surrounding pedestrians, thereby extracting the interactive information characteristics between them.

[0040] S404. Under the walking behavior constraint mechanism, GAT can accurately extract the interaction information between pedestrians by assigning different attention weights to the neighbor nodes of each pedestrian. In pedestrian trajectory prediction, the fusion of this constraint not only considers the characteristics of the target pedestrian, but also combines the interaction between surrounding pedestrians, especially when following certain walking behavior constraints. This fusion mechanism can more comprehensively capture the interaction patterns between pedestrians and more accurately represent the walking behavior between pedestrians.

[0041] S405. Utilizing an MLP (Multilayer Perceptron) mechanism to fuse temporal and interaction features allows for a more comprehensive modeling of pedestrian motion behavior. In this process, temporal features (extracted by an adaptive BiLSTM) capture the temporal dependencies in pedestrian trajectories, while interaction features (extracted by a GAT) describe the interactions between pedestrians. The extracted temporal and interaction features are then concatenated to form a comprehensive feature vector.

[0042] S406. Input the concatenated integrated feature vector into the MLP. The MLP consists of 3 fully connected layers, each followed by a non-linear activation function (such as ReLU) to capture the complex non-linear relationships between features.

[0043] S407 and MLP learn the higher-order correlation between temporal features and interaction features through multi-layer nonlinear mapping, generating a fused feature representation.

[0044] S408. Decoding of trajectory prediction: The feature vector obtained by fusing trajectory motion features and trajectory interaction features through the MLP mechanism is decoded using LSTM to finally complete the trajectory prediction. This step involves interpreting the encoded information into the final trajectory prediction result.

[0045] The above description represents preferred embodiments of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Modifications and variations made by those skilled in the art based on the core ideas of the present invention should be within the scope of protection of the appended claims.

Claims

1. A pedestrian trajectory prediction method combining an adaptive weighted fusion mechanism and a spatial graph attention mechanism based on pedestrian walking behavior norms, characterized in that, include: S1. A BiLSTM network based on an adaptive weighted fusion mechanism acquires temporal features of pedestrian motion trajectories. ; S2. Construct preliminary interaction features by utilizing the relationship between the target pedestrian and surrounding pedestrians. It also acquires spatial attention features of the target pedestrian and surrounding pedestrians through a spatial attention mechanism. ; S3. Utilize the spatial graph attention features obtained in step S2. Under the constraints of pedestrian walking behavior norms, a secondary dynamic programming of the spatial attention interaction characteristics of the target pedestrian is performed using a spatial graph attention mechanism. Step S3 includes the following steps: S301. Based on the pedestrian walking behavior norms, among the pedestrians in the area that the target pedestrian is focusing on when walking, dynamically weight the pedestrians within the area and update the adjacency matrix of the interaction relationship. S302. Traditional pedestrian trajectory prediction methods are typically based on data-driven models, focusing on directly inferring future movement paths from historical trajectories, neglecting the interaction between pedestrians within the walking range. By introducing a dynamic weighting factor allocation mechanism for pedestrians around the target prediction walking range, and considering the constraints of interaction factors, the prediction model can more accurately reflect the decision-making process of pedestrians in reality, thereby improving prediction accuracy, especially in complex scenarios, including intersections or congested sidewalks. The key area of ​​focus when pedestrians are walking is a radius of 1 for the target pedestrian. The circular area, and the pedestrian interaction matrix values ​​within this area. This will make the target pedestrian Based on the distance to the target pedestrian The distance is dynamically assigned, and its value is: Indicates in Pedestrian nodes at all times pedestrian nodes The distance between them is defined as: S303, The updated interaction matrix value is based on node characteristics. For the target pedestrian Neighbors and pedestrians Assign different weights ; S304. Based on the walking behavior range, dynamically assign different attention weights to surrounding pedestrians and simultaneously extract and learn multiple interaction features between pedestrians, including distance, relative speed, and direction of movement; by assigning different weights to the fused multiple interaction features within the walking area of ​​the target predicted pedestrian, the interaction influence between pedestrians can be reflected more effectively. S305. Pedestrian interaction characteristics exhibit different behavioral patterns at different distance ranges. The improved pedestrian trajectory prediction method flexibly adjusts the weights of these characteristics by combining their attention mechanism with the walking range distance, thereby adapting to complex dynamic behaviors in different scenarios and enabling the expression of avoidance behavior when walking in crowds or free movement in open environments. S306. Finally, based on the improved dynamic adjustment interaction features combined with the GAT mechanism, the influence relationship between pedestrians is captured more accurately, thereby generating more accurate feature extraction and weight allocation. S4. Using the pedestrian trajectory features obtained from S1, S2, and S3, the features are fused. S4 includes the following steps: S401. The temporal features and spatial attention features are concatenated and input into the decoder. The decoded features are obtained by decoding through the Long Short-Term Memory (LSTM) network and then processed through a fully connected layer. S402. Using a sequence-to-sequence Seq2Seq model, pedestrian trajectories for future periods are generated based on pedestrian state information within historical time periods.

2. The pedestrian trajectory prediction method according to claim 1, which combines an adaptive weighted fusion mechanism with a spatial graph attention mechanism based on pedestrian walking behavior norms, wherein step S1 includes the following steps: S101, Collection The historical movement trajectories of the target pedestrian and surrounding pedestrians at each moment are predicted, and the temporal features of the trajectories are extracted. S102. Traditional unidirectional LSTM networks can only extract features from one direction of the time series, which means the model can only use information from a single direction for feature extraction. Since pedestrian trajectories have bidirectional time dependencies, ignoring information from the reverse time series will lead to a decrease in prediction accuracy. A BiLSTM network with an adaptive fusion mechanism is proposed for pedestrian trajectory temporal feature extraction. By simultaneously introducing forward and backward LSTMs, bidirectional dependencies in the time series are captured. The forward LSTM in the BiLSTM is responsible for extracting features from the past to the present of the time series, and the generated output features are represented as follows: In BiLSTM, the backward LSTM is responsible for extracting features from the future back to the past, and the generated output features are represented as follows: ; S103. For pedestrian temporal feature extraction, trainable weight parameters are introduced. and These correspond to the output features of the forward LSTM and the backward LSTM, respectively; Using these weight parameters, the model can automatically learn the optimal fusion ratio of forward and backward features during training; the final fused features Calculated using the weighted summation formula: S104. Based on the adaptive weighted fusion mechanism, the features from these two directions are further integrated, enabling the model to consider the past and future movement patterns of pedestrians at the same time, thereby comprehensively improving the accuracy of pedestrian temporal feature extraction. S105, Finally, the collection will be completed. The historical motion trajectories of the target pedestrian and surrounding pedestrians at each time point are predicted. An improved adaptive fusion mechanism using a BiLSTM module is employed to extract temporal feature vectors from the preprocessed motion history trajectories of each pedestrian in the collected group, thus obtaining the temporal features of the predicted target group. .

3. The pedestrian trajectory prediction method according to claim 1, which combines an adaptive weighted fusion mechanism with a spatial graph attention mechanism based on pedestrian walking behavior norms, wherein step S2 includes the following steps: S201. Construct a graph structure by predicting the relationship between the target pedestrian and surrounding pedestrians. Treat each pedestrian as a node in the graph, and model the interaction behavior between pedestrians through edges to obtain preliminary interaction features of the relationship between the target pedestrian and surrounding pedestrians. ; S202. By using a graph attention mechanism, different importance weights are assigned to different interactions, thereby capturing the complex relationships between pedestrians more precisely, and simultaneously obtaining the spatial graph attention features of the target pedestrian and surrounding pedestrians. .

4. A pedestrian trajectory prediction method combining an adaptive weighted fusion mechanism and a spatial graph attention mechanism based on pedestrian walking behavior norms, as described in claim 1 or 2, is characterized in that... The pedestrian trajectory features obtained in steps S1, S2, and S3 are fused together. A method for mapping a pedestrian's state over a certain period of time to a trajectory over a future period of time using a sequence-to-sequence model (Seq2Seq), wherein S4 includes the following steps: Time series features Spatial attention features After concatenation, the data is fed into the decoder to obtain LSTM decoding features. Fully connected operations are performed on the decoding features, and finally, the Long Short-Term Memory (LSTM) network is used for decoding to complete trajectory prediction.

5. A pedestrian trajectory prediction method system combining an adaptive weighted fusion mechanism and a spatial graph attention mechanism based on pedestrian walking behavior norms, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, it implements the method as described in any one of claims 1-4.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the pedestrian trajectory prediction method that combines an adaptive weighted fusion mechanism with a spatial graph attention mechanism based on pedestrian walking norms as described in any one of claims 1 to 4.

Images