Space map attention mechanism pedestrian trajectory prediction method combining adaptive weighted fusion mechanism and pedestrian walking behavior specification
By combining the adaptive weighted fusion mechanism and the graph space attention neural network, considering the spatial interaction and historical trajectory characteristics between pedestrians, the problem of insufficient accuracy of pedestrian trajectory prediction in the existing technology is solved, and a more accurate pedestrian trajectory is achieved. Predictions have improved the navigation security of mobile robots in dense crowd environments.
Patent Information
- Application Number
- CN202411239120.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The prior art is difficult to effectively predict the spatial interaction and historical trajectory characteristics between pedestrians, resulting in insufficient accuracy in pedestrian trajectory prediction, especially when mobile robots navigate in dense crowd environments, it is difficult to ensure safe driving.
The recurrent neural network and graph space attention neural network adopt a fusion adaptive weighted fusion mechanism, combining pedestrian historical motion characteristics and peripheral feature attention characteristics, set the pedestrian walking specification influence weight factor, and improve the trajectory by generating the trajectory interaction characteristics of surrounding pedestrians. Precision algorithm accuracy.
By dynamically considering the impact of surrounding pedestrians, the accuracy and robustness of pedestrian trajectory prediction are improved, and the interaction patterns and walking behaviors between pedestrians can be captured more accurately, enhancing the navigation safety of mobile robots in dense crowd environments.
Smart Images

Figure BDA0005028681270000013 
Figure BDA0005028681270000021 
Figure BDA0005028681270000033
Abstract
Description
Technical field:
[0001] The present invention relates to the technical field of trajectory prediction, and in particular to pedestrian trajectory prediction based on graph neural network. Technical background:
[0002] Due to the uncertainty of pedestrian movement, the complexity of group movement and the influence of environmental factors, the problem of dynamic object trajectory prediction has always been a complex problem. When a mobile robot navigates in a dense crowd, it is necessary to consider the driving trajectory of the surrounding people in the current situation and predict the future driving trajectory of the surrounding pedestrians, so as to plan the future path of the mobile robot and drive safely, which can make the autonomous mobile robot move more safely and the planned path smoother.
[0003] Traditional pedestrian trajectory prediction methods predict trajectories by building a social model between pedestrians, and use the attraction between pedestrians and the destination and the repulsion between pedestrians to drive the prediction of the general direction of pedestrians. However, this method cannot predict the situation where pedestrians walk together. Subsequently, recurrent neural networks, such as RNN and LSTM, were used to predict based on the historical trajectory characteristics of pedestrians. These methods lack the interaction between pedestrians. The movement of pedestrians is not a single independent individual, but has an interactive relationship with all objects that may appear around them. At the same time, the interactions between dynamic pedestrian groups in space are different. Considering the walking distance constraint of pedestrians, the closer pedestrians within the walking range based on distance have a greater impact on the future movement decision of the target pedestrian. For example, the closer a person is to the target pedestrian, the greater the impact on his / her future trajectory. Therefore, in pedestrian trajectory prediction, the impact of surrounding pedestrians on the target predicted individual can be dynamically considered based on the distance of the pedestrian's walking range, which can greatly improve the accuracy of trajectory prediction. Summary of the invention:
[0004] The present invention aims to overcome the problem of lack of consideration of the spatial interaction between the target pedestrian and the surrounding pedestrians and the problem of pedestrian historical trajectory feature extraction in the prior art. By combining the temporal feature extraction characteristics of the recurrent neural network module integrating the adaptive weighted fusion mechanism and the spatial interaction feature extraction characteristics of the graph space attention neural network, a novel pedestrian trajectory prediction method integrating the recurrent neural network mechanism and the graph space attention neural network mechanism is proposed. The historical motion characteristics of pedestrians and the characteristic attention characteristics of the surrounding pedestrians are combined with the generated trajectory interaction characteristics of the surrounding pedestrians, and the weight factor affecting the pedestrian walking norms is set. The motion trajectory and attention characteristics of the surrounding pedestrians are added to the trajectory prediction module to perform trajectory prediction of the target pedestrian, so as to improve the accuracy of the pedestrian trajectory prediction algorithm. Description of the drawings:
[0005] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:
[0006] Figure 1 The trajectory prediction framework diagram based on graph neural network of the present invention;
[0007] Figure 2 A schematic diagram of a pedestrian trajectory prediction process for implementing the present invention; Specific implementation method:
[0008] The present invention will be further described below in conjunction with the accompanying drawings and specific implementations.
[0009] like Figure 1 As shown, a pedestrian trajectory prediction method combining an adaptive weighted fusion mechanism with a spatial graph attention mechanism of pedestrian walking behavior norms includes the following steps:
[0010] S1. Obtaining the time series characteristics of the target population to be predicted includes the following steps:
[0011] S101. Calculate the relative position between the target pedestrian and the surrounding obstacles, including the relative distance and direction from the target pedestrian to each obstacle. This information is expressed in a Cartesian coordinate system to capture the spatial relationship between the target and the obstacles. For example, for pedestrian i, we need to capture the pedestrian at observation time t obs The continuous spatial coordinate position is defined as the spatial coordinate position of pedestrian i at a single moment t. The pedestrian observes the time t in the scene. obs The spatial position coordinate sequence within is Its value is defined as:
[0012] S102, obtain the speed and acceleration information of the target pedestrian and the obstacle. Preprocessing the motion state of the target pedestrian and the surrounding obstacles will help predict their future interaction relationship, and form a time series feature vector of a single graph node based on the position, speed and acceleration relationship between the target pedestrian and the surrounding obstacles. The single time series feature vector is defined as: in represents the speed of person i on the x-axis at time t, represents the speed of person i on the y-axis at time t represents the acceleration of person i on the x-axis at time t represents the acceleration of person i on the y-axis at time t represents the direction of person i at time t, when hour, when hour,
[0013] S103. The traditional unidirectional LSTM network can only extract features from one direction of the time series (usually from the past to the present), which means that the model can only use information from a single direction for feature extraction. However, since pedestrian trajectories have bidirectional time dependencies, ignoring the information in the reverse time series will lead to a decrease in prediction accuracy.
[0014] S104, by introducing both forward and backward LSTM, the bidirectional dependencies in the time series are captured. The forward LSTM in BiLSTM is responsible for extracting features from the past to the present of the time series, and the generated output features are represented as h forward The backward LSTM in BiLSTM is responsible for extracting features from the future to the past, and the generated output features are represented as h backward .
[0015] S105, introduce trainable weight parameters α and β, which correspond to the output features of the forward LSTM and backward LSTM respectively. Through these weight parameters, the model can automatically learn the optimal fusion ratio of the forward and backward features during the training process. The final fusion feature h fused Calculated by weighted summation formula: h fused =α·h forward +β·h backward (3) The weight parameters α and β are not static, but are dynamically adjusted according to the characteristics of the input data and the learning status of the model. The adaptive characteristics enable the model to reasonably allocate the weights of the forward and backward features at different time steps and different input conditions.
[0016] S106. During the model training process, the weight parameters α and β are optimized together with other model parameters by gradient descent. The goal is to minimize the loss function of the downstream task and thus find the best weight configuration.
[0017] S107. By automatically adjusting the weights of forward and backward features, this mechanism can flexibly cope with time series feature extraction in different scenarios and adapt to diverse data features. The fused features not only contain the historical information of the forward LSTM and the future information of the backward LSTM, but also find the best balance between the two.
[0018] S108. Finally, the historical motion trajectories of the target predicted pedestrians and the surrounding pedestrians at T moments are collected, and the BiLSTM module with the improved adaptive fusion mechanism is used to extract the time series feature vector of the pre-processed motion history trajectory of each pedestrian in the collected group to obtain the time series feature f of the predicted target group. a.
[0019] S2. Obtain the interaction features f between the target pedestrian to be predicted and the surrounding pedestrians b , including the following steps:
[0020] S201. Pedestrians in the crowd and surrounding obstacles are represented as graph nodes. Each node has a feature vector, which contains information about the node, such as position, speed, acceleration, etc. Sensor data, such as cameras or lidar, can be used to obtain this information.
[0021] S202: Use the relationship between pedestrians and surrounding pedestrians to build a graph structure, and establish a dynamic graph G along time through the Euclidean distance between the absolute positions of pedestrians t , where G t =(V t ,E t ),in It represents the number of interactions between pedestrian i and the surrounding pedestrians at the current time t, and n represents the number of n pedestrians in the scene. Indicates whether there is interaction between pedestrians in the scene. Its value is defined as: In this graph structure, nodes represent target pedestrians and surrounding pedestrians, and edges represent the interaction relationships between them. represents the Euclidean distance between pedestrians i and j at time t, ε is a hyperparameter set artificially. When the distance between pedestrians is greater than the set ε, it is judged that there is no interaction between them. According to this method, the basic interaction feature f between the target pedestrian and the surrounding pedestrian nodes is obtained. b .
[0022] S203, obtaining the spatial graph attention features f of the target pedestrian to be detected and the surrounding pedestrians based on the spatial attention mechanism c , including the following steps:
[0023] S204. Apply the graph attention neural network to focus on the important information in the graph, and use the attention mechanism to aggregate neighbor nodes, realizing the adaptive allocation of weights to different neighbors. Use the output of the graph attention neural network (GAT) to aggregate node features. This is achieved by weighted summation of node features, with weights provided by the graph attention neural network. The initial input part is the features of the pedestrian node feature vector fusion, which is defined as w c is a weight matrix, the purpose is to transform the target pedestrian node features Transformed to the new feature space, the new feature vector dimension is D′, || represents the concatenation of the new feature representations of the target pedestrian and the neighbor pedestrian after conversion, and we get Its dimension is
[0024] S205: Establish an attention matrix between nodes based on the node feature fusion method. At time t, the attention of pedestrian graph node i to surrounding pedestrian graph node j is Indicates that its value depends on the spatial interaction of pedestrian i at time t Corresponding node features Assign different weights to the neighbor pedestrians of target pedestrian i In the above formula, σ is a nonlinear activation function, and exp represents an exponential function, which is used to complete softmax normalization. T is a weight vector, the purpose is to pass and The matrix multiplication of is used to obtain the attention coefficient of the surrounding neighbor i to the target pedestrian j at time t. It refers to the pedestrians around pedestrian i who are connected to him by edges in the graph and who interact with each other in the space.
[0025] S206. For pedestrian i, the features after being processed by the graph attention mechanism are: Finally, after multiple trainings, the training parameters are transferred backwards, and the graph attention neural network (GAT) is used to output the feature vector value of a single node i as follows:
[0026] S207: Extract the interaction features between the target pedestrian and the surrounding obstacles from the aggregated features. This may include the strength of association, relative position, speed difference, etc. between the target pedestrian and other nodes. gat are all training parameters of the spatial graph attention neural network GAT.
[0027] S3, integrating the spatial graph attention features of the target pedestrian's walking behavior constraints, and performing secondary dynamic programming on the target pedestrian's spatial attention features, including the following steps:
[0028] S301. Here, we focus on the circular area within one meter of the pedestrian to be predicted, which will become the focus area of the spatial graph. According to the pedestrian's walking behavior norms, we focus on the circular area with a radius of 1m of the pedestrian to be predicted. Among the surrounding pedestrians in the focus area, dynamic weighted assignment is used to update the adjacency matrix of the interaction relationship. This shows that the model dynamically adjusts the association strength between pedestrians according to the relative positions of pedestrians and other dynamic factors.
[0029] S302: Take the circular area with a radius of 1m of the target pedestrian as the focus area, and create a pedestrian interaction matrix in the focus area, which includes the interaction relationship between pedestrians. The pedestrian interaction matrix value in this area Dynamically assign values based on the distance to the target pedestrian. The value of is expressed as the distance between pedestrian node i and pedestrian node j at time t, defined as: and Represents the distance between all pedestrian nodes and pedestrian node i in the space. The spatial graph attention interaction feature integrating the walking behavior constraints of the target pedestrian is defined as: Formula (9) assigns values based on the dynamic constraints within the walking distance according to the interaction characteristics between the target pedestrian and the surrounding interactive objects. When the target predicted object is close to the surrounding pedestrians, the distances of all surrounding pedestrians from the target pedestrian are collected to assign weights. The value range is between 0 and 1. Since pedestrians at a closer distance have a greater impact on the trajectory of the target predicted pedestrian, a larger interaction feature is finally obtained by subtracting its value.
[0030] S303. For the values in the pedestrian interaction matrix, the specific values are dynamically assigned by the distance from the target pedestrian. The calculation of this specific value is based on the distance function rule to ensure that pedestrians closer to the target pedestrian have a greater weight in the interaction relationship. Pedestrians outside this range are assumed to have little interaction with the target predicted pedestrian, and the spatial graph attention interaction feature is not updated at this time.
[0031] S304. When the graph attention network model uses spatial graph data for processing, the application of the spatial attention mechanism is crucial, which allows the model to selectively focus on specific areas instead of comprehensively processing the entire spatial graph. The model's focus is on the circular area within one meter around the target pedestrian. In this way, the model can focus on the area most relevant to the target pedestrian, thereby improving processing efficiency and accuracy.
[0032] S305. The graph attention network model will pay special attention to a circular area with a radius of 1 meter around the target pedestrian. The selection of this area is based on the characteristics of pedestrian behavior, because during walking, the surrounding pedestrians may have a direct impact on the behavior of the target pedestrian.
[0033] S306. In this focused area, the graph attention network model will dynamically update the interaction relationship between pedestrians. This dynamic update is achieved through dynamic weighted assignment, taking into account the relative positions between pedestrians and other dynamic factors. This processing method enables the model to more flexibly adapt to pedestrian interactions in different environments, improving the applicability and generalization ability of the model.
[0034] S307. Further, the graph attention network model dynamically adjusts the weight of the interaction relationship according to the distance from the target pedestrian. This adjustment is calculated based on the distance function rule, ensuring that pedestrians closer to the target pedestrian have greater influence in the interaction relationship. In this way, the model can more accurately capture the environment around the target pedestrian and make more accurate predictions.
[0035] S308. In summary, by combining the spatial graph attention features with the walking behavior constraints of the target pedestrians, the graph attention network model can better understand the interaction relationship between pedestrians, thereby improving the accuracy and robustness of pedestrian prediction.
[0036] S4, using the various trajectory features of pedestrians obtained in steps S1, S2 and S3 to fuse their features, and mapping the state of pedestrians in a certain period of time to the trajectory of a future period of time through a sequence-to-sequence model (Seq2Seq), including the following steps:
[0037] S401, based on adaptive BiLSTM (bidirectional long short-term memory network), is used to more accurately extract pedestrian trajectory features.
[0038] S402, BiLSTM processes the forward and backward information of the trajectory at the same time, so that the model can capture more comprehensive temporal dependencies. And the ability of BiLSTM is further enhanced through the adaptive mechanism, so that it can dynamically adjust the network parameters and weights according to the characteristics of different pedestrian trajectories, so as to extract the best features of different trajectory information. This method can more accurately extract the temporal features in pedestrian trajectories, improve the accuracy of trajectory prediction and the generalization performance of the model.
[0039] S403, using a graph attention neural network (GAT) to process different trajectories of surrounding pedestrians, thereby extracting interactive information characteristics between them.
[0040] S404. Based on the walking behavior constraint mechanism, GAT can accurately extract the interaction information between pedestrians by assigning different attention weights to each pedestrian's neighbor nodes. In pedestrian trajectory prediction, the fusion of this constraint will not only take into account the characteristics of the target pedestrian itself, but also combine the interactions between surrounding pedestrians, especially when following certain walking behavior constraints. This fusion mechanism can more comprehensively capture the interaction patterns between pedestrians and more accurately characterize the walking behavior between pedestrians.
[0041] S405. Use the MLP (Multi-layer Perceptron) mechanism to fuse the temporal features and the interactive features. Fusing the temporal features with the interactive features can more comprehensively model pedestrian motion behavior. In this process, the temporal features (extracted by the adaptive BiLSTM) capture the temporal dependencies in pedestrian trajectories, while the interactive features (extracted by the GAT) describe the interaction relationship between pedestrians. The extracted temporal features and interactive features are concatenated to form a comprehensive feature vector.
[0042] S406, input the concatenated comprehensive feature vector into the MLP. The MLP consists of three fully connected layers, each of which is followed by a nonlinear activation function (such as ReLU) to capture the complex nonlinear relationship between features.
[0043] S407, MLP learns the high-order correlation between temporal features and interaction features through multi-layer nonlinear mapping to generate a fused feature representation.
[0044] S408, decoding of trajectory prediction, using LSTM to decode the feature vector after the trajectory motion features and trajectory interaction features are fused through the MLP mechanism, and finally completing the trajectory prediction. This step involves interpreting the encoded information into the final trajectory prediction result.
[0045] The above is a preferred embodiment of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Modifications and changes made by ordinary technicians in this field based on the core idea of the present invention should be within the protection scope of the appended claims of the present invention.
Claims
1. A pedestrian trajectory prediction method based on spatial graph attention mechanism combining adaptive weighted fusion mechanism with pedestrian walking behavior norms, characterized in that: include: S1. A BiLTSM network based on a fusion adaptive weighted fusion mechanism method obtains temporal features of pedestrian motion trajectory f a . S2. Use the relationship between the target pedestrian and the surrounding pedestrians to construct a preliminary interaction feature f b , and obtain the spatial graph attention features f of the target pedestrian to be detected and the surrounding pedestrians through the spatial attention mechanism c . S3, using the spatial graph attention feature f obtained in step S2 c ,Under the constraints of pedestrian walking behavior norms, the spatial graph attention mechanism is used to perform secondary dynamic programming on the spatial attention interaction characteristics of the target pedestrian. S4: The features of the pedestrian trajectories obtained in steps S1, S2, and S3 are fused, and the state of the pedestrian in a certain period of time is mapped to the trajectory in the future period of time through a sequence-to-sequence model (Seq2Seq).
2. According to the method for predicting pedestrian trajectories by using a spatial graph attention mechanism combining an adaptive weighted fusion mechanism with pedestrian walking behavior norms according to claim 1, the S1 comprises the following steps: S101, collecting historical movement trajectories of the target predicted pedestrian and surrounding pedestrians at T moments, and extracting trajectory time series features. S102. The traditional unidirectional LSTM network can only extract features from one direction of the time series, resulting in the model being able to use information from a single direction for feature extraction. However, since pedestrian trajectories have bidirectional time dependencies, ignoring the information in the reverse time series will result in reduced prediction accuracy. The present invention proposes a BiLSTM network with an adaptive fusion mechanism for pedestrian trajectory time series feature extraction, and captures the bidirectional dependency in the time series by simultaneously introducing forward and backward LSTMs. The forward LSTM in the BiLSTM is responsible for extracting features from the past to the present of the time series, and the generated output features are represented as h forward The backward LSTM in BiLSTM is responsible for extracting features from the future to the past, and the generated output features are represented as h backward . S103. At the same time, the present invention proposes to extract pedestrian time series features by introducing trainable weight parameters α and β, which correspond to the output features of the forward LSTM and the backward LSTM respectively. Through these weight parameters, the model can automatically learn the optimal fusion ratio of the forward and backward features during the training process. The final fusion feature h fused Calculated by weighted summation formula: h fused =a·h forward +β·h backward (1) S104. The features of these two directions are further integrated based on the adaptive weighted fusion mechanism, so that the model can simultaneously consider the past and future movement patterns of pedestrians, thereby comprehensively improving the accuracy of pedestrian temporal feature extraction. S105. Finally, the historical motion trajectories of the target predicted pedestrians and the surrounding pedestrians at T moments are collected, and the BiLSTM module with the improved adaptive fusion mechanism is used to extract the time series feature vector of the pre-processed motion history trajectory of each pedestrian in the collected group to obtain the time series feature f of the predicted target group. a .
3. According to the method for predicting pedestrian trajectories by using a spatial graph attention mechanism combining an adaptive weighted fusion mechanism with pedestrian walking behavior norms according to claim 1, the S2 comprises the following steps: S201. Use the relationship between the predicted pedestrian and the surrounding pedestrians to build a graph structure. Each pedestrian is regarded as a node in the graph. The interaction between pedestrians is modeled through edges to obtain the preliminary interaction features f between the target pedestrian and the surrounding pedestrians. b . S202: Through the graph attention mechanism, different importance weights are assigned to different interactions, so as to more finely capture the complex relationship between pedestrians and obtain the spatial graph attention features f of the target pedestrian to be detected and the surrounding pedestrians. c .
4. According to claim 1 or 2, using the spatial graph attention features obtained in step S2, under the constraints of pedestrian walking behavior norms, using the spatial graph attention mechanism to perform secondary dynamic programming on the spatial attention interaction characteristics of the target pedestrian, said S3 includes the following steps: S301. According to the pedestrian's walking behavior norm, among the pedestrians around the area that is the focus of the predicted target pedestrian's walking, dynamically weighted values are assigned to the pedestrians within the area, and the adjacency matrix of the interaction relationship is updated. S302. Traditional pedestrian trajectory prediction methods are usually based on data-driven models, focusing on directly inferring future movement paths from historical trajectories, ignoring the interaction between pedestrians within the walking range. The present invention introduces a dynamic weight factor allocation mechanism for pedestrians around the target predicted walking range. By considering the constraints of the interaction factors, the prediction model can more accurately reflect the decision-making process of pedestrians in reality, thereby improving the prediction accuracy, especially in complex scenarios such as intersections or crowded sidewalks. For example, when pedestrians are walking, the area of focus is a circular area with a radius of 1m for the target pedestrian. The pedestrian interaction matrix value in this area is The target pedestrian i will be dynamically assigned a value according to the distance from the target pedestrian j, and its value is: S303, the updated interaction matrix value is based on the node characteristics Assign different weights to the neighbor pedestrians of target pedestrian i S304, dynamically assigning different attention weight relationships to surrounding pedestrians based on the walking behavior range and extracting and learning multiple interactive features between pedestrians, such as distance, relative speed, movement direction, etc. By assigning different weights to the fused multiple interactive features within the walking area of the target predicted pedestrian, the interactive influence between pedestrians can be more effectively reflected. S305. In different distance ranges, pedestrian interaction characteristics show different behavior patterns. The improved pedestrian trajectory prediction method can flexibly adjust the weights of these characteristics through its attention mechanism combined with the walking range distance, so as to adapt to complex dynamic behaviors in different scenarios, such as avoiding behavior when walking in a crowd or free movement in an open environment. S306. Finally, based on the improved dynamically adjusted interaction features combined with the GAT mechanism, the influence relationship between pedestrians is more accurately captured, thereby generating more accurate feature extraction and weight allocation.
5. According to claim 1 or 2, a method for fusing the features of the pedestrian trajectories obtained in steps S1, S2 and S3, and mapping the state of the pedestrian in a certain period of time to the trajectory of a future period of time through a sequence-to-sequence model (Seq2Seq), wherein S4 includes the following steps: The time series feature f a and the spatial attention feature f c After splicing, the features are sent to the decoder to obtain LSTM decoding features, and the decoding features are fully connected. Finally, they are decoded using the long short-term memory network LSTM to complete trajectory prediction.
6. A pedestrian trajectory prediction method system combining an adaptive weighted fusion mechanism with a spatial graph attention mechanism and pedestrian walking behavior norms, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.
7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a pedestrian trajectory prediction method using a spatial graph attention mechanism combining an adaptive weighted fusion mechanism with pedestrian walking behavior specifications as described in any one of claims 1 to 4 are implemented.
Citation Information
Patent Citations
Graph neural network pedestrian trajectory prediction method based on attention mechanism
CN113888638A
Pedestrian trajectory prediction method and system based on complementary attention
CN114429490A
Pedestrian trajectory prediction method and system based on multi-interaction spatiotemporal graph network
US11495055B1
Cited By
Smart home control method and system based on multi-modal fusion and deep learning
CN120630745A
