Spatial and temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transform

Abstract

The invention relates to the technical field of motion prediction, in particular to a spatial-temporal feature fusion pedestrian trajectory prediction method based on a graph convolutional network and Transfor mer, and the method comprises the steps: obtaining a to-be-processed pedestrian trajectory coordinate data set; processing the to-be-processed pedestrian trajectory coordinate data set by adopting a pre-trained spatial-temporal feature fusion model based on a graph convolutional network and Transform to obtain a pedestrian trajectory prediction result; according to the method, pedestrian space interaction is optimized by dynamically combining group interaction and local interaction, combining a graph convolutional network, direction perception enhancement and a space Transform, and combining a time domain convolutional network and a time Transform architecture, and short-term motion changes and long-term tracks are captured at the same time, so that the accuracy of track prediction is improved; and in combination with the spatio-temporal feature integration capability of Transform, the internal relevance between spatio-temporal data is deeply mined, and the universality of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of motion prediction, and particularly to a spatio-temporal feature fusion pedestrian trajectory prediction method based on a graph convolutional network and a Transformer. Background Art

[0002] As one of the core technologies in fields such as autonomous driving, robot navigation, and smart cities, pedestrian trajectory prediction aims to predict the future motion trajectory of a target by analyzing its historical motion state and the dynamic changes in its surrounding environment. A high-precision pedestrian trajectory prediction algorithm can identify potential collision risks early, effectively reduce the incidence of traffic accidents, and provide important technical support for the construction of future intelligent transportation systems, having profound social significance and application value.

[0003] In the prior art, deep learning-based pedestrian trajectory prediction methods use various deep learning models to learn pedestrian features from a large amount of trajectory data, and use the trained model to predict the future walking path according to historical trajectory information. However, the following problems generally exist:

[0004] The model overly relies on the design of kernel functions based on spatial distance, and does not sufficiently mine the local spatial interaction features of pedestrians; it ignores the actual laws of pedestrian movement and does not consider the group features corresponding to the group movement pattern; there are limitations in time feature extraction, and it is difficult to accurately capture the historical movement laws of pedestrians; when fusing time and space features, a simple direct splicing method is mostly used, lacking an effective adaptive fusion mechanism, and failing to deeply mine the internal correlation between spatio-temporal data. Summary of the Invention

[0005] In view of this, the present invention discloses a spatio-temporal feature fusion pedestrian trajectory prediction method based on a graph convolutional network and a Transformer to solve the above problems; it includes: obtaining a pedestrian trajectory coordinate data set to be processed, and using a pre-trained spatio-temporal feature fusion model based on a graph convolutional network and a Transformer to process the pedestrian trajectory coordinate data set to be processed to obtain a pedestrian trajectory prediction result;

[0006] Furthermore, the spatio-temporal feature fusion model based on a graph convolutional network and a Transformer includes: a pedestrian trajectory embedding module for constructing a global interaction graph and a historical trajectory according to the pedestrian trajectory coordinate data set to be processed; a pedestrian spatial feature extraction module for extracting group features and local features from the global interaction graph based on graph convolution and an attention mechanism; a hierarchical time feature extraction module for performing feature extraction on historical trajectories at different time scales and obtaining time features through aggregation processing; a spatio-temporal feature fusion module for adaptively fusing group features, local features, and time features to generate a pedestrian trajectory prediction result.

[0007] The beneficial effects of the present invention include:

[0008] By constructing parallel global spatial interaction feature modules and local spatial interaction feature modules, respectively extracting global and local spatial interaction information, and combining with a group recognition module, a graph convolutional network, a direction-aware region enhancement module, and a spatial Transformer mechanism, the problems of missing group spatial interaction modeling and insufficient extraction of local spatial interaction features are effectively solved, and pedestrian spatial interaction features are fully mined;

[0009] By constructing a hierarchical time feature extraction module to model pedestrian trajectories at different time scales, combining a temporal convolutional network, a Transformer architecture, and a long-short-term time feature gating mechanism, the model can simultaneously capture trajectory patterns of short-term and long-term motion changes, accurately capture the historical motion laws of pedestrians, and thus improve the accuracy of trajectory prediction;

[0010] By constructing a spatio-temporal feature fusion module, combining the spatio-temporal feature integration ability of the Transformer, deeply mining the internal correlation between spatio-temporal data, realizing the adaptive and efficient fusion of spatio-temporal features, and improving the adaptability of the model in different scenarios. Brief Description of the Drawings

[0011] Figure 1 It is a schematic structural diagram of the spatio-temporal feature fusion model based on a graph convolutional network and a Transformer in the present invention;

[0012] Figure 2 It is a schematic structural diagram of the local spatial Transformer module in the present invention;

[0013] Figure 3 It is a schematic structural diagram of the hierarchical time feature extraction module in the present invention;

[0014] Figure 4 It is a schematic structural diagram of the spatio-temporal feature fusion module in the present invention. Detailed Embodiments

[0015] In order to make the purpose, technical solutions, features, and advantages of the present invention clearer and more understandable, the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0016] This embodiment includes a spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer, which includes: obtaining a pedestrian trajectory coordinate data set to be processed, and using a pre-trained spatio-temporal feature fusion (Spatio-Temporal Feature Fusion Using Graph Convolutional Networks and Transformer, STFGT) model to process the pedestrian trajectory coordinate data set to be processed to obtain a pedestrian trajectory prediction result.

[0017] Specifically, as Figure 1 shown, the STFGT model includes:

[0018] A pedestrian trajectory embedding module, which is used to construct a global interaction graph and a historical trajectory according to the pedestrian trajectory coordinate data set to be processed.

[0019] A pedestrian spatial feature extraction module, which is used to extract group features and local features from the global interaction graph based on graph convolution and attention mechanism.

[0020] A hierarchical time feature extraction module, which is used to extract features from historical trajectories at different time scales and obtain time features through aggregation processing.

[0021] A spatio-temporal feature fusion module, which is used to adaptively fuse group features, local features and time features to generate a pedestrian trajectory prediction result.

[0022] Furthermore, the pedestrian trajectory embedding module retains all the original interaction information between pedestrians within the time step and constructs a global interaction graph G t =(V t , E t ) and a historical trajectory. Among them, the global interaction graph G t uses the vertex set V t to represent all pedestrians, and the edge set E t to represent the interaction relationship between pedestrians.

[0023] Furthermore, the pedestrian spatial feature extraction module includes a parallel global spatial interaction feature module and a local spatial interaction feature module.

[0024] Specifically, the global spatial interaction feature module includes a dynamic group recognition module and a graph convolutional network; the processing of the dynamic group recognition module for data includes:

[0025] Step 1: Divide the pedestrian groups and calculate the group centers

[0026] Specifically, the present invention uses a dynamic group recognition method to extract global spatial interaction features, determines groups from three aspects: social distance constraint, direction consistency, and acceleration difference, divides the global interaction graph into groups, and determines that pedestrians i and j belong to the same group The conditional formula is:

[0027]

[0028] Among them, represents the distance between pedestrian i and pedestrian j at time t, and γ d represents the preset social distance constraint threshold, represents the difference in the direction angles of pedestrian i and pedestrian j at time t, and γ θ represents the preset direction consistency threshold, represents the acceleration difference between pedestrian i and pedestrian j at time t, and γ a represents the preset acceleration difference threshold.

[0029] Group center is expressed as:

[0030]

[0031] Among them, represents the position mean term, represents the speed mean term. The mean position is the position that minimizes the sum of the squares of the Euclidean distances from all members within the group to this point; in a real social group, such as relatives and friends walking together, they will spontaneously maintain speed synchronization, and the mean speed can reflect the overall moving direction. Therefore, the group center combining the position and speed means can more reasonably reflect the geometric center and overall movement trend of group members. For a group containing N k pedestrians The position mean term and the speed mean term The calculation formulas are:

[0032]

[0033] Step 2, taking as nodes, construct a group interaction graph Among them and ε t respectively represent the nodes and edges of

[0034] Step 3, use a graph convolutional network (GCN) to extract global spatial interaction features of to obtain group features.

[0035] Specifically, the extraction of global spatial interaction features includes updating the GCN, and the update formula of the GCN is as follows:

[0036]

[0037] Among them, represents the input feature matrix of the l-th layer, represents the weight matrix at the current moment, is 's adjacency matrix, and the value is the reciprocal of the central distance , indicating that the closer the distance, the closer the relationship. represents the node degree matrix after including self-loop edges in the graph convolutional network, and σ represents the Sigmod activation function. After updating, the group feature F g is obtained.

[0038] The local spatial interaction feature module includes: a graph pruning module, a direction-aware region enhancement module, and a local spatial Transformer module. Specifically:

[0039] The graph pruning module optimizes the global interaction graph based on the relative spatial relationship, motion direction, and distance threshold to obtain a local graph.

[0040] The direction-aware region enhancement module is used to process the local graph, encodes the pedestrian motion direction information into learnable features, and concatenates the learnable features with the pedestrian's speed and position information to obtain the interaction weight between pedestrians.

[0041] Specifically, for each pedestrian i, calculate its motion direction angle. Embed the direction angle into a high-dimensional space through the sine function and cosine function to obtain the direction encoding; concatenate the speed and position information, that is, the original motion features, with the direction encoding to obtain the interaction weight between pedestrians. The direction-aware region enhancement module enhances the model's perception ability of direction changes by encoding the pedestrian motion direction information into learnable features.

[0042] The local spatial Transformer module, as Figure 2 shown, is used to further extract local features according to the interaction weight between pedestrians, transform the self-attention mechanism into a graph attention mechanism, and obtain the local feature F l through processing by a feed-forward neural network.

[0043] Furthermore, the hierarchical time feature extraction module: adopts a two-layer time modeling architecture to extract information at different time scales. As Figure 3 shown, the time feature extraction module includes a short-term layer branch and a long-term layer branch.

[0044] In the short-term layer branch, a temporal convolutional network is used to model short-term dynamic features and obtain local temporal information. Dilated convolution can efficiently calculate and extract local temporal dynamic information, providing the model with refined motion features. The calculation formula is:

[0045] F short =TCN(X 1:T )

[0046] Among them, F short Represents local time information, X 1:T Represents historical trajectory.

[0047] In the long-term layer branch, the temporal Transformer is used to extract the global trajectory pattern, and the long-term series modeling is performed based on the global attention mechanism to obtain long-scale time information. The calculation formula is:

[0048]

[0049] Among them, F long represents long-scale time information, Q represents the query matrix, K represents the key matrix, V represents the value matrix, d represents the dimension of K, and Q, K, and V are mapped from historical trajectories.

[0050] Based on the long-term and short-term time feature gating mechanism, the time information obtained by the short-term layer branch and the long-term layer branch is aggregated and extracted to obtain the time aggregation feature g t , the formula is:

[0051] g t =σ(W g ·[F short ; F long ]+b g )

[0052] Among them, W g represents the weight parameter matrix, b g represents the bias term.

[0053] Use the fully connected layer to t Processing is performed to obtain the time feature F tem , the formula is:

[0054] F tem =g t ⊙F short +(1-g t )⊙F long

[0055] Further, if Figure 4 As shown, the spatiotemporal feature fusion module adopts a multi-layer encoder architecture. In this embodiment, the number of encoders used is 3, and each encoder layer contains 4 attention heads. The spatiotemporal feature fusion module processes data including:

[0056] Step 1: Integrate the group features, local features, and temporal features to obtain integrated features. The integration formula is as follows:

[0057] F fusion = MLP(F g , W g ) + F l + MLP(F tem , W tem )

[0058] where F fusion represents the integrated features, MLP() represents the multi-layer perceptron, W g , W tem represent the weight parameter matrices of the spatial features and temporal features in the multi-layer perceptron respectively, F g represents the group features, F l represents the local features, and F tem represents the temporal features.

[0059] Step 2: Encode the integrated features to obtain the integrated sequence F input . Use the multi-head attention mechanism MultiHeadAttention() to calculate the dependencies between the features in the integrated sequence F input to obtain the attention scores F attn . The formula is as follows:

[0060] F attn = MultiHeadAttention(F input )

[0061] Step 3: Perform residual connection and layer normalization on F attn to ensure smooth information flow and avoid problems such as gradient vanishing.

[0062] Step 4: Use the feed-forward neural network FeedForward() to perform feature extraction on F input and the layer-normalized F attn to obtain high-order features F ffn . The formula is as follows:

[0063] F ffn = FeedForward(LayerNorm(F input + F attn ))

[0064] Specifically, the dimension of the feed-forward network used in this embodiment is 1024.

[0065] Step 5: Perform on the high-order features F ffnApply residual connections and layer normalization, stack the encoders, and output the feature F final , that is, the pedestrian trajectory prediction result.

[0066] Finally, it should be noted that the above only describes some embodiments of the present invention. For those skilled in the art, various changes, modifications, substitutions, and deformations can be envisioned without departing from the principles and spirit of the present invention. The protection scope of the present invention is defined by the appended claims and their equivalents, and the above actions should all be covered within the protection scope of the present invention.

Claims

1. A spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer, characterized in that Including: Obtain the pedestrian trajectory coordinate data set to be processed, and use a pre-trained spatio-temporal feature fusion model based on graph convolutional network and Transformer to process the pedestrian trajectory coordinate data set to be processed, and obtain the pedestrian trajectory prediction result; The spatio-temporal feature fusion model based on graph convolutional network and Transformer includes: A pedestrian trajectory embedding module for constructing a global interaction graph and a historical trajectory according to the pedestrian trajectory coordinate data set to be processed; A pedestrian spatial feature extraction module for extracting group features and local features from the global interaction graph based on graph convolution and attention mechanism; A hierarchical time feature extraction module for extracting features from historical trajectories at different time scales and obtaining time features through aggregation processing; A spatio-temporal feature fusion module for adaptively fusing group features, local features and time features to generate pedestrian trajectory prediction results.

2. The spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer according to claim 1, characterized in that The pedestrian spatial feature extraction module includes a parallel global spatial interaction feature module and a local spatial interaction feature module.

3. The spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer according to claim 2, characterized in that The global spatial interaction feature module uses a dynamic group recognition method to divide pedestrian groups and extract global spatial interaction features from the global interaction graph; The processing of data by the global spatial interaction feature module includes: Step 1, divide the line population and calculate the population center Step 2: Using as a node, construct a group interaction graph Step 3: Use a graph convolutional network to extract the global spatial interaction features to obtain the group features.

4. The spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer according to claim 3, characterized in that The judgment basis for dividing pedestrian groups includes: social distance constraint, direction consistency and acceleration difference, and the formula is: Among them, represents the distance between pedestrian i and pedestrian j at time t, and γ d represents the preset social distance constraint threshold, represents the difference in the direction angles between pedestrian i and pedestrian j at time t, and γ θ represents the preset direction consistency threshold, represents the difference in acceleration between pedestrian i and pedestrian j at time t, and γ a represents the preset acceleration difference threshold.

5. The spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer according to claim 2, characterized in that The local spatial interaction feature module includes: A graph pruning module that optimizes the global interaction graph based on relative spatial relationships, motion directions and distance thresholds to obtain a local graph; A direction-aware region enhancement module for processing the local graph, encoding the pedestrian motion direction information into learnable features, and splicing the learnable features with the pedestrian speed and position information to obtain the interaction weight between pedestrians; A local spatial Transformer module for extracting local features based on the interaction weight between pedestrians.

6. The spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer according to claim 1, characterized in that The hierarchical time feature extraction module adopts a two-layer time modeling architecture, and the two-layer time modeling architecture includes a short-term layer branch and a long-term layer branch, where: In the short-term layer branch, a time convolutional network is used for short-term dynamic feature modeling; In the long-term layer branch, a time Transformer is used to extract the global trajectory pattern and perform long-time series modeling based on the global attention mechanism.

7. The spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer according to claim 6, characterized in that, The hierarchical time feature extraction module also includes: adopting a long short-term time feature gating mechanism to aggregate and extract the time information obtained from the short-term layer branch and the long-term layer branch to obtain a time aggregation feature; using a fully connected layer to process the time aggregation feature to obtain a time feature.

8. The spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer according to claim 1, characterized in that The spatio-temporal feature fusion module adopts a multi-layer encoder architecture, and the processing of data by the spatio-temporal feature fusion module includes: Step 1: Fuse the group features, local features and time features to obtain a fusion feature; Step 2: Encode the fused features to obtain a fused sequence F input , and use the multi-head attention mechanism to calculate the dependencies between features in the fused sequence to obtain attention scores F attn ; Step 3. Perform residual connection and layer normalization on F attn ; Step 4: Use a feedforward neural network for F input and the F after layer normalization processing attn to perform feature extraction to obtain high-order feature F ffn ; Step 5, for F ffn Apply residual connection and layer normalization, stack the encoders, and output feature F final .

9. The spatio-temporal feature fusion pedestrian trajectory prediction method based on graph convolutional network and Transformer according to claim 1, wherein, The formula for fusing group features, local features and time features is: F fusion = MLP(F g , W g ) + F l + MLP(F tem , W tem ) Among them, F fusion represents the fusion feature, MLP() represents the multi-layer perceptron, and W g , W tem respectively represent the weight parameter matrices of the spatial feature and the temporal feature of the multi-layer perceptron MLP(), F g represents the group feature, F l represents the local feature, and F tem represents the temporal feature.

Images