A pedestrian trajectory prediction method and system based on Bi-LSTM and multi-head self-attention

Abstract

This invention discloses a pedestrian trajectory prediction method and system based on Bi-LSTM and multi-head self-attention. The method includes: acquiring the historical trajectory coordinates of a pedestrian, inputting them into a bidirectional long short-term memory network for bidirectional temporal encoding to obtain forward hidden states and backward hidden states; inputting the forward and backward hidden states into a bidirectional gated attention fusion module, and generating bidirectional temporal features through gating weights and multi-head self-attention optimization; concatenating the direction probability map output by the direction prediction module with the pedestrian's current coordinate information and the bidirectional temporal features, and inputting it into a multi-head self-attention encoder for cross-modal deep fusion to obtain comprehensive features; directly inputting the comprehensive features into a fully connected layer to output the trajectory coordinates at the next time step, and generating prediction results of a specified time length through iterative prediction. This invention significantly improves the accuracy and generalization ability of pedestrian trajectory prediction through bidirectional feature fusion, multimodal feature complementarity, and staged collaborative training.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving, intelligent transportation, and neural network model technology, specifically to a pedestrian trajectory prediction method and system based on Bi-LSTM and multi-head self-attention. Background Technology

[0002] Pedestrian trajectory prediction is a key technology in intelligent transportation and computer vision, widely used in scenarios such as dynamic decision-making for autonomous vehicles, obstacle avoidance and navigation for robots, and abnormal behavior warnings in intelligent video surveillance. Its core task is to predict the movement trajectory of a pedestrian over a future period based on the pedestrian's two-dimensional spatial position sequence within a given historical observation time step. The accuracy of pedestrian trajectory prediction directly affects the safety and reliability of related intelligent systems in environmental perception, behavioral decision-making, and operational control.

[0003] Currently, deep learning-based sequence modeling technology has become the mainstream technical solution for pedestrian trajectory prediction. Various model architectures have been attempted for feature mining of trajectory sequences: Long Short-Term Memory Networks (LSTM) and Gated Recurrent Units (GRUs) remain the most basic and widely used pedestrian trajectory prediction models due to their ability to handle long sequence dependencies; Transformer-type models, with their self-attention mechanism's advantage in modeling long-distance temporal correlations, have gradually become a research hotspot for high-precision trajectory prediction; Graph Neural Networks (GNNs) are widely used to integrate social interaction information between pedestrians and vehicles, giving rise to typical models such as Social-GNN and ST-GAT; In addition, generative models such as GANs and VAEs have also been introduced to solve the multimodal problem of pedestrian trajectory prediction. To improve prediction accuracy, existing optimization paths can be divided into two main categories: one is to integrate complex external modules such as social modeling, scene semantic segmentation, and pedestrian motion target regions on basic recurrent models such as LSTM / GRU to make up for the lack of feature mining in a single model; the other is to directly transfer mature architectures in natural language processing and computer vision (such as Bi-LSTM and Transformer encoder-decoder) to pedestrian trajectory prediction scenarios, hoping to extract more temporal features through more powerful general sequence modeling capabilities.

[0004] However, existing technologies face significant technical bottlenecks in practical applications. First, traditional unidirectional recurrent models (LSTM / GRU) have inherent limitations in feature mining, only able to extract unidirectional trajectory features from the past to the future, failing to capture the reverse motion dependencies in the trajectory sequence (e.g., the constraint of the pedestrian's final movement trend on its intermediate trajectory, and the reverse temporal correlation of trajectory inflection points), resulting in insufficient feature mining and limited prediction accuracy. Simultaneously, unidirectional LSTMs are weak at capturing key local features of pedestrian trajectories (e.g., turning points, sudden speed changes, and avoidance actions in crowded scenes), easily smoothing out motion details, leading to significantly increased prediction bias in complex motion scenarios, failing to meet the high accuracy requirements of practical applications. Second, some existing technologies directly transfer general-domain Bi-LSTM to pedestrian trajectory prediction scenarios, failing to adapt to the spatial and temporal continuity characteristics of pedestrian trajectory prediction tasks, easily leading to imbalances in feature fusion and feature redundancy, not only failing to significantly improve accuracy but also easily causing imbalanced predicted trajectories. Furthermore, existing pedestrian trajectory prediction models rely excessively on the fusion of complex auxiliary modules such as social modeling and scene analysis to improve prediction accuracy. This leads to redundant model structures, and multimodal feature fusion is prone to information conflicts, resulting in limited accuracy improvement. Specifically, while Transformer-type models can model global temporal relationships through self-attention, their high computational complexity and susceptibility to introducing invalid long-distance association noise make them difficult to deploy at edge environments. GNN-type models, on the other hand, rely excessively on the effectiveness of pedestrian social interaction information, resulting in a significant decrease in feature mining capabilities in sparse scenarios, and information conflicts easily arise when fusing social features with other features. Even bidirectional sequence models such as Bi-LSTM and Bi-GRU simply copy general domain architectures without optimization for the specific characteristics of pedestrian trajectory prediction tasks. This leads to problems such as imbalance in forward and backward feature fusion and redundant feature interference, which not only fails to improve accuracy but also increases computational complexity.

[0005] In summary, existing technologies, whether deep learning models such as recurrent neural networks, Transformers, and GNNs, or traditional dynamic physics models, all have technical bottlenecks. Therefore, developing an improved sequence modeling architecture for pedestrian trajectory prediction tasks to overcome the accuracy bottlenecks of existing models while also considering their lightweight nature and deployability has become an urgent technical problem to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to provide a pedestrian trajectory prediction method and system based on Bi-LSTM and multi-head self-attention, aiming to solve the technical problems of insufficient feature mining, imbalance of bidirectional feature fusion, and limited prediction accuracy in existing pedestrian trajectory prediction technologies.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A pedestrian trajectory prediction method based on Bi-LSTM and multi-head self-attention includes the following steps:

[0009] Step S1: Obtain the historical trajectory coordinates of the pedestrian;

[0010] Step S2: Input the historical trajectory coordinates into the bidirectional long short-term memory network Bi-LSTM to obtain the forward hidden state and the backward hidden state;

[0011] Step S3: Input the forward hidden state and the backward hidden state into the bidirectional gated attention fusion module, and generate the fused bidirectional temporal features through the gated attention mechanism;

[0012] Step S4: Input the pedestrian's historical trajectory coordinates and the semantic segmentation map of the current scene into the direction prediction module to generate a direction probability map; obtain the pedestrian's current coordinate information and bidirectional temporal features; after concatenating the direction probability map, current coordinate information and bidirectional temporal features, input them into the multi-head self-attention encoder for cross-modal deep fusion to obtain comprehensive features;

[0013] Step S5: Input the integrated features directly into the fully connected layer and output the predicted pedestrian trajectory coordinates for the next time step.

[0014] Furthermore, step S3 specifically includes:

[0015] Step S31: Perform linear transformations on the forward hidden state and the backward hidden state respectively, and generate forward gating weights and backward initial gating weights through the Sigmoid activation function;

[0016] Step S32: Input the forward hidden state and the backward hidden state into the multi-head self-attention layer respectively to obtain the optimized forward hidden state and backward hidden state;

[0017] Step S33: The forward hidden state and the optimized forward hidden state are weighted and summed using forward gating weights. The reverse hidden state and the optimized reverse hidden state are weighted and summed using reverse gating weights. The weighted sums are then weighted and fused to obtain bidirectional temporal features.

[0018] Furthermore, the calculation formula for the dynamic weighted fusion is as follows:

[0019]

[0020] in, and These are the forward hidden state and the reverse hidden state, respectively. and These are the forward gating weights and the backward gating weights, respectively. and These represent the optimized forward hidden state and backward hidden state, respectively, with a and b being the preset bidirectional fusion weights. This indicates element-wise multiplication.

[0021] Furthermore, the preset bidirectional fusion weight 'a' is set to 0.8 and 'b' is set to 0.2.

[0022] Furthermore, the input to the direction prediction module is the pedestrian's historical trajectory points and the semantic segmentation map of the current scene, and the output is the direction probability map of the pedestrian's movement at the next moment.

[0023] Furthermore, in step S4, the direction probability map, current coordinate information, and bidirectional temporal features are concatenated, specifically including:

[0024] The orientation probability map and current coordinate information are mapped to 64 dimensions through a fully connected layer.

[0025] By concatenating the directional features, spatial coordinate features, and bidirectional temporal features of dimension 128, a 256-dimensional multimodal feature is obtained.

[0026] Furthermore, in step S4, the multi-head self-attention encoder learns the attention weights of different modal features through a self-attention mechanism, thereby achieving cross-modal deep fusion of the orientation probability map, current coordinate information, and bidirectional temporal features.

[0027] Furthermore, by repeating steps S2 to S5 in a cyclical prediction manner, a pedestrian prediction trajectory for a specified time length is generated; the specified time length is 12 time steps.

[0028] Furthermore, the pedestrian trajectory prediction model employs a phased collaborative training strategy, including:

[0029] Phase 1: Independently train the direction prediction module until the model converges;

[0030] The second stage: Freeze the parameters of the direction prediction module after training, and train the trajectory prediction network using the direction probability map output by the direction prediction module;

[0031] The third stage involves unfreezing the parameters of the direction prediction module and fine-tuning the trajectory prediction network and the direction prediction module together, with the overall model's trajectory prediction loss as the optimization objective.

[0032] A pedestrian trajectory prediction system based on Bi-LSTM and multi-head self-attention includes:

[0033] The bidirectional temporal feature encoding module inputs the historical trajectory coordinates of the pedestrian into the Bi-LSTM to obtain the forward hidden state and the backward hidden state.

[0034] The bidirectional feature fusion module inputs the forward hidden state and the backward hidden state into the bidirectional gated attention fusion module to generate bidirectional temporal features.

[0035] The direction prediction module generates a direction probability map based on the pedestrian's historical trajectory coordinates and the semantic segmentation map of the current scene;

[0036] The multimodal feature fusion module takes the orientation probability map, the pedestrian's current coordinate information, and bidirectional temporal features as input. After concatenating the orientation probability map, current coordinate information, and bidirectional temporal features, it inputs them into a multi-head self-attention encoder for cross-modal deep fusion to obtain comprehensive features.

[0037] The trajectory coordinate prediction module directly inputs the comprehensive features into the fully connected layer and outputs the predicted pedestrian trajectory coordinates for the next moment.

[0038] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0039] (I) More comprehensive bidirectional temporal feature mining and significantly improved prediction accuracy: This invention uses Bi-LSTM to bidirectionally encode pedestrian historical trajectories. The forward hidden state captures the pedestrian's movement trend from the past to the present, while the backward hidden state captures the pedestrian's movement intention from the future to the present, achieving comprehensive capture of pedestrian movement temporal features. Based on this, this invention innovatively designs a bidirectional gated attention fusion module. Through the combination of gating weight generation and multi-head attention optimization, it effectively focuses on key movement nodes in the pedestrian trajectory (such as turning points, deceleration points, and avoidance actions in crowded scenes), suppressing the feature influence of noisy time steps. This solves the problem of traditional unidirectional LSTM's weak ability to capture local key features and its tendency to smooth out movement details, significantly improving trajectory prediction accuracy.

[0040] (II) Deep Fusion of Direction Semantics and Motion Features to Achieve Three-Dimensional Feature Complementarity: This invention designs a multi-head self-attention encoder that performs cross-modal deep fusion of the direction probability map output by the direction prediction module, the bidirectional temporal features output by the bidirectional feature fusion module, and the pedestrian's current coordinate information, achieving three-dimensional feature complementarity of direction features, motion features, and current position features. The direction probability map provides semantic constraints for trajectory prediction, the bidirectional temporal features provide motion trend and intent information, and the current coordinate information provides spatial location reference. The synergistic effect of these three types of features provides comprehensive and accurate feature support for trajectory prediction. Compared with existing technologies that overly rely on complex auxiliary modules such as social modeling and scene semantics, this invention avoids the information conflict problem in multi-module feature fusion and achieves higher prediction accuracy with a more streamlined model structure.

[0041] (III) Multi-module collaborative optimization enhances model performance and generalization ability: This invention decouples the direction prediction module from the trajectory prediction network and adopts a phased collaborative training strategy: first, the direction prediction module is trained independently to ensure direction prediction accuracy; then, the parameters of the direction prediction module are frozen to train the trajectory prediction network; finally, joint fine-tuning achieves collaborative optimization of each module. This training strategy avoids the mutual propagation of errors between modules, ensuring that each module can achieve optimal performance and solving the feature adaptation problem between modules. Experimental verification shows that this model achieves optimal results on multiple public datasets, exhibits good robustness in complex scenarios, and its generalization ability is significantly better than existing mainstream models.

[0042] (iv) Outstanding prediction capability in key scenarios, meeting practical application needs: Through the multi-head attention optimization mechanism in the bidirectional gating attention fusion module, this invention can effectively focus on key motion nodes in pedestrian trajectories, and has a stronger ability to capture complex motion scenarios such as turning, changing direction, sudden speed changes, and avoidance actions in crowded scenarios. It maintains excellent prediction performance in both simple and complex scenarios, and can meet the high-precision requirements of autonomous vehicles, social sensing robots and intelligent transportation systems for pedestrian trajectory prediction.

[0043] (V) Wide applicability, scalable to trajectory prediction for various road users: This invention is not only applicable to pedestrian trajectory prediction, but also, because the model architecture relies only on trajectory coordinate sequences and semantic segmentation maps of the scene as input, without depending on pedestrian-specific prior knowledge, it can directly adapt to the trajectory prediction needs of various road users such as bicycles, electric two-wheelers, and cars. The technical solution of this invention can be widely applied in fields such as autonomous vehicles, social sensing robots, and intelligent transportation systems, and has good industrial application prospects and promotional value. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the architecture of the bidirectional feature fusion module of the present invention;

[0045] Figure 2 This is a schematic diagram of the trajectory prediction model structure based on multimodal feature fusion of the present invention;

[0046] Figure 3 This is a flowchart of the phased collaborative training of the present invention;

[0047] Figure 4 This is a schematic diagram of the loss weight matrix of the prediction result when the true label is directly in front, according to an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in further detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0049] The core technical solution of this invention is to provide a pedestrian trajectory prediction model that integrates bidirectional feature fusion, multi-module collaboration, and multi-modal feature fusion. By independently optimizing and collaboratively linking each module, it overcomes the shortcomings of existing technologies and improves the accuracy of pedestrian trajectory prediction. The technical solution of this invention will be described in detail below with reference to specific embodiments.

[0050] Example 1

[0051] This embodiment provides a pedestrian trajectory prediction method based on Bi-LSTM and multi-head self-attention, which specifically includes the following steps.

[0052] 1. Bidirectional Feature Fusion Module

[0053] This invention designs a bidirectional feature fusion module to achieve comprehensive capture and accurate fusion of pedestrian movement temporal features.

[0054] (1) Bidirectional temporal feature coding

[0055] Bi-LSTM is used to encode the historical trajectory coordinates of pedestrians, resulting in forward and backward hidden states. The forward hidden state encodes the pedestrian's movement trend from the past to the present; for example, if the pedestrian slowed down previously, he / she may be about to turn. The backward hidden state encodes the pedestrian's future movement intention; for example, if the pedestrian wants to cross the road in the future, he / she may be stopping at the intersection to observe the road conditions now. Through bidirectional encoding, comprehensive capture of the temporal features of pedestrian movement is achieved.

[0056] like Figure 1 As shown, this module takes the bidirectional hidden state of Bi-LSTM as input and outputs fused features through a three-step process of gating generation, attention optimization, and weighted fusion.

[0057] (2) Generation of gating weights

[0058] For the forward hidden state of the Bi-LSTM output and reverse hidden state Where T is the time step and d is the hidden dimension, the initial gating weights are generated through a fully connected layer and a sigmoid activation function, respectively:

[0059]

[0060]

[0061] in , For the weights of the fully connected layer, , For bias, The Sigmoid activation function has an output range of (0,1), representing the importance of features at each time step.

[0062] (3) Multi-head attention optimization

[0063] Introducing a multi-head self-attention mechanism to adjust the initial gating weights , The input multi-head self-attention layer captures the dependencies between time steps through the self-attention mechanism, and performs secondary optimization on the initial gating weights, focusing on key motion nodes (such as turning points and deceleration points) in the pedestrian trajectory to suppress the feature influence of noisy time steps.

[0064]

[0065]

[0066] Gating weights , It can focus on key nodes of pedestrian trajectories (such as intersection turning points) and suppress the weight of noise time steps.

[0067] (4) Dynamic weighted fusion

[0068] Finally, by dynamically weighted summing, the optimized forward and backward hidden states are fused to obtain bidirectional fused temporal features that combine motion trends and motion intentions, thus solving the problem of key information loss caused by traditional simple fusion methods.

[0069]

[0070] in, and These are the forward hidden state and the reverse hidden state, respectively. and These are the forward gating weights and the backward gating weights, respectively. and These are the optimized forward and reverse hidden states, respectively. This indicates element-wise multiplication, where a and b are preset bidirectional fusion weights. In this embodiment, a = 0.8 and b = 0.2.

[0071] 2. Trajectory prediction model based on multimodal feature fusion

[0072] This invention designs a multi-head self-attention encoder that performs cross-modal deep fusion of the direction probability map output by the direction prediction module, the bidirectional fused temporal features output by the bidirectional feature fusion module, and the current coordinate information of the pedestrian. This achieves three-dimensional feature complementarity of the pedestrian's direction features, motion features, and current position features, providing comprehensive and accurate feature support for trajectory prediction.

[0073] like Figure 2 As shown, this invention proposes a pedestrian trajectory prediction model that integrates directional semantics and bidirectional temporal features. The model includes two independently trainable networks: a pedestrian direction prediction network and a pedestrian trajectory prediction network. The direction prediction module and trajectory prediction network are decoupled to avoid the propagation of errors between modules and to achieve optimal performance for each module.

[0074] (1) Direction prediction module

[0075] The direction prediction network is used to predict the direction of movement of pedestrians in the next moment based on the pedestrian's historical trajectory information and the semantic segmentation map of the current scene.

[0076] ① Input and output

[0077] The direction prediction module takes as input a historical trajectory map derived from the pedestrian's historical trajectory points (a two-dimensional coordinate sequence at 8 time points) and a semantic segmentation map of the current scene (containing scene information such as roads, obstacles, and sidewalks). The output is a probability map of 9 types of movement directions: left-front, front, right-front, left, stationary, right, left-back, front-back, and right-back, corresponding to a 3×3 grid probability distribution.

[0078] ② Network Structure

[0079] This embodiment employs a 2D CNN architecture to construct a direction prediction network. The network first selects the obstacle map surrounding the target and its own motion trajectory map as inputs to the convolutional blocks. Each convolutional block consists of convolutional layers, batch normalization layers, and a ReLU activation function. Convolution enables the joint extraction of spatiotemporal features, capturing the spatiotemporal correlation between scene information and historical trajectory direction features. The decoder consists of deconvolutional layers, upsampling the fused features extracted by the encoder to a 3×3 grid size. A Softmax activation function is then used to output the probability values ​​for each direction, forming a direction probability map.

[0080] ③ Loss Function

[0081] The direction prediction module is trained using a direction-weighted loss function, and a tiered penalty mechanism is used to finely constrain the direction prediction error. Taking the true direction as 1 (directly forward) as an example, the loss weight matrix is ​​as follows: Figure 4 As shown, where:

[0082] Weights in the same direction are 0 (no penalty);

[0083] Strongly adjacent directions (such as front to left front) have a weight of 1 (light penalty);

[0084] Moderate deviation from the direction (such as straight ahead → left) has a weight of 2 (moderate penalty);

[0085] Strong deviations from the direction (such as straight ahead → left rear) have a weight of 3 (severe penalty);

[0086] Completely opposite directions (such as forward → backward) have a weight of 4 (the heaviest penalty).

[0087] The formula for calculating the direction-weighted loss function is as follows:

[0088]

[0089] in, The weight matrix is ​​the direction-weighted loss matrix. The logit represents the true class output by the model. This represents the logit corresponding to the k-th category of the model output. The logit refers to the raw output value of the last layer of the model before it is processed by the activation function.

[0090] ④ Training methods

[0091] The direction prediction module is trained independently, taking the semantic segmentation map and historical trajectory map as input, and is optimized through the aforementioned direction-weighted loss function until the model converges, thus obtaining the trained direction prediction module.

[0092] (2) Trajectory prediction network

[0093] When the trajectory prediction network is trained alone, it only needs to input the pedestrian's historical trajectory coordinates (eight time points), with the direction input calculated using the historical trajectory. When the two networks are trained collaboratively, the direction input uses the output of the direction prediction network.

[0094] The trajectory prediction network includes the following sub-modules:

[0095] 2.1 Bidirectional Temporal Feature Encoding Module

[0096] The historical trajectory coordinates are input into the Bi-LSTM to obtain the forward hidden state and the backward hidden state. The forward hidden state encodes the pedestrian's movement trend from the past to the present, while the backward hidden state encodes the pedestrian's movement intention from the future to the present.

[0097] 2.2 Bidirectional Feature Fusion Module

[0098] The bidirectional gating attention fusion module described above is used to fuse the forward and backward hidden states output by Bi-LSTM to generate bidirectional temporal features.

[0099] 2.3 Multimodal Feature Fusion Module

[0100] A multi-head self-attention fusion unit is designed to achieve cross-modal deep fusion of the pedestrian motion direction probability map output by the direction prediction module, the bidirectional fused temporal features output by the bidirectional feature fusion module, and the pedestrian's current coordinate information. These three types of features reflect the pedestrian's motion characteristics from three dimensions: direction semantics, temporal trend, and spatial location, respectively. They differ in feature dimensions and types. The multi-head self-attention mechanism can learn the importance of different features through attention weights, achieving adaptive feature fusion. The specific process is as follows:

[0101] 1) Unified feature dimensions: The orientation probability map and the pedestrian's current coordinates are mapped through a fully connected layer to obtain orientation features. Spatial coordinate characteristics and bidirectional timing characteristics ;

[0102] 2) Feature splicing: , , Feature concatenation is performed to obtain multimodal features. = , .

[0103] 3) Multi-head self-attention fusion: Multimodal features Fm are input into a multi-head self-attention layer. The attention weights of different modal features are learned through the self-attention mechanism to achieve cross-modal deep fusion and obtain the fused comprehensive features. The formula is:

[0104] =

[0105] The multimodal feature fusion unit achieves three-dimensional feature complementarity of "directional semantic constraints, bidirectional temporal trends and spatial location information", providing comprehensive and accurate feature support for trajectory prediction and solving the problem of insufficient feature mining in traditional models.

[0106] 2.4 Trajectory Coordinate Prediction Module

[0107] The integrated characteristics after fusion Without going through a decoder, the data is directly input to the fully connected layer, which outputs the predicted pedestrian trajectory coordinates for the next time step. By repeating the above prediction steps through a loop, a predicted pedestrian trajectory for a specified time period is generated.

[0108] 3. Model Training

[0109] The pedestrian trajectory prediction model proposed in this invention consists of two networks and adopts a phased collaborative training strategy to achieve independent optimization and collaborative linkage of each module.

[0110] like Figure 3 As shown, the specific training steps are as follows:

[0111] Phase 1: Independent training of the direction prediction module. Using the semantic segmentation map and historical trajectory map as input, the direction prediction module is independently trained through a direction-weighted loss function until the model converges, resulting in a trained direction prediction module that ensures the accuracy of direction prediction.

[0112] Phase 2: Freezing the training of the trajectory prediction network. The parameters of the trained direction prediction module are frozen, and its output direction probability map is input into the trajectory prediction network to train the trajectory prediction network and optimize the parameters of the bidirectional feature fusion module and the trajectory prediction network.

[0113] Phase 3: Joint Model Fine-Tuning. The parameters of the unfreezing direction prediction module are jointly fine-tuned, with the overall model's trajectory prediction loss as the optimization objective. This achieves synergistic optimization of each module and improves the overall performance of the model.

[0114] Through phased training, independent optimization of each module was achieved, ensuring that each module could achieve optimal performance. At the same time, joint fine-tuning enabled the coordinated linkage of each module, solving the feature adaptation problem between modules.

[0115] 4. Loss Function

[0116] This model uses the Mean Squared Error (MSE) loss function during training, and the calculation formula is as follows:

[0117]

[0118] Where n is the prediction length, and in the experiment n=12. ( ) represents the actual pedestrian coordinates, ) represents the predicted pedestrian coordinates output by the model.

[0119] 5. Model Effects

[0120] Through the aforementioned core design, this invention achieves full utilization of semantic information about pedestrian trajectory direction, accurate fusion of bidirectional temporal features, and collaborative optimization of multiple modules. Compared with existing technologies, it can significantly reduce average displacement error (ADE) and final displacement error (FDE), improve prediction accuracy in key scenarios (such as turning and changing direction), and enhance the model's generalization ability and robustness. It is not only suitable for pedestrian trajectory prediction systems but also adaptable to the trajectory prediction needs of various road users such as bicycles, electric two-wheelers, and cars, and can be widely applied in the fields of autonomous driving and intelligent transportation.

[0121] The technical effects of the present invention are verified by experimental data below.

[0122] (1) Experimental setup

[0123] Datasets: Validated on public trajectory prediction datasets SDD (Standford Drone Dataset), ETH, and UCY, including pedestrian trajectory data and complex scenarios (such as intersections and squares).

[0124] Baseline Model: More than ten mainstream trajectory prediction models were selected for comparison.

[0125] Evaluation metrics: Two core evaluation metrics in trajectory prediction, the Average Displacement Error (ADE) and the Final Displacement Error (FDE), are used to quantitatively evaluate the model's performance. Smaller values ​​indicate higher prediction accuracy.

[0126] Average Displacement Error (ADE): Calculated as the average Euclidean distance between the predicted and actual trajectories of a pedestrian across all time steps, measuring the overall deviation between the predicted and actual trajectories. The formula is:

[0127]

[0128] in, , To predict coordinates, , For the actual coordinates, For predicting time steps;

[0129] Final Displacement Error (FDE): Calculates the Euclidean distance between the predicted endpoint of the pedestrian's trajectory and the actual endpoint of the trajectory, measuring the deviation from the predicted endpoint. The formula is:

[0130]

[0131] in, , The predicted coordinates for the last time step. , These are the actual coordinates of the last time step.

[0132] (2) Experimental results of trajectory prediction model

[0133] The experimental results on the ETH / UCY dataset are shown in Table 1:

[0134] Table 1. Experimental results of the trajectory prediction model on the ETH / UCY dataset. Unit: meters.

[0135] Method ETH HOTEL UNIV ZARA1 ZARA2 AVG Social-LSTM 1.09 / 2.35 0.79 / 1.76 0.67 / 1.40 0.47 / 1.00 0.56 / 1.17 0.72 / 1.54 Social-GAN 0.81 / 1.52 0.72 / 1.61 0.60 / 1.26 0.34 / 0.69 0.42 / 0.84 0.58 / 1.18 ST-GAT 0.65 / 1.12 0.35 / 0.66 0.52 / 1.10 0.34 / 0.69 0.29 / 0.60 0.43 / 0.83 Transformer-TF 0.61 / 1.12 0.18 / 0.30 0.35 / 0.65 0.22 / 0.38 0.17 / 0.32 0.31 / 0.55 STAR 0.36 / 0.65 0.17 / 0.36 0.31 / 0.62 0.26 / 0.55 0.22 / 0.46 0.26 / 0.53 Trajectron++ 0.39 / 0.83 0.12 / 0.21 0.20 / 0.44 0.15 / 0.33 0.11 / 0.25 0.19 / 0.41 AgentFormer 0.26 / 0.39 0.11 / 0.14 0.26 / 0.46 0.15 / 0.23 0.14 / 0.24 0.18 / 0.29 Goal-GAN 0.59 / 1.18 0.19 / 0.35 0.60 / 1.19 0.43 / 0.87 0.32 / 0.65 0.43 / 0.85 PECNet 0.54 / 0.87 0.18 / 0.24 0.35 / 0.60 0.22 / 0.39 0.17 / 0.30 0.29 / 0.48 MG-GAN 0.47 / 0.91 0.14 / 0.24 0.54 / 1.07 0.36 / 0.73 0.29 / 0.60 0.36 / 0.71 Y-net 0.28 / 0.33 0.10 / 0.14 0.24 / 0.41 0.17 / 0.27 0.13 / 0.22 0.18 / 0.27 Goal-SAR 0.28 / 0.39 0.12 / 0.17 0.25 / 0.43 0.17 / 0.26 0.15 / 0.22 0.19 / 0.29 Ours 0.18 / 0.22 0.10 / 0.12 0.14 / 0.17 0.10 / 0.13 0.09 / 0.10 0.12 / 0.14

[0136] Experimental results show that on the ETH / UCY dataset, the average ADE of the proposed method is 0.122 meters, which is 36.8% lower than the second-ranked Goal-SAR model, and the FDE is 0.148 meters, which is 51.7% lower.

[0137] The experimental results on the SDD dataset are shown in Table 2:

[0138] Table 2. Experimental results of the trajectory prediction model on the SDD dataset. Unit: pixels.

[0139] Method ADE FDE SGAN 27.23 41.44 STGAT 14.23 26.67 Trajectron++ 9.90 16.80 PECNet 9.96 15.88 MID 9.73 15.32 MemoNet 8.56 12.70 GroupNet 9.31 16.11 End-to-End 8.60 13.90 GSMNet 8.30 12.70 Goal-SAR 7.75 11.83 Ours 5.68 7.31

[0140] Experiments show that on the SDD dataset, the ADE of the proposed method is 5.68 pixels, which is 18.1% lower than the second-ranked SocialVAE model, and the FDE is 7.31 pixels, which is 22.7% lower.

[0141] Example 2

[0142] This embodiment provides a pedestrian trajectory prediction system based on Bi-LSTM and multi-head self-attention, the system comprising:

[0143] The bidirectional temporal feature encoding module inputs the historical trajectory coordinates of the pedestrian into the Bi-LSTM to obtain the forward hidden state and the backward hidden state.

[0144] The bidirectional feature fusion module inputs the forward hidden state and the backward hidden state into the bidirectional gated attention fusion module to generate bidirectional temporal features.

[0145] The direction prediction module generates a direction probability map based on the pedestrian's historical trajectory coordinates and the semantic segmentation map of the current scene;

[0146] The multimodal feature fusion module acquires the orientation probability map, the pedestrian's current coordinate information, and bidirectional temporal features. After concatenating the orientation probability map, current coordinate information, and bidirectional temporal features, it inputs them into a multi-head self-attention encoder for cross-modal deep fusion to obtain comprehensive features.

[0147] The trajectory coordinate prediction module directly inputs the comprehensive features into the fully connected layer and outputs the predicted pedestrian trajectory coordinates for the next moment.

[0148] The specific implementation methods of each module of the system in this embodiment are the same as the corresponding steps in Embodiment 1, and will not be repeated here.

[0149] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A pedestrian trajectory prediction method based on Bi-LSTM and multi-head self-attention, characterized in that, Includes the following steps: Step S1: Obtain the historical trajectory coordinates of the pedestrian; Step S2: Input the historical trajectory coordinates into the bidirectional long short-term memory network Bi-LSTM to obtain the forward hidden state and the backward hidden state; Step S3: Input the forward hidden state and the backward hidden state into the bidirectional gated attention fusion module, and generate the fused bidirectional temporal features through the gated attention mechanism; Step S4: Input the pedestrian's historical trajectory coordinates and the semantic segmentation map of the current scene into the direction prediction module to generate a direction probability map; obtain the pedestrian's current coordinate information and bidirectional temporal features; after concatenating the direction probability map, current coordinate information and bidirectional temporal features, input them into the multi-head self-attention encoder for cross-modal deep fusion to obtain comprehensive features; Step S5: Input the integrated features directly into the fully connected layer and output the predicted pedestrian trajectory coordinates for the next time step.

2. The method according to claim 1, characterized in that, Step S3 specifically includes: Step S31: Perform linear transformations on the forward hidden state and the backward hidden state respectively, and generate forward gating weights and backward gating weights through the Sigmoid activation function; Step S32: After linear transformation, the forward hidden state and the backward hidden state are input into the multi-head self-attention layer to obtain the optimized forward hidden state and backward hidden state; Step S33: The forward hidden state and the optimized forward hidden state are weighted and summed using forward gating weights. The reverse hidden state and the optimized reverse hidden state are weighted and summed using reverse gating weights. The weighted sums are then fused to obtain bidirectional temporal features.

3. The method according to claim 2, characterized in that, The calculation formula for the dynamic weighted fusion is as follows: in, and These are the forward hidden state and the reverse hidden state, respectively. and These are the forward gating weights and the backward gating weights, respectively. and These represent the optimized forward hidden state and backward hidden state, respectively, with a and b being the preset bidirectional fusion weights. This indicates element-wise multiplication.

4. The method according to claim 3, characterized in that, The preset bidirectional fusion weights a and b are set to 0.8 and 0.2 respectively.

5. The method according to claim 1, characterized in that, The direction prediction module takes the pedestrian's historical trajectory points and the semantic segmentation map of the current scene as input, and outputs a probability map of the pedestrian's movement direction at the next moment.

6. The method according to claim 1, characterized in that, In step S4, the direction probability map, current coordinate information, and bidirectional temporal features are concatenated, specifically including: The orientation probability map and current coordinate information are mapped to 64 dimensions respectively through a fully connected layer; By concatenating the directional features, spatial coordinate features, and bidirectional temporal features of dimension 128, a 256-dimensional multimodal feature is obtained.

7. The method according to claim 1, characterized in that, In step S4, the multi-head self-attention encoder learns the attention weights of different modal features through a self-attention mechanism, thereby achieving cross-modal deep fusion of the orientation probability map, current coordinate information, and bidirectional temporal features.

8. The method according to claim 1, characterized in that, By repeating steps S2 to S5 in a cyclical prediction manner, a pedestrian prediction trajectory for a specified time length is generated; the specified time length is 12 time steps.

9. The method according to claim 1, characterized in that, The pedestrian trajectory prediction model employs a phased collaborative training strategy, including: Phase 1: Independently train the direction prediction module until the model converges; The second stage: Freeze the parameters of the direction prediction module after training, and train the trajectory prediction network by outputting the direction probability map through the direction prediction module. The third stage involves unfreezing the parameters of the direction prediction module and fine-tuning the trajectory prediction network and the direction prediction module together, with the overall model's trajectory prediction loss as the optimization objective.

10. A pedestrian trajectory prediction system based on Bi-LSTM and multi-head self-attention, characterized in that, include: The bidirectional temporal feature encoding module inputs the historical trajectory coordinates of the pedestrian into the Bi-LSTM to obtain the forward hidden state and the backward hidden state. The bidirectional feature fusion module inputs the forward hidden state and the backward hidden state into the bidirectional gated attention fusion module to generate bidirectional temporal features. The direction prediction module generates a direction probability map based on the pedestrian's historical trajectory coordinates and the semantic segmentation map of the current scene; The multimodal feature fusion module takes the orientation probability map, the pedestrian's current coordinate information, and bidirectional temporal features as input. After concatenating the orientation probability map, current coordinate information, and bidirectional temporal features, it inputs them into a multi-head self-attention encoder for cross-modal deep fusion to obtain comprehensive features. The trajectory coordinate prediction module directly inputs the comprehensive features into the fully connected layer and outputs the predicted pedestrian trajectory coordinates for the next moment.

Images