Pedestrian track prediction method and device based on pedestrian scene interaction modeling

Abstract

The invention discloses a pedestrian trajectory prediction method and device based on pedestrian scene interactive modeling, and the method comprises the steps: extracting the trajectory feature space of each pedestrian through employing a multi-feature map, obtaining the feature maps of the relative position, speed and acceleration, and carrying out the convolution of each feature map, superposing to obtain pedestrian interaction features; encoding the scene semantic segmentation map by using a SwinTransform, mapping the pedestrian coordinate sequence into a position map, and aligning and fusing the position map with the semantic features to obtain space-time scene representation; cross-modal fusion of pedestrian interaction features and scene features is completed through a bidirectional self-attention mechanism; predicting a bivariate Gaussian distribution parameter of a future trajectory by using a time extrapolation convolutional network, and performing end-to-end training by using negative log-likelihood loss; according to the method, pedestrian interaction behaviors and scene constraints are comprehensively considered, and the future trajectory of the pedestrian can be more accurately predicted in a crowded and complex scene.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a pedestrian trajectory prediction method and device based on pedestrian scene interaction modeling. Background Art

[0002] Human trajectory prediction aims to predict the future trajectories of pedestrians based on historical observation data. In daily life, human trajectory prediction has a wide range of applications across multiple fields, including advanced surveillance systems, autonomous vehicles, social robotics, and human-computer interaction. In surveillance systems, pedestrian trajectory prediction can be used to identify and track pedestrian activities, providing more efficient monitoring solutions. By predicting pedestrian trajectories, autonomous vehicles can demonstrate intelligent behavior similar to that of human drivers, better adapting to complex traffic environments and avoiding potential collision risks. In the fields of social robotics and human-computer interaction, predicting pedestrian trajectories enables robots to interact with humans more naturally, avoid collisions, and provide a more humane service experience.

[0003] Existing pedestrian trajectory prediction methods utilize the Transformer architecture, which has a strong ability to model long-term and short-term temporal dependencies. However, due to its inherent lack of prior knowledge about pedestrian motion patterns, the Transformer lacks the ability to generalize behavior across diverse scenarios. Furthermore, the original Transformer lacks sensitivity to local spatiotemporal details, making it difficult to accurately capture fine-grained variations in trajectories. This is particularly true for modeling complex interactive behaviors. Furthermore, current research still faces limitations in integrating pedestrian scene features. Many methods fail to fully consider the interaction between scene information and pedestrian trajectories, often relying solely on historical trajectory information or simply concatenating scene features with trajectory features, lacking a deep feature fusion mechanism. This approach fails to effectively model the relationship between scene and trajectory, resulting in insufficient utilization of environmental information and making it difficult for the model to make accurate predictions in complex and dynamic scene changes. Therefore, for pedestrian trajectory prediction, more advanced trajectory modeling techniques and feature fusion strategies are needed to improve the accuracy of prediction models in complex environments. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a pedestrian trajectory prediction solution with high trajectory prediction accuracy in complex scenarios.

[0005] In order to achieve the above-mentioned objectives, the present invention provides, on the one hand, a pedestrian trajectory prediction method based on pedestrian scene interaction modeling, comprising: step 1: using multiple feature maps to extract the trajectory feature space of each pedestrian, obtaining feature maps of relative position, velocity and acceleration, and performing graph convolution on each feature map respectively, and superimposing them to obtain pedestrian interaction features; step 2: using Swin Transformer to encode the scene semantic segmentation map, and mapping the pedestrian coordinate sequence into a position map, aligning and fusing it with the semantic features to obtain spatiotemporal scene features; step 3: cross-modally fusing the pedestrian interaction features with the spatiotemporal scene features through a bidirectional self-attention mechanism to obtain scene trajectory features; step 4: inputting the fused scene trajectory features into a time extrapolation convolutional network to predict the bivariate Gaussian distribution parameters of the future trajectory, and performing end-to-end training with negative log-likelihood loss until convergence, and finally using the converged model to predict pedestrian trajectories.

[0006] Optionally, the step 1 includes: using the relative positions of pedestrians to define the interaction based on relative positions, including defining a feature map G p =(V p ,E p ), where V p Represents the node features of pedestrians. The features of each node are defined as the coordinates of the pedestrian in the coordinate system E P The edge set that represents the position correlation between graph nodes is defined as follows in, represents the position of pedestrian i at time t, represents the position of pedestrian j at time t, and ò represents a very small positive number to avoid the denominator being zero;

[0007] Use the pedestrian's velocity at time step t to define velocity-based interactions, including constructing the feature map G v =(V v ,E v ), where V v Represents the speed characteristics of pedestrians. The characteristics of each node are defined as the speed of the pedestrian in the coordinate system. E v The edge set representing the speed correlation between graph nodes is defined as follows in, represents the speed of pedestrian i at time t, represents the speed of pedestrian j at time t, and ò represents a very small positive number to avoid the denominator being zero;

[0008] Use the acceleration of the pedestrian at time step t to define the acceleration-based interaction; construct the feature map G a =(V a ,E a ), where Va Represents the speed characteristics of pedestrians. The characteristics of each node are defined as the acceleration of the pedestrian in the coordinate system. is the set of edges representing the acceleration correlation between graph nodes, E a The edge set representing the acceleration correlation between graph nodes is defined as follows in, represents the acceleration of pedestrian i at time t, represents the acceleration of pedestrian j at time t, and ò represents a very small positive number to avoid the denominator being zero;

[0009] Perform graph convolution operation on each feature map separately.

[0010]

[0011]

[0012] Where σ(·) represents the Sigmoid activation function, Represent the position, velocity, and acceleration feature matrices of the lth layer respectively, Represents the weight matrix corresponding to the lth layer, A p , A v , A a represents the corresponding adjacency matrix;

[0013] Based on the splicing features, the convolution output of each feature map is superimposed to obtain the final fusion feature:

[0014] Optionally, in step 1, a graph attention network is used to model the fusion features, and then a Transformer is used to capture the time dependency along the time axis to obtain pedestrian interaction features.

[0015] Optionally, the step 2 includes: using Swin Transformer to extract the C s ×H×W scene semantic features S, where C s represents the number of semantic categories, H and W are the spatial resolutions;

[0016] For each frame, the two-dimensional coordinates of the pedestrian (x t ,y t ), where t∈[1,T obs ] represents the observation time, T obs Represents the length of the observation sequence; First, the normalized coordinates of the pedestrian are mapped to the pixel coordinate system of the scene graph to obtain the integer position (i t ,j t ); Initialize a two-dimensional zero matrix M of size H×W t, and set the pixel value corresponding to the pedestrian's current position to 1, and other positions to 0;

[0017] Generate T in this way obs Frame position map Splicing to get an information tensor containing the time dimension Position graph sequence and scene semantic graph Aligned in spatial dimensions, where C s is the number of semantic categories;

[0018] The pedestrian position map M of each frame is spliced ​​with the global semantic segmentation map S in the channel dimension to form the scene feature information

[0019] Optionally, step 3 includes: performing a self-attention mechanism on pedestrian interaction features and scene features respectively; using pedestrian interaction features as query sources and scene features as reference information to model pedestrian perception of the environment; using scene features as query sources and pedestrian interaction features as reference information to model the environment's feedback on pedestrian behavior; and using a gated network to obtain fused trajectory features and environmental features.

[0020] Optionally, step 4 includes: passing the output into a time-extrapolated convolutional network with a residual connection to obtain a predicted trajectory including five binary Gaussian distribution parameters;

[0021] The predicted trajectory information is obtained according to the Gaussian distribution parameters.

[0022] Optionally, in step 4, the parameters in the network training process are corrected by gradient descent, and the Adam optimizer is used to adjust the model parameters by continuously iterating the training data to minimize the loss function.

[0023] Optionally, the loss function is in, Represents the predicted coordinates of pedestrian n at time t, which follows a bivariate Gaussian distribution. represents the average value, represents the standard deviation, represents the correlation coefficient, W represents all parameters of model training, T obs represents the observation length, T pred Indicates the predicted length.

[0024] On the other hand, the present invention also provides a pedestrian trajectory prediction device based on pedestrian scene interaction modeling, including: a pedestrian interaction feature extraction unit, which is used to extract the trajectory feature space of each pedestrian using multiple feature maps, obtain feature maps of relative position, velocity and acceleration, and perform graph convolution on each feature map respectively, and superimpose them to obtain pedestrian interaction features; a scene trajectory feature extraction unit, which is used to encode the scene semantic segmentation map using Swin Transformer, and map the pedestrian coordinate sequence into a position map, align and fuse it with the semantic features to obtain spatiotemporal scene features; a fusion unit, which is used to cross-modally fuse the pedestrian interaction features with the spatiotemporal scene features through a bidirectional self-attention mechanism to obtain scene trajectory features; a prediction unit, which is used to input the fused scene trajectory features into a time extrapolation convolutional network to predict the bivariate Gaussian distribution parameters of the future trajectory, and perform end-to-end training with negative log-likelihood loss until convergence, and finally use the converged model to predict the pedestrian trajectory.

[0025] The advantages of this invention over existing technologies are as follows: This prediction method, through a graph construction mechanism that integrates prior knowledge, combines multi-feature graphs with Transformer modeling methods, and proposes a multi-feature graph structure to construct a new pedestrian spatial mapping matrix to more accurately capture the interaction characteristics between pedestrians. Furthermore, through a cross-attention module, contextual features that are instructive for the current pedestrian movement are extracted from environmental information, while pedestrian movement features are simultaneously injected into environmental perception, achieving bidirectional modeling between features. This mechanism not only obtains a trajectory representation constrained by the environment, but also extracts environmental features that include the influence of pedestrian movement, effectively improving trajectory prediction accuracy in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flowchart of a pedestrian trajectory prediction method based on pedestrian scene interaction modeling provided by one embodiment of the present invention;

[0027] Figure 2 This is a framework diagram of a pedestrian trajectory prediction model provided by one embodiment of the present invention;

[0028] Figure 3 is a gating model diagram provided by an embodiment of the present invention;

[0029] Figure 4 It is a structural diagram of a pedestrian trajectory prediction device based on pedestrian scene interaction modeling provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] Reference Figure 1-3 This embodiment provides a pedestrian trajectory prediction method based on pedestrian scene interaction modeling, comprising the following steps:

[0032] Step S1: Data Acquisition. In different scenarios, accurate calibration is performed using target detection and tracking algorithms to obtain the historical trajectories of each pedestrian in the scene. In this embodiment, the coordinate values ​​of each pedestrian in the image coordinate system are extracted from each frame of the road video. This creates a detailed pedestrian trajectory dataset containing the pedestrian's location coordinates, timestamp, and pedestrian ID.

[0033] Step S2: For the acquired trajectory data, construct a feature map based on position, velocity and acceleration.

[0034] Specifically, the position-based feature map: uses the relative positions of pedestrians to define the position-based interaction; define the feature map G p =(V p ,E p ), where V p Represents the node features of pedestrians. The features of each node are defined as the coordinates of the pedestrian in the coordinate system E P A set of edges representing the positional dependencies between graph nodes. The edges are defined as follows:

[0035]

[0036] in, represents the position of pedestrian i at time t, represents the position of pedestrian j at time t, and ò represents a very small positive number to avoid the denominator being zero.

[0037] Speed-based feature map: Use the speed of pedestrians at time step t to define speed-based interactions; construct feature map G v =(V v ,E v ), where V v Represents the speed characteristics of pedestrians. The characteristics of each node are defined as the speed of the pedestrian in the coordinate system. E v The set of edges representing the speed correlation between graph nodes. The edges are defined as follows:

[0038]

[0039] in, represents the speed of pedestrian i at time t, represents the speed of pedestrian j at time t, and ò represents a very small positive number to avoid the denominator being zero.

[0040] Acceleration-based feature map: Use the acceleration of the pedestrian at time step t to define the acceleration-based interaction; construct the feature map G a =(V a ,E a ), where V a Represents the acceleration characteristics of pedestrians. The characteristics of each node are defined as the acceleration of the pedestrian in the coordinate system E a is a set of edges representing the acceleration correlation between graph nodes. The edges are defined as follows:

[0041]

[0042] in, represents the acceleration of pedestrian i at time t, represents the acceleration of pedestrian j at time t, and ò represents a very small positive number to avoid the denominator being zero.

[0043] Step S3: Based on the output from step S2, a graph convolutional neural network (GCN) is used to extract spatial interaction information and perform weighted fusion of different feature information. During the graph convolution operation, information from different feature maps is integrated, and a graph convolutional neural network (GCN) is used to capture the complex interactions between nodes. A graph convolution operation is performed on each feature map, and then their outputs are element-wise added to achieve feature fusion and obtain pedestrian interaction features. The specific calculation steps are as follows:

[0044]

[0045] Among them, σ(·) represents the Sigmoid activation function, Represent the position, velocity, and acceleration feature matrices of the lth layer respectively, Represents the weight matrix corresponding to the lth layer, A p , A v , A a represents the corresponding adjacency matrix.

[0046] Based on the splicing features, the convolution output of each feature map is superimposed to obtain the final fusion feature:

[0047]

[0048] Step S4: In the second stage, the graph attention network GAT is used to further optimize the node features based on the GCN prior knowledge. According to the idea of ​​GAT, for each node v in the graph i , its new representation vector is calculated by the following process:

[0049]

[0050] Among them, h j is the input feature vector of node j; W is the learnable linear transformation weight matrix; N(i) is the set of neighbor nodes of node i (which may include itself); σ(·) is the LeakyReLU activation function; α ij is the attention weight of neighbor node j to center node i, indicating the importance of the neighbor. The calculation formula is as follows:

[0051]

[0052] Where a is the weight vector of the self-attention mechanism. || represents the concatenation operation.

[0053] For the information F extracted by GAT, Transformer is used to extract the dependent features in the time dimension. The final pedestrian trajectory feature F is obtained x , the specific formula is as follows:

[0054] F x = TransformerEncoder(F);

[0055] Step S4: For scene information, use SwinTransformer to extract the size of C s ×H×W scene semantic features S, where C s is the number of semantic categories, H and W are the spatial resolutions.

[0056] The size of the global scene graph is H×W, and the two-dimensional coordinates of the pedestrian in each frame are (x t ,y t ), where t∈[1,T obs ] represents the observation time, T obs Represents the length of the observation sequence. First, the normalized coordinates of the pedestrian are mapped to the pixel coordinate system of the scene graph to obtain the integer position (i t ,j t ). Then, initialize a two-dimensional zero matrix M of size (H×W) t , and set the pixel value corresponding to the pedestrian's current position to 1, and other positions to 0.

[0057] Generate T in this way obs Frame position map Splicing to get an information tensor containing the time dimension The position graph sequence and the scene semantic graph (where C s is the number of semantic categories) are aligned in the spatial dimension to facilitate subsequent splicing and joint encoding.

[0058] In order to further integrate pedestrian trajectories and scene semantic information, the pedestrian position map M of each frame is spliced ​​with the global semantic segmentation map S in the channel dimension to form scene feature information

[0059] F s =Concat(M,S)

[0060] Step S5: The scene feature F s And pedestrian trajectory features F x Embed position encoding, extract according to the self-attention mechanism, and fuse with the cross-attention mechanism. The specific implementation steps are as follows:

[0061] Positional encoding embedding introduces the position information of elements in the sequence, and adds positional encoding respectively:

[0062]

[0063] The position encoding formula is as follows

[0064]

[0065] Where: pos represents the position of the element in the sequence (starting from 0); i represents the current dimension index; d represents the dimension of the embedding.

[0066] Pedestrian interaction features Perform self-attention calculation:

[0067]

[0068] in represents the weight matrix, and d represents the feature dimension

[0069] The scene features Perform self-attention calculation:

[0070]

[0071] in represents the weight matrix, and d represents the feature dimension

[0072] For pedestrian interaction features and scene features after self-attention operation, a cross-attention mechanism is introduced to learn pedestrian motion features under environmental constraints; pedestrian interaction features are used as query sources, scene features are used as query sources, and the cross-attention mechanism is introduced to learn pedestrian motion features under environmental constraints.

[0073] The features are used as reference information to model pedestrians’ perception of the environment;

[0074]

[0075]

[0076] in represents the weight matrix, and d represents the feature dimension

[0077] Similarly, in order to learn the environmental features that include the influence of pedestrian motion, the scene features are used as the query source and the pedestrian interaction features are used as reference information to model the feedback of the environment on pedestrian behavior:

[0078]

[0079] in represents the weight matrix, and d represents the feature dimension

[0080] In order to further explore the relationship between different features, it involves inputting various features into a multi-layer feedforward neural network, which contains nonlinear activation functions to facilitate training of the feedforward network fusion. After processing of self-attention and cross-attention, the model further enhances the feature representation capability through the feedforward neural network.

[0081] The structure of a feedforward network usually consists of two linear transformations and a nonlinear activation function.

[0082]

[0083] in W1 x , W1 s Represents the weight matrix.

[0084] In order to integrate the above spatial relationship information, a gating mechanism is introduced to convert the scene flow information U s With trajectory flow information U x Specifically, the calculation formula of the gate value g is as follows:

[0085] g=σ(f X (U x )+f S (U s )+B)

[0086] Where g(·) represents the Sigmoid activation function; f S (·), f X (·) represents the fully connected transformation (i.e., fully connected layer); B is the bias term. The final fusion result is expressed as:

[0087] Ωfusion =g·U f +(1-g)·U f

[0088] This gate mechanism enables the model to adaptively adjust the weights of the two information streams according to different scenarios and behavioral dynamics, thereby improving the expressiveness and predictive performance of feature fusion.

[0089] Step S6: Obtain fusion feature Ω fusion Finally, the output is passed into the TXP-CNN with residual connection to obtain the predicted trajectory, including five binary Gaussian distribution parameters, including five binary Gaussian distribution parameters, and the predicted trajectory information is obtained according to the obtained Gaussian distribution parameters.

[0090] Probability density function of the bivariate Gaussian distribution

[0091]

[0092] where μ x ,μ y Represents the mean vector of the predicted position, and ∑ represents the covariance matrix:

[0093] where σ x ,σ y represents the standard deviation, and ρ represents the correlation coefficient.

[0094] Set the loss function, calculate the loss value of the predicted trajectory and the actual trajectory, and correct the parameters in the training network process through gradient descent, so that the predicted trajectory of the pedestrian is as consistent as possible with the actual trajectory.

[0095] The loss function is defined as

[0096]

[0097] in, Represents the predicted coordinates of pedestrian n at time t, which follows a bivariate Gaussian distribution. represents the average value, represents the standard deviation, represents the correlation coefficient, W represents all parameters of model training, T obs represents the observation length, T pred Indicates the predicted length.

[0098] By using gradient descent, the parameters in the training network are corrected, and ultimately the predicted pedestrian trajectory is made as consistent as possible with the real trajectory. By continuously iterating the training data, the Adam optimizer is used to adjust the model parameters to minimize the loss function.

[0099] The dataset was divided into several subsets, with one subset selected at a time as the validation set and the remaining subsets used as the training set. Training and validation were repeated to evaluate the generalization ability of the model on different datasets. This was performed using the PyTorch deep learning framework, using the Adam optimizer for optimization, with a learning rate of 0.01 and a batch size of 64.

[0100] The training dataset is used to train the weight parameters and model parameters of the various networks involved in the method, calculate various loss functions, select the best predicted trajectory, save the corresponding various weight parameters and model parameters, obtain the trained prediction model, and then use the test dataset to perform the above steps to predict the future trajectory of pedestrians.

[0101] Although the Transformer structure has the ability to model long-term and short-term temporal dependencies through a multi-head self-attention mechanism, it still faces two major challenges in pedestrian trajectory prediction tasks: first, the lack of prior modeling of pedestrian motion patterns requires a large amount of data training to achieve strong generalization capabilities; second, the original Transformer has deficiencies in local information modeling, making it difficult to capture fine-grained behavioral changes. Considering the above issues, the pedestrian trajectory prediction method provided in this embodiment uses a graph construction mechanism that integrates prior knowledge, combines multiple feature graphs with the Transformer modeling method, and proposes a multi-feature graph structure to construct a new pedestrian spatial mapping matrix to more accurately capture the interaction characteristics between pedestrians.

[0102] In addition, many traditional trajectory prediction methods mainly focus on modeling the historical trajectory of pedestrians, while ignoring the semantic information of the environment in which the pedestrians are located. Although some methods have introduced scene images or semantic segmentation maps, the fusion method is mostly direct splicing or simple weighting, which lacks the ability to model the deep correlation between the two types of information. To this end, the pedestrian trajectory prediction method provided in this embodiment uses a feature fusion method based on a cross-attention mechanism to achieve a deep fusion of pedestrian features and environmental features. Through the cross-attention module, this method extracts contextual features that are instructive for the current pedestrian movement from the environmental information, and at the same time, reversely injects the pedestrian movement features into the environmental perception to achieve two-way modeling between features. Through this mechanism, on the one hand, a trajectory representation constrained by the environment is obtained, and on the other hand, environmental features affected by pedestrian movement are extracted, thereby effectively improving the trajectory prediction accuracy in complex scenes.

[0103] Reference Figure 4 This embodiment further provides a pedestrian trajectory prediction device based on pedestrian scene interaction modeling, comprising:

[0104] The pedestrian interaction feature extraction unit 100 is used to extract the trajectory feature space of each pedestrian using multiple feature maps, obtain feature maps of relative position, velocity and acceleration, and perform graph convolution on each feature map respectively, and superimpose them to obtain pedestrian interaction features; it should be noted that since the specific extraction method and process have been explained in detail in steps S2 and S3 of the above-mentioned pedestrian trajectory prediction method based on pedestrian scene interaction modeling, they will not be repeated here.

[0105] The scene trajectory feature extraction unit 200 is used to encode the scene semantic segmentation map using SwinTransformer, and map the pedestrian coordinate sequence into a position map, which is aligned and fused with the semantic features to obtain the spatiotemporal scene features. It should be noted that since the specific extraction method and process have been explained in detail in step S4 of the above-mentioned pedestrian trajectory prediction method based on pedestrian scene interaction modeling, they will not be repeated here.

[0106] The fusion unit 300 is used to cross-modally fuse the pedestrian interaction features with the spatiotemporal scene features through a bidirectional self-attention mechanism to obtain scene trajectory features. It should be noted that since the specific fusion method and process have been explained in detail in step S5 of the above-mentioned pedestrian trajectory prediction method based on pedestrian scene interaction modeling, they will not be repeated here.

[0107] The prediction unit 400 is used to input the fused scene trajectory features into the time extrapolation convolutional network to predict the bivariate Gaussian distribution parameters of the future trajectory, and perform end-to-end training with negative log-likelihood loss until convergence, and finally use the converged model to predict the pedestrian trajectory; it should be noted that since the specific prediction method and process have been explained in detail in step S6 of the above-mentioned pedestrian trajectory prediction method based on pedestrian scene interaction modeling, they will not be repeated here.

[0108] In addition, an embodiment of the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, which, when executed, includes part or all of the steps of any pedestrian trajectory prediction method based on pedestrian scene interaction modeling recorded in the above method embodiment.

[0109] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, and other media that can store program code.

[0111] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0112] The above describes an exemplary flowchart for implementing a pedestrian trajectory prediction method according to an embodiment of the present invention with reference to the accompanying drawings. It should be noted that the numerous details included in the above description are merely exemplary illustrations of the present invention and are not intended to limit the present invention. In other embodiments of the present invention, the method may have more, fewer, or different steps, and the order, inclusion, functionality, and other relationships between the steps may differ from those described and illustrated.

Claims

1. A pedestrian trajectory prediction method based on pedestrian scene interaction modeling, characterized in that: include: Step 1: Use multiple feature maps to extract the trajectory feature space of each pedestrian, obtain feature maps of relative position, velocity, and acceleration, and perform graph convolution on each feature map separately, and superimpose them to obtain pedestrian interaction features; Step 2: Use SwinTransformer to encode the scene semantic segmentation map and map the pedestrian coordinate sequence into a position map, which is aligned and fused with the semantic features to obtain the spatiotemporal scene features; Step 3: The pedestrian interaction features are cross-modally fused with the spatiotemporal scene features through a bidirectional self-attention mechanism to obtain scene trajectory features. Step 4: The fused scene trajectory features are input into the temporal extrapolation convolutional network to predict the bivariate Gaussian distribution parameters of the future trajectory, and end-to-end training is performed with negative log-likelihood loss until convergence. Finally, the converged model is used to predict the pedestrian trajectory.

2. The pedestrian trajectory prediction method according to claim 1, characterized in that The step 1 includes: using the relative positions of pedestrians to define the interaction based on relative positions, including defining a feature map G p =(V p ,E p ), where V p Represents the node features of pedestrians. The features of each node are defined as the coordinates of the pedestrian in the coordinate system E P The edge set that represents the position correlation between graph nodes is defined as follows in, represents the position of pedestrian i at time t, represents the position of pedestrian j at time t, and ò represents a very small positive number to avoid the denominator being zero; Use the pedestrian's velocity at time step t to define velocity-based interactions, including constructing the feature map G v =(V v ,E v ), where V v Represents the speed characteristics of pedestrians. The characteristics of each node are defined as the speed of the pedestrian in the coordinate system. E v The edge set representing the speed correlation between graph nodes is defined as follows in, represents the speed of pedestrian i at time t, represents the speed of pedestrian j at time t, and ò represents a very small positive number to avoid the denominator being zero; Use the acceleration of the pedestrian at time step t to define the acceleration-based interaction; construct the feature map G a =(V a ,E a ), where V a Represents the acceleration characteristics of pedestrians. The characteristics of each node are defined as the acceleration of the pedestrian in the coordinate system is the set of edges representing the acceleration correlation between graph nodes, E a The edge set representing the acceleration correlation between graph nodes is defined as follows in, represents the acceleration of pedestrian i at time t, represents the acceleration of pedestrian j at time t, and ò represents a very small positive number to avoid the denominator being zero; Perform graph convolution operation on each feature map obtained. Where σ(·) represents the Sigmoid activation function, Represent the position, velocity, and acceleration feature matrices of the lth layer respectively, Represents the weight matrix corresponding to the lth layer, A p , A v , A a represents the corresponding adjacency matrix; Based on the splicing features, the convolution output of each feature map is superimposed to obtain the final fusion feature:

3. The pedestrian trajectory prediction method according to claim 1 or 2, characterized in that: In step 1, the graph attention network is used to model the fusion features, and then the Transformer is used to capture the time dependency along the time axis to obtain the pedestrian interaction features.

4. The pedestrian trajectory prediction method according to claim 1, characterized in that: The step 2 includes: using SwinTransformer to extract the size of C s ×H×W scene semantic features S, where C s represents the number of semantic categories, H and W are the spatial resolutions; For each frame, the two-dimensional coordinates of the pedestrian (x t ,y t ), where t∈[1,T obs ] represents the observation time, T obs Represents the length of the observation sequence; First, the normalized coordinates of the pedestrian are mapped to the pixel coordinate system of the scene graph to obtain the integer position (i t ,j t ); Initialize a two-dimensional zero matrix M of size H×W t , and set the pixel value corresponding to the pedestrian's current position to 1, and other positions to 0; Generate T in this way obs Frame position map Splicing to get an information tensor containing the time dimension Combine the position graph sequence with the scene semantic graph Aligned in spatial dimensions, where C s is the number of semantic categories; Then, the pedestrian position map M of each frame is spliced ​​with the global semantic segmentation map S in the channel dimension to form the scene feature information 5. The pedestrian trajectory prediction method according to claim 1, characterized in that: The step 3 includes: performing a self-attention mechanism on pedestrian interaction features and scene features respectively; Using pedestrian interaction features as query sources and scene features as reference information, we model pedestrian perception of the environment. Using scene features as query sources and pedestrian interaction features as reference information, we model the environment’s feedback on pedestrian behavior. The gating network is used to obtain the fused trajectory features and environment features.

6. The pedestrian trajectory prediction method according to claim 1, characterized in that: The step 4 includes: passing the output into a time-extrapolated convolutional network with a residual connection to obtain a predicted trajectory including five binary Gaussian distribution parameters; The predicted trajectory information is obtained according to the Gaussian distribution parameters.

7. The pedestrian trajectory prediction method according to claim 6, characterized in that: In step 4, the parameters in the training network are corrected by gradient descent, and the Adam optimizer is used to adjust the model parameters by continuously iterating the training data to minimize the loss function.

8. The pedestrian trajectory prediction method according to claim 7, characterized in that: The loss function is in, Represents the predicted coordinates of pedestrian n at time t, which follows a bivariate Gaussian distribution. represents the average value, represents the standard deviation, represents the correlation coefficient, W represents all parameters of model training, T obs represents the observation length, T pred Indicates the predicted length.

9. A pedestrian trajectory prediction device based on pedestrian scene interaction modeling, characterized in that: include: The pedestrian interaction feature extraction unit is used to extract the trajectory feature space of each pedestrian using multiple feature maps, obtain feature maps of relative position, velocity and acceleration, and perform graph convolution on each feature map to obtain pedestrian interaction features through superposition; The scene trajectory feature extraction unit is used to encode the scene semantic segmentation map using SwinTransformer, map the pedestrian coordinate sequence into a position map, and align and fuse it with the semantic features to obtain the spatiotemporal scene features; A fusion unit, configured to cross-modally fuse the pedestrian interaction features with the spatiotemporal scene features through a bidirectional self-attention mechanism to obtain scene trajectory features; The prediction unit is used to input the fused scene trajectory features into the time extrapolation convolutional network to predict the bivariate Gaussian distribution parameters of the future trajectory, and perform end-to-end training with negative log-likelihood loss until convergence. Finally, the converged model is used to predict the pedestrian trajectory.

10. A computer-readable storage medium, comprising a computer program stored therein, characterized in that: When the computer program is executed by a processor, the steps of a pedestrian trajectory prediction method based on pedestrian scene interaction modeling as described in any one of claims 1 to 8 are implemented.

Images