A pedestrian trajectory prediction method based on multi-cue transformation network

Abstract

This invention relates to the field of computer vision technology, specifically disclosing a pedestrian trajectory prediction method based on a multi-cue transformation network. The method includes: constructing a trajectory feature extraction module to obtain a high-dimensional representation of the pedestrian's observed trajectory; randomly generating learnable target point tokens using a normal distribution to represent pedestrian target point information; constructing a hierarchical cross-attention module to obtain information about the pedestrian's temporal and spatial dependencies; constructing a loss calculation module to obtain an optimal pedestrian trajectory prediction model using asymptotic target guidance loss and mean squared error loss; and using the optimal pedestrian trajectory prediction model to obtain the pedestrian's future trajectory. This invention combines cross-attention mechanisms, self-attention mechanisms, and the guiding role of target point cues, while simultaneously modeling the dependencies between pedestrian trajectory information and social, environmental, and target point cues, thereby achieving target-environment-social interaction and further improving the accuracy of pedestrian trajectory prediction.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for predicting pedestrian trajectories based on a multi-cue transformation network. Background Technology

[0002] Pedestrian trajectory prediction, a key technology in intelligent systems, plays a crucial role in various fields such as autonomous driving, intelligent transportation, video surveillance, and robot navigation, attracting widespread attention from both academia and industry. The core task of this technology is to predict the trajectory of a pedestrian over the next 4.8 seconds (12 frames), based on the observed trajectory of the pedestrian over the previous 3.2 seconds (8 frames). However, the future trajectory of a pedestrian is not only constrained by common-sense rules but also influenced by physical obstacles. Common-sense rules mainly include social behavioral norms, while physical obstacles exist in the environment, such as trees, buildings, and moving objects. Therefore, pedestrian trajectory prediction faces significant challenges.

[0003] Understanding social cues between pedestrians is crucial for predicting their future trajectories. Early research primarily used hand-designed functions to model social interactions. Helbing et al. proposed the Social Force model, treating attraction and repulsion between pedestrians as social cues to simulate social interactions. However, using hand-designed functions to model social cues between pedestrians has limited predictive power when dealing with complex nonlinear motions, making it difficult to apply to dynamic and complex scenarios.

[0004] Due to the powerful representational capabilities of deep learning models, their application in pedestrian trajectory prediction has significantly improved prediction accuracy in complex scenarios. Some methods employ pooling operations to understand social cues between pedestrians. For example, Alahi et al. innovatively proposed a social pooling strategy using Long Short-Term Memory (LSTM) networks to effectively model interaction information between pedestrians. Zhang et al. took a different approach, effectively combining element-wise motion gating and a pedestrian-based attention mechanism to mine social cues between pedestrians. Other methods use attention mechanisms and Graph Convolutional Neural Networks (GCNs) to mine social cues between pedestrians to predict future trajectories. For example, Xie et al. proposed Dynamic Attention Generative Adversarial Network (DTGAN), which captures implicit social interaction patterns from graph sequence data through a random weight mechanism to achieve refined modeling of trajectory prediction; Mohamed et al. modeled pedestrians as graph nodes based on graph convolutional neural networks, using graph structures to represent spatial relationships between pedestrians and learning dynamic social interaction information through graph topology; Wong et al. designed an angle-based attention mechanism that enables target pedestrians to perceive the relative position and direction of movement of neighboring pedestrians and dynamically adjust their decisions, thereby more accurately modeling group interaction behavior.

[0005] Secondly, acquiring environmental cues between pedestrians and their surroundings is equally crucial for improving trajectory prediction performance. Some studies capture environmental cues by modeling overall environmental information. For example, Sadeghian et al. used RGB images of the surrounding environment as input and employed convolutional neural networks (CNNs) to extract environmental cues containing semantic features such as road structure and area function. Other researchers focus on physical obstacle modeling; for instance, Liang et al. used a pre-trained environmental scene segmentation network to obtain pixel-level physical obstacle category information in the surrounding environment. These methods expand the contextual information of trajectory prediction from both environmental semantics and obstacle constraints.

[0006] Finally, human behavior is often guided by specific target points, so target point cues are also used to improve trajectory prediction performance. Rehder et al. proposed the Particle Filter method, which uses a probabilistic planning approach to model the distribution of pedestrian target points. Subsequently, Chiara et al. fused an attention-based recurrent network with an environment-aware target point guidance module for pedestrian trajectory prediction.

[0007] However, existing deep learning-based pedestrian trajectory prediction methods only model the interaction between trajectory information and a single cue, making it difficult to capture the complex relationships between multiple sources of cues. In addition, target-point-oriented methods assign equal weight to the coordinate differences between the predicted target point and the real target point at different times, but as time increases, the predicted target point of the pedestrian gradually approaches the real target point, and the existing "equal weight" setting method will affect the trajectory prediction performance of the model. Summary of the Invention

[0008] This invention provides a pedestrian trajectory prediction method based on a multi-clue transformation network. The purpose is to solve the problems in the prior art that the interaction between pedestrian trajectory information and multiple clues is not modeled simultaneously, and the setting of equal weight between the predicted target point and the actual target point of pedestrians at different times leads to unreasonable trajectory prediction.

[0009] To achieve the above objectives, this invention provides a pedestrian trajectory prediction method based on a multi-clue transformation network, comprising the following steps:

[0010] S1. Input the two-dimensional coordinates of the pedestrian observation trajectory into the trajectory feature extraction module to obtain its high-dimensional representation, and use learnable target point tokens. This indicates the pedestrian's target point information;

[0011] S2. Combine the high-dimensional representation with learnable target point tokens. By splicing the data, the fused features are obtained. ; to render the scene semantic graph The input is fed into the environmental feature extraction module to obtain environmental semantic features. ;

[0012] S3, Integrating Features With environmental semantic features The input is fed into the first-level cross-attention module to mine the target-environment-time interaction relationship of pedestrians and obtain the temporal interaction features of pedestrians. ;

[0013] S4. Based on the first-level cross-attention module, the representation... Temporal interaction characteristics of pedestrian goal-environment-time interaction relationship After combination, we get Spatiotemporal correlation characteristics of individual pedestrians Then, in conjunction with the semantic features of the pedestrian's surrounding environment. Together, they are input into the second-level cross-attention module to mine the target-environment-social interaction relationship of pedestrians, and to process the obtained multi-dimensional interaction features of pedestrians. Decode the data to obtain the pedestrian's predicted coordinates at the next moment and the target point coordinates at the last moment.

[0014] S5. Input the predicted coordinates and target point coordinates into the loss calculation module, and optimize the model using the progressive target guidance loss and mean square error loss to obtain the optimal pedestrian trajectory prediction model.

[0015] S6. Use the optimal pedestrian trajectory prediction model to predict the pedestrian trajectory in the scene to be predicted.

[0016] Preferably, S1 includes the following steps:

[0017] S11. A trajectory feature extraction module is constructed based on fully connected layers and activation functions. The two-dimensional coordinates of the pedestrian observation trajectory are input into the trajectory feature extraction module for dimensionality upscaling to obtain a high-dimensional representation of the pedestrian observation trajectory. The expression for the two-dimensional coordinates of the pedestrian observation trajectory is as follows:

[0018] ;

[0019] in, express Time of the first Two-dimensional coordinates of a pedestrian Indicates the first A pedestrian time The coordinate values ​​along the axis. Indicates the first A pedestrian time The coordinate values ​​along the axis. Indicates the current time of the pedestrian. and These represent the lengths of the pedestrian's observed trajectory and predicted trajectory, respectively. Indicates the first A pedestrian, This indicates the total number of pedestrians;

[0020] S12. Generate learnable target point tokens using a random normal distribution. To indicate the first Information on the destination points of individual pedestrians.

[0021] Preferably, S2 includes the following steps:

[0022] S21. Integrating the high-dimensional representation of pedestrian observation trajectories with learnable target point tokens. By piecing together the images, the integrated features of the pedestrians can be obtained. ;

[0023] S22. Construct an environmental feature extraction module based on the convolutional neural network VGG-19, wherein the convolutional neural network VGG-19 is used to extract features of the scene image;

[0024] S23. Render the scene semantic map. The input is fed into the environmental feature extraction module, which then mines the environmental information around the pedestrian to obtain the semantic features of the pedestrian's surrounding environment. .

[0025] Preferably, in S21, the pedestrian fusion features Represented as:

[0026] ;

[0027] in, For serial operation, Represents a mapping function. This represents the embedding matrix in the mapping function.

[0028] Preferably, S3 includes the following steps:

[0029] S31. A first-level cross-attention module is constructed based on the layer normalization module, cross-attention mechanism, self-attention mechanism, feedforward neural network module, residual structure and multilayer perceptron module. The first-level cross-attention module consists of two cross-information interaction layers and one global information interaction layer.

[0030] S32, Integrating pedestrian characteristics semantic features of the environment surrounding pedestrians The data is input into the first-level cross-attention module, which is then used to mine the pedestrian's target-environment-time interaction relationship to obtain the pedestrian's temporal interaction features. It is represented as:

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] in, and These represent cross-attention mechanism and self-attention mechanism, respectively. This represents the feedforward neural network module. This represents a multilayer perceptron module. Presentation layer standardization module, and These represent the outputs of the top and bottom cross-information interaction layers in the first-level cross-attention module, respectively. This represents the element-wise sum of the outputs from two cross-information interaction layers. This represents the intermediate output of the global information interaction layer in the first-level cross-attention module.

[0037] Preferably, S4 includes the following steps:

[0038] S41. Based on the constructed first-level cross-attention module, the representation... Temporal interaction characteristics of pedestrian goal-environment-time interaction relationship After combination, we get Spatiotemporal correlation characteristics of individual pedestrians Then semantic features of the environment surrounding pedestrians The data is input into the second-level cross-attention module, which is then used to mine the pedestrian's target-environment-social interaction relationships, resulting in multi-dimensional pedestrian interaction features. The construction of the second-level cross-attention module is the same as that of the first-level cross-attention module. Spatiotemporal correlation characteristics of individual pedestrians Represented as:

[0039] ;

[0040] in, For parallel operation, timing characteristics To characterize the first Characteristics of pedestrian goal-environment-time interaction relationships This indicates the total number of pedestrians;

[0041] S42. Multi-dimensional interaction features of pedestrians based on two independent fully connected layers. Perform decoding to obtain the pedestrian's predicted coordinates at the next moment and the target point coordinates at the last moment;

[0042] pedestrians in the next moment Predicted coordinates and final time The target point coordinates are represented as follows:

[0043] ;

[0044] ;

[0045] in, and They represent the first A pedestrian at a moment and Two-dimensional predicted coordinates and These represent two independent fully connected layers. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. This indicates a splicing operation. This represents random Gaussian noise.

[0046] Preferably, S5 includes the following steps:

[0047] S51. The trajectory feature extraction module, hierarchical cross-attention module, and loss calculation module are combined to form a pedestrian trajectory prediction model, and the loss is guided by a progressive target. and mean square error loss Construct a loss calculation module;

[0048] S52. Based on the predicted target point and the actual target point of the pedestrian at the last moment generated by the pedestrian trajectory prediction model, the loss is guided by a progressive target. The loss value between the predicted target point and the actual target point coordinates of the pedestrian at the last moment is calculated;

[0049] S53. Based on the pedestrian trajectory prediction model, the predicted coordinates of the pedestrian at the next moment and the actual coordinates of the pedestrian at the next moment are generated, and the mean square error loss is used. The loss value between the predicted coordinates and the actual coordinates of the pedestrian at the next moment is calculated.

[0050] S54. Calculate the total loss The pedestrian trajectory prediction model is optimized using the total loss to obtain the optimal pedestrian trajectory prediction model.

[0051] Preferably, in S52, the asymptotic target-guided loss... Represented as:

[0052] ;

[0053] in, Represents the time index, used to traverse from time... arrive At each moment, This represents the total length of the pedestrian's observed and predicted trajectories. and They represent the first Predicted target point and actual target point of each pedestrian; parameters It is used to measure the proximity between the predicted target point and the actual target point of a pedestrian at different times.

[0054] Preferably, in S53, the mean square error loss Represented as:

[0055] ;

[0056] in, and They represent the first A pedestrian The predicted coordinates and the actual coordinates at a given time.

[0057] Preferably, in S54, the total loss Represented as:

[0058] ;

[0059] in, Used to balance the loss in asymptotic target guidance Total loss The proportion of contribution.

[0060] This invention employs the aforementioned pedestrian trajectory prediction method based on a multi-threaded transform network, and its beneficial effects are as follows:

[0061] This invention mines the interaction between pedestrian trajectory information and multiple cues from the perspective of temporal and spatial dependencies through a hierarchical cross-attention module, and utilizes an asymptotic target-guided loss function to fully leverage the guiding role of target point cues on the future trajectory of pedestrians by assigning increasing weights to the predicted target point coordinates at different times. This achieves dynamic coupling modeling between multiple source cues, improves the rationality of predicted trajectories, and increases the accuracy of trajectory prediction. Attached Figure Description

[0062] Figure 1 This is a flowchart of a pedestrian trajectory prediction method based on a multi-clue transformation network according to an embodiment of the present invention;

[0063] Figure 2 This is a visualization result of a pedestrian trajectory prediction method based on a multi-threaded transformation network according to an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0065] like Figure 1 As shown in the figure, an embodiment of the present invention provides a pedestrian trajectory prediction method based on a multi-clue transformation network, which includes the following steps:

[0066] S1. Construct a trajectory feature extraction module. Input the two-dimensional coordinates of the pedestrian's observed trajectory into the trajectory feature extraction module to obtain a high-dimensional representation of the pedestrian's observed trajectory, and utilize learnable target point tokens. To represent the target point information of pedestrians, the specific steps are as follows:

[0067] S11. A trajectory feature extraction module is constructed based on fully connected layers and activation functions. Independent fully connected layers and activation functions are used to extract the two-dimensional coordinates of the pedestrian's observed trajectory. In this embodiment, a dimensionality-upgrading operation is performed, mapping to 32 dimensions to obtain a high-dimensional representation of the pedestrian observation trajectory; the expression for the two-dimensional coordinates of the pedestrian observation trajectory is:

[0068] ;

[0069] in, express Time of the first Two-dimensional coordinates of a pedestrian Indicates the first A pedestrian time The coordinate values ​​along the axis. Indicates the first A pedestrian time The coordinate values ​​along the axis. Indicates the current time of the pedestrian. and These represent the lengths of the pedestrian's observed trajectory and predicted trajectory, respectively. Indicates the first A pedestrian, This indicates the total number of pedestrians.

[0070] S12. Generate learnable target point tokens using a random normal distribution. To indicate the first In this embodiment, the target point token is the target point information of each pedestrian. The dimension is 32.

[0071] S2. Combine the high-dimensional representation of the pedestrian observation trajectory with learnable target point tokens. By splicing the images together, the fused features of the pedestrians can be obtained. Furthermore, an environmental feature extraction module was constructed to extract scene semantic graphs. The input is fed into the environmental feature extraction module, which then mines the environmental information around the pedestrian to obtain the semantic features of the pedestrian's surrounding environment. The specific steps are as follows:

[0072] S21. Integrating the high-dimensional representation of pedestrian observation trajectories with learnable target point tokens. By piecing together the images, the integrated features of the pedestrians can be obtained. pedestrian integration characteristics Represented as:

[0073] ;

[0074] in, For serial operation, Represents a mapping function. This represents the embedding matrix in the mapping function.

[0075] S22. Construct an environment feature extraction module based on the VGG-19 convolutional neural network. The VGG-19 convolutional neural network is a classic convolutional neural network architecture in deep learning, used to extract features from scene images.

[0076] S23. Render the scene semantic map. The input is fed into the environmental feature extraction module, which then mines the environmental information around the pedestrian to obtain the semantic features of the pedestrian's surrounding environment. The obtained pedestrian fusion features semantic features of the environment surrounding pedestrians The dimensions of each element are 32.

[0077] S3. Construct the first-level cross-attention module to integrate the pedestrian's features. semantic features of the environment surrounding pedestrians The data is input into the first-level cross-attention module, which is then used to mine the pedestrian's target-environment-time interaction relationship to obtain the pedestrian's temporal interaction features. The specific steps are as follows:

[0078] S31. A first-level cross-attention module is constructed based on the layer normalization module, cross-attention mechanism, self-attention mechanism, feedforward neural network module, residual structure and multilayer perceptron module. The first-level cross-attention module consists of two cross-information interaction layers and one global information interaction layer.

[0079] In the first-level cross-attention module, the input is the pedestrian fusion feature. With environmental semantic features At that time, the top cross-information interaction layer is As the dominant feature, a cross-attention mechanism is used to mine feature interaction relationships, which are then processed by a feedforward neural network module and connected via residuals; the bottom cross-information interaction layer uses... The same operation is performed on the dominant features. The outputs of the two cross-information interaction layers are fused into the global information interaction layer. The global information interaction layer learns relevant pedestrian information through a self-attention mechanism and a multilayer perceptron module to obtain the temporal interaction features of pedestrians. At the same time, the layer standardization module is used to accelerate the model convergence speed.

[0080] In another embodiment, the top cross-information interaction layer and the bottom cross-information interaction layer can be connected in parallel, and then connected serially to the global information interaction layer.

[0081] S32, Integrating pedestrian characteristics semantic features of the environment surrounding pedestrians The data is input into the first-level cross-attention module, which is then used to mine the pedestrian's target-environment-time interaction relationship to obtain the pedestrian's temporal interaction features. It is represented as:

[0082] ;

[0083] ;

[0084] ;

[0085] ;

[0086] ;

[0087] in, and These represent cross-attention mechanism and self-attention mechanism, respectively. This represents the feedforward neural network module. This represents a multilayer perceptron module. Presentation layer standardization module, and These represent the outputs of the top and bottom cross-information interaction layers in the first-level cross-attention module, respectively. This represents the element-wise sum of the outputs from two cross-information interaction layers. This represents the intermediate output of the global information interaction layer in the first-level cross-attention module. The layer normalization module, cross-attention mechanism, self-attention mechanism, feedforward neural network module, residual structure, and multilayer perceptron module are commonly used computational modules in this field. Furthermore, in this embodiment, both the cross-attention mechanism and the self-attention mechanism use multi-head attention, and the number of multi-heads is set to... .

[0088] S4. To achieve learning of social interactions among pedestrians, the representation will then be... Temporal interaction characteristics of pedestrian goal-environment-time interaction relationship Combining them, we get Spatiotemporal correlation characteristics of individual pedestrians ,Will semantic features of the environment surrounding pedestrians The inputs are fed into the second-level cross-attention module, which has the same structure as the first-level cross-attention module. The second-level cross-attention module is used to mine the target-environment-social interaction relationships of pedestrians, obtaining multi-dimensional pedestrian features. It utilizes two independent fully connected layers to analyze multi-dimensional pedestrian features. Perform decoding to obtain the pedestrian's predicted coordinates at the next moment and the target point coordinates at the last moment; the specific steps are as follows:

[0089] S41. Based on the constructed first-level cross-attention module, the representation... Temporal interaction characteristics of pedestrian goal-environment-time interaction relationship After combination, we get Spatiotemporal correlation characteristics of individual pedestrians Then semantic features of the environment surrounding pedestrians The input is fed into the second-level cross-attention module, which is then used to mine the pedestrian's target-environment-social interaction relationships, resulting in multi-dimensional pedestrian interaction features. , Spatiotemporal correlation characteristics of individual pedestrians Represented as:

[0090] ;

[0091] in, For parallel operation, timing characteristics To characterize the first Characteristics of pedestrian goal-environment-time interaction relationships This indicates the total number of pedestrians.

[0092] In the second-level cross-attention module, the input is Spatiotemporal correlation characteristics of individual pedestrians semantic features of the environment surrounding pedestrians At that time, the top cross-information interaction layer is As the dominant feature, a cross-attention mechanism is used to mine feature interaction relationships, which are then processed by a feedforward neural network module and connected via residuals; the bottom cross-information interaction layer uses... The same operation is performed on the dominant features. The outputs of the two cross-information interaction layers are fused into the global information interaction layer. The global information interaction layer learns relevant pedestrian information through a self-attention mechanism and a multilayer perceptron module to obtain multi-dimensional pedestrian interaction features. At the same time, the layer standardization module is used to accelerate the model convergence speed.

[0093] S42. Multi-dimensional interaction features of pedestrians based on two independent fully connected layers. The decoding operation is performed to obtain the pedestrian's predicted coordinates at the next moment and the target point coordinates at the last moment.

[0094] pedestrians in the next moment Predicted coordinates and final time The target point coordinates are represented as follows:

[0095] ;

[0096] ;

[0097] in, and They represent the first A pedestrian at a moment and Two-dimensional predicted coordinates and These represent two independent fully connected layers. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. This indicates a splicing operation. This represents random Gaussian noise, in this embodiment The dimension size is 16.

[0098] S5. Construct a loss calculation module. Input the predicted coordinates of the pedestrian at the next moment and the target point coordinates at the last moment into the loss calculation module. Optimize the pedestrian trajectory prediction model composed of the trajectory feature extraction module, hierarchical cross-attention module, and loss calculation module using asymptotic target guidance loss and mean squared error loss to obtain the optimal pedestrian trajectory prediction model. The specific steps are as follows:

[0099] S51. The trajectory feature extraction module, hierarchical cross-attention module, and loss calculation module are combined to form a pedestrian trajectory prediction model, and the loss is guided by a progressive target. and mean square error loss Build a loss calculation module.

[0100] S52. Based on the predicted target point and the actual target point of the pedestrian at the last moment generated by the pedestrian trajectory prediction model, the loss is guided by a progressive target. The loss value between the predicted target point and the actual target point coordinates at the last moment of the pedestrian's journey is calculated.

[0101] Progressive goals guide loss Represented as:

[0102] ;

[0103] in, Represents the time index, used to traverse from time... arrive At each moment, This represents the total length of the pedestrian's observed trajectory and predicted trajectory. and They represent the first Predicted target point and actual target point of each pedestrian, parameters It is used to measure the proximity between the predicted target point and the actual target point of a pedestrian at different times.

[0104] S53. Based on the pedestrian trajectory prediction model, the predicted coordinates of the pedestrian at the next moment and the actual coordinates of the pedestrian at the next moment are generated, and the mean square error loss is used. The loss value between the predicted coordinates and the actual coordinates of the pedestrian at the next moment is calculated.

[0105] Mean square error loss Represented as:

[0106] ;

[0107] in, and They represent the first A pedestrian The predicted coordinates and the actual coordinates at a given time.

[0108] S54. Calculate the total loss The pedestrian trajectory prediction model is optimized using the total loss to obtain the optimal pedestrian trajectory prediction model.

[0109] Total loss Represented as:

[0110] ;

[0111] in, Used to balance the loss in asymptotic target guidance Total loss The contribution ratio in this embodiment .

[0112] S6. During the testing phase, the optimal pedestrian trajectory prediction model is used to predict the pedestrian trajectory in the scenario to be predicted, thereby obtaining the future trajectory of the pedestrian at the next moment. The specific steps are as follows:

[0113] Using publicly available online pedestrian trajectory prediction databases as test subjects, such as the SDD database, the pedestrian trajectory prediction method of this invention achieved an average ADE / FDE ratio of 6.52 / 10.06 (in pixels); when tested on the inD database, the method achieved an average ADE / FDE ratio of 0.28 / 0.51 (in meters); and when tested on the ETH / UCY database, the method achieved an average ADE / FDE ratio of 0.14 / 0.19 (in meters).

[0114] like Figure 2 Several visualizations are presented to facilitate intuitive verification. Blue represents the observed trajectory, while red and green represent the pedestrian's actual and predicted future trajectories, respectively. Figure 2 Figures (a) to (d) show several examples of pedestrians walking in a straight line on the ETH / UCY dataset, demonstrating that the predicted trajectory of the method proposed in this invention is almost identical to the actual trajectory. Figure 2 Figures (e) through (f) illustrate several examples of pedestrian turning on the SDD and inD datasets. These visualizations further confirm that the present invention can accurately predict future pedestrian trajectories.

[0115] Therefore, this invention mines the interaction relationship between pedestrian trajectory information and multiple cues from the perspective of time and space dependence through a hierarchical cross-attention module, and uses an asymptotic target-guided loss function to fully leverage the guiding role of target point cues on the future trajectory of pedestrians by assigning increasing weights to the predicted target point coordinates at different times. This achieves fusion and interaction modeling among multiple source cues, improves the rationality of predicted trajectories, and increases the accuracy of trajectory prediction. Thus, the effectiveness of the method of this invention is evident.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A pedestrian trajectory prediction method based on a multi-threaded transform network, characterized in that, Includes the following steps: S1. Input the two-dimensional coordinates of the pedestrian observation trajectory into the trajectory feature extraction module to obtain its high-dimensional representation, and use learnable target point tokens. This indicates the pedestrian's target point information; S2. Combine the high-dimensional representation with learnable target point tokens. By splicing the data, the fused features are obtained. ; to render the scene semantic graph The input is fed into the environmental feature extraction module to obtain environmental semantic features. ; S3, Integrating Features With environmental semantic features The input is fed into the first-level cross-attention module to mine the target-environment-time interaction relationship of pedestrians and obtain the temporal interaction features of pedestrians. ; S4. Based on the first-level cross-attention module, the representation... Temporal interaction characteristics of pedestrian goal-environment-time interaction relationship After combination, we get Spatiotemporal correlation characteristics of individual pedestrians Then, in conjunction with the semantic features of the pedestrian's surrounding environment. Together, they are input into the second-level cross-attention module to mine the target-environment-social interaction relationship of pedestrians, and to process the obtained multi-dimensional interaction features of pedestrians. Decode the data to obtain the pedestrian's predicted coordinates at the next moment and the target point coordinates at the last moment. S5. Input the predicted coordinates and target point coordinates into the loss calculation module, and optimize the model using the progressive target guidance loss and mean square error loss to obtain the optimal pedestrian trajectory prediction model. S6. Use the optimal pedestrian trajectory prediction model to predict the pedestrian trajectory in the scene to be predicted; S3 includes the following steps: S31. A first-level cross-attention module is constructed based on the layer normalization module, cross-attention mechanism, self-attention mechanism, feedforward neural network module, residual structure and multilayer perceptron module. The first-level cross-attention module consists of two cross-information interaction layers and one global information interaction layer. S32, Integrating pedestrian characteristics semantic features of the environment surrounding pedestrians The data is input into the first-level cross-attention module, which is then used to mine the pedestrian's target-environment-time interaction relationship to obtain the pedestrian's temporal interaction features. , Represented as: ； ； ； ； ； in, and These represent cross-attention mechanism and self-attention mechanism, respectively. This represents the feedforward neural network module. This represents a multilayer perceptron module. Presentation layer standardization module, and These represent the outputs of the top and bottom cross-information interaction layers in the first-level cross-attention module, respectively. This represents the element-wise sum of the outputs from two cross-information interaction layers. This represents the intermediate output of the global information interaction layer in the first-level cross-attention module; S4 includes the following steps: S41. Based on the constructed first-level cross-attention module, the representation... Temporal interaction characteristics of pedestrian goal-environment-time interaction relationship After combination, we get Spatiotemporal correlation characteristics of individual pedestrians Then semantic features of the environment surrounding pedestrians The data is input into the second-level cross-attention module, which is then used to mine the pedestrian's target-environment-social interaction relationships, resulting in multi-dimensional pedestrian interaction features. The construction of the second-level cross-attention module is the same as that of the first-level cross-attention module. Spatiotemporal correlation characteristics of individual pedestrians Represented as: ； in, For parallel operation, timing characteristics To characterize the first Characteristics of pedestrian goal-environment-time interaction relationships Indicates the total number of pedestrians; S42. Multi-dimensional interaction features of pedestrians based on two independent fully connected layers. Perform decoding to obtain the pedestrian's predicted coordinates at the next moment and the target point coordinates at the last moment; pedestrians in the next moment Predicted coordinates and final time The target point coordinates are represented as follows: ； ； in, and They represent the first A pedestrian at a moment and Two-dimensional predicted coordinates and These represent two independent fully connected layers. Indicates the first A pedestrian Predicted coordinate values ​​in the axial direction. Indicates the first A pedestrian Predicted coordinate values ​​in the axial direction. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. Indicates the first A pedestrian time Predicted coordinate values ​​in the axial direction. This indicates a splicing operation. This represents random Gaussian noise.

2. The pedestrian trajectory prediction method based on a multi-cue transformation network according to claim 1, characterized in that, S1 includes the following steps: S11. A trajectory feature extraction module is constructed based on fully connected layers and activation functions. The two-dimensional coordinates of the pedestrian observation trajectory are input into the trajectory feature extraction module for dimensionality upscaling to obtain a high-dimensional representation of the pedestrian observation trajectory. The expression for the two-dimensional coordinates of the pedestrian observation trajectory is as follows: ； in, express Time of the first Two-dimensional coordinates of a pedestrian Indicates the first A pedestrian time The coordinate values ​​along the axis. Indicates the first A pedestrian time The coordinate values ​​along the axis. Indicates the current time of the pedestrian. and These represent the lengths of the pedestrian's observed trajectory and predicted trajectory, respectively. Indicates the first A pedestrian, Indicates the total number of pedestrians; S12. Generate learnable target point tokens using a random normal distribution. To indicate the first Information on the destination points of individual pedestrians.

3. The pedestrian trajectory prediction method based on a multi-clue transformation network according to claim 2, characterized in that, S2 includes the following steps: S21. Integrating the high-dimensional representation of pedestrian observation trajectories with learnable target point tokens. By piecing together the images, the integrated features of the pedestrians can be obtained. ; S22. Construct an environmental feature extraction module based on the convolutional neural network VGG-19, wherein the convolutional neural network VGG-19 is used to extract features of the scene image; S23. Render the scene semantic map. The input is fed into the environmental feature extraction module, which then mines the environmental information around the pedestrian to obtain the semantic features of the pedestrian's surrounding environment. .

4. The pedestrian trajectory prediction method based on a multi-clue transformation network according to claim 3, characterized in that, In S21, the pedestrian fusion features Represented as: ； in, For serial operation, Represents a mapping function. This represents the embedding matrix in the mapping function.

5. The pedestrian trajectory prediction method based on a multi-clue transformation network according to claim 4, characterized in that, S5 includes the following steps: S51. The trajectory feature extraction module, hierarchical cross-attention module, and loss calculation module are combined to form a pedestrian trajectory prediction model, and the loss is guided by a progressive target. and mean square error loss Construct a loss calculation module; S52. Based on the predicted target point and the actual target point of the pedestrian at the last moment generated by the pedestrian trajectory prediction model, the loss is guided by a progressive target. The loss value between the predicted target point and the actual target point coordinates of the pedestrian at the last moment is calculated; S53. Based on the pedestrian trajectory prediction model, the predicted coordinates of the pedestrian at the next moment and the actual coordinates of the pedestrian at the next moment are generated, and the mean square error loss is used. The loss value between the predicted coordinates and the actual coordinates of the pedestrian at the next moment is calculated. S54. Calculate the total loss The pedestrian trajectory prediction model is optimized using the total loss to obtain the optimal pedestrian trajectory prediction model.

6. The pedestrian trajectory prediction method based on a multi-cue transformation network according to claim 5, characterized in that, In S52, progressive target-guided loss Represented as: ； in, Represents the time index, used to iterate from time... arrive At each moment, This represents the total length of the pedestrian's observed and predicted trajectories. and They represent the first Predicted target point and actual target point of each pedestrian; parameters It is used to measure the proximity between the predicted target point and the actual target point of a pedestrian at different times.

7. The pedestrian trajectory prediction method based on a multi-cue transformation network according to claim 6, characterized in that, In S53, the mean square error loss Represented as: ； in, and They represent the first A pedestrian The predicted coordinates and the actual coordinates at time t.

8. The pedestrian trajectory prediction method based on a multi-clue transformation network according to claim 6, characterized in that, In S54, the total loss Represented as: ； in, Used to balance the loss in asymptotic target guidance Total loss The proportion of contribution.

Images