Pedestrian trajectory prediction method, device, computer equipment, and storage medium

Abstract

The present application relates to a pedestrian trajectory prediction method, apparatus, computer equipment, storage medium, and computer program product. The method comprises: obtaining multiple pedestrian trajectory sequence data of multiple time steps, and extracting the first hidden state features of the multiple time steps in the multiple pedestrian trajectory sequence data; extracting local pedestrian interaction features and global pedestrian interaction features from the first hidden state features of the multiple time steps, respectively, to obtain local pedestrian interaction features and global pedestrian interaction features of the multiple time steps; fusing the local interaction pedestrian features and global pedestrian interaction features of the multiple time steps to obtain target pedestrian interaction features of the multiple time steps; and predicting the multiple pedestrian trajectories of the next time step based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps. In this way, the predicted multiple pedestrian trajectories of the next time step can be more accurate.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a pedestrian trajectory prediction model training method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] With the development of computer vision technology, pedestrian trajectory prediction technology has emerged. Pedestrian trajectory prediction aims to predict the movement path of pedestrians over a period of time. Pedestrian trajectory prediction is of great significance in fields such as urban traffic management and autonomous mobile platforms.

[0003] Early pedestrian trajectory prediction methods often relied on handcrafted features and traditional machine learning algorithms such as random forests, hidden Markov models, and support vector machines. However, these traditional techniques have limited effectiveness and are only suitable for simple scenarios and short-term prediction tasks. This is because handcrafted features may not capture complex scene information and social relationships between pedestrians. In contrast, deep learning-based methods automatically learn feature representations from trajectories, thereby better capturing scene information and pedestrian relationships, and improving prediction accuracy.

[0004] However, the current pedestrian trajectory prediction method based on deep learning has inaccurate feature representation of pedestrian trajectories due to the complexity and uncertainty of the pedestrian environment, resulting in low accuracy of pedestrian trajectory prediction. Summary of the Invention

[0005] Based on this, it is necessary to provide a pedestrian trajectory prediction method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of pedestrian trajectory prediction in order to address the above technical problems.

[0006] In a first aspect, the present application provides a method for predicting pedestrian trajectories. The method comprises:

[0007] Acquire multiple pedestrian trajectory sequence data at multiple time steps, and extract first hidden state features of the multiple time steps in the multiple pedestrian trajectory sequence data;

[0008] Extracting local pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps;

[0009] Performing global pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps;

[0010] Fusing the local pedestrian interaction features of the multiple time steps with the global pedestrian interaction features to obtain target pedestrian interaction features of the multiple time steps;

[0011] Predicting multiple pedestrian trajectories at a next time step based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps.

[0012] In one embodiment, the multiple pedestrian trajectory sequence data is extracted from pedestrian images at multiple time steps; and the extracting of local pedestrian interaction features from the first hidden state features of the multiple time steps to obtain the local pedestrian interaction features of the multiple time steps includes:

[0013] The first hidden state feature is sliced ​​to obtain hidden state sub-features corresponding to each pedestrian target in the pedestrian image; the pedestrian position of each pedestrian target is located in the pedestrian image, and the pedestrian image is divided into regions according to each pedestrian position to obtain position information of each region; the hidden state sub-features of each pedestrian target in the image region corresponding to each regional position information are respectively summarized to obtain hidden features of each region; pedestrian interaction features are extracted from the hidden features of each region to obtain pedestrian interaction features of each region; the pedestrian interaction features of each region are fused to obtain local pedestrian interaction features.

[0014] In one embodiment, extracting global pedestrian interaction features from the first hidden state features of the multiple time steps to obtain global pedestrian interaction features of the multiple time steps includes:

[0015] Normalizing the first hidden state features of the multiple time steps respectively to obtain the first normalized features of the multiple time steps; performing pedestrian interaction feature extraction on the first normalized features of the multiple time steps respectively to obtain the first pedestrian interaction features of the multiple time steps; performing nonlinear processing on the first pedestrian interaction features of the multiple time steps respectively to obtain the second pedestrian interaction features of the multiple time steps; performing dimensionality reduction processing on the second pedestrian interaction features of the multiple time steps respectively to obtain the global pedestrian interaction features of the multiple time steps.

[0016] In one embodiment, the multiple pedestrian trajectory sequence data are extracted from pedestrian images at multiple time steps; and the first normalized features of the multiple time steps are respectively subjected to pedestrian interaction feature extraction to obtain the first pedestrian interaction features of the multiple time steps, including:

[0017] The first normalized features of each time step are fully connected respectively to obtain the first fully connected features of multiple time steps; attention weight information of multiple time steps is generated according to the relative position information between the pedestrian targets in the pedestrian images of the multiple time steps and the relative moving direction information between the pedestrian targets; the attention weight information of the multiple time steps is masked to obtain mask weight information of the multiple time steps; the mask weight information of the multiple time steps is normalized to obtain normalized weight information of the multiple time steps; according to the normalized weight information of the multiple time steps, the first fully connected features of the multiple time steps are weighted respectively to obtain the first pedestrian interaction features of the multiple time steps.

[0018] In one embodiment, predicting the trajectories of multiple pedestrians at the next time step based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps includes:

[0019] The first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps are fused to obtain the first fused features of the multiple time steps; the second hidden state features of the multiple time steps are extracted from the first fused features of the multiple time steps; the second hidden state features of each of the multiple time steps are fused with the preset noise features to obtain the second fused features of the multiple time steps; the third hidden state features of the multiple time steps are extracted from the second fused features of the multiple time steps; and the trajectories of multiple pedestrians in the next time step are predicted based on the third hidden state features of the multiple time steps.

[0020] In one embodiment, before acquiring multiple pedestrian trajectory sequence data at multiple time steps and extracting first hidden state features of the multiple time steps in the multiple pedestrian trajectory sequence data, the method further includes:

[0021] Acquire training data of a multi-pedestrian trajectory sequence of multiple time steps and a multi-pedestrian trajectory prediction model to be trained, wherein the multi-pedestrian trajectory prediction model includes a first recurrent neural network, a second recurrent neural network, a first feature extraction module and a second feature extraction module; based on the first recurrent neural network, perform hidden state feature extraction on the training data to obtain first training hidden state features of multiple time steps; based on the first feature extraction module, perform local pedestrian interaction feature extraction on the first training hidden state features of the multiple time steps to obtain training local pedestrian interaction features of multiple time steps; based on the second feature extraction module, perform local pedestrian interaction feature extraction on the first training hidden state features of the multiple time steps to obtain training local pedestrian interaction features of the multiple time steps Hidden state features are trained to extract global pedestrian interaction features to obtain trained global pedestrian interaction features for multiple time steps; the trained local interaction pedestrian features for the multiple time steps are fused with the trained global pedestrian interaction features to obtain trained pedestrian interaction features for the multiple time steps; based on the second recurrent neural network and the trained pedestrian interaction features, pedestrian trajectory information for the next time step is predicted; according to the model loss determined based on the pedestrian trajectory information for the next time step, the multi-pedestrian trajectory prediction model is iteratively optimized; wherein, the multi-pedestrian trajectory prediction model is used to predict the multi-pedestrian trajectories for the next time step based on the multi-pedestrian trajectory sequence data for the multiple time steps.

[0022] In a second aspect, the present application also provides a pedestrian trajectory prediction device. The device comprises:

[0023] A hidden state feature extraction module is used to obtain multiple pedestrian trajectory sequence data at multiple time steps and extract the first hidden state features of the multiple time steps in the multiple pedestrian trajectory sequence data;

[0024] A local feature extraction module is used to extract local pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps;

[0025] A global feature extraction module is used to extract global pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps;

[0026] A feature fusion module is used to fuse the local interactive pedestrian features of the multiple time steps with the global pedestrian interaction features to obtain target pedestrian interaction features of the multiple time steps;

[0027] The pedestrian trajectory prediction module is used to predict the trajectories of multiple pedestrians in the next time step based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps.

[0028] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0029] Acquire multi-pedestrian trajectory sequence data of multiple time steps, and extract the first hidden state features of the multiple time steps in the multi-pedestrian trajectory sequence data; perform local pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps; perform global pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps; fuse the local interaction pedestrian features of the multiple time steps with the global pedestrian interaction features to obtain target pedestrian interaction features of the multiple time steps; predict the multi-pedestrian trajectory of the next time step based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0031] Acquire multi-pedestrian trajectory sequence data of multiple time steps, and extract the first hidden state features of the multiple time steps in the multi-pedestrian trajectory sequence data; perform local pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps; perform global pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps; fuse the local interaction pedestrian features of the multiple time steps with the global pedestrian interaction features to obtain target pedestrian interaction features of the multiple time steps; predict the multi-pedestrian trajectory of the next time step based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps.

[0032] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0033] Acquire multi-pedestrian trajectory sequence data of multiple time steps, and extract the first hidden state features of the multiple time steps in the multi-pedestrian trajectory sequence data; perform local pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps; perform global pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps; fuse the local interaction pedestrian features of the multiple time steps with the global pedestrian interaction features to obtain target pedestrian interaction features of the multiple time steps; predict the multi-pedestrian trajectory of the next time step based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps.

[0034] The above-mentioned pedestrian trajectory prediction method, device, computer equipment, storage medium and computer program product obtain multiple pedestrian trajectory sequence data of multiple time steps and extract the first hidden state features of the multiple time steps in the multiple pedestrian trajectory sequence data, wherein the first hidden state features of each time step are used to predict the pedestrian trajectory of the next time step; local pedestrian interaction features are extracted from the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps; global pedestrian interaction features are extracted from the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps; the local interaction pedestrian features of the multiple time steps and the global pedestrian interaction features are fused to obtain target pedestrian interaction features of the multiple time steps; and the multiple pedestrian trajectories of the next time step are predicted based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps. In the process of pedestrian trajectory prediction, the present application not only extracts global features between pedestrians, but also considers local pedestrian interaction features of interactions between pedestrians in local areas, and fuses the local pedestrian interaction features and the global pedestrian interaction features to obtain target pedestrian interaction features. Since the target pedestrian interaction features fuse the local pedestrian interaction features and the global pedestrian interaction features, the target pedestrian interaction features obtained in this way have higher accuracy. Therefore, based on the first hidden state features and the target pedestrian interaction features, the trajectories of the multiple pedestrians are predicted, and the predicted multiple pedestrian trajectories have higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a flow chart of a pedestrian trajectory prediction method in one embodiment;

[0036] Figure 2 Schematic diagram of the process of extracting local pedestrian interaction features in one embodiment;

[0037] Figure 31 is a flow chart of the first pedestrian interaction feature extraction step in one embodiment;

[0038] Figure 4 A schematic diagram of a process for training a multi-pedestrian trajectory prediction model in one embodiment;

[0039] Figure 5 is a structural block diagram of a pedestrian trajectory prediction device in one embodiment;

[0040] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0042] In one embodiment, Figure 1 As shown, a pedestrian trajectory prediction method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0043] Step 202 : Acquire multiple pedestrian trajectory sequence data at multiple time steps, and extract first hidden state features at multiple time steps in the multiple pedestrian trajectory sequence data.

[0044] Among them, this embodiment can be used in pedestrian prediction systems in the field of unmanned driving. Multiple pedestrian trajectory sequence data of multiple time steps can be extracted from pedestrian images of multiple time steps. For example, in the field of unmanned vehicles, the vehicle's visual system can collect pedestrian images in the vehicle's driving environment. The step length of each time step can be 1 second or 0.5 seconds. The time step length can be set according to the required pedestrian trajectory prediction accuracy. The smaller the time step length, the more accurate the predicted pedestrian trajectory; the hidden state feature is used to represent the representation of pedestrian trajectory prediction information observed before the next time step, and the hidden state feature is used to predict the pedestrian trajectory at the next time step.

[0045] As an example, the extraction of hidden state features can be accomplished through a recurrent neural network.

[0046] As an example, step 202 includes: extracting multiple pedestrian trajectory sequence data of multiple time steps from pedestrian images of multiple time steps, performing hidden state feature extraction on the multiple pedestrian trajectory sequence data of multiple time steps based on an LSTM (Long Short Term Memory) neural network, and obtaining first hidden state features of the multiple time steps.

[0047] Step 204 : extract local pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps.

[0048] Among them, in the actual environment, the trajectory of the pedestrian in the next time step may be affected by the surrounding environment of the pedestrian at the current time. The closer the distance to the pedestrian target in trajectory prediction, the greater the impact on the next trajectory of the pedestrian target. The local pedestrian interaction feature is used to characterize the interaction influence information between pedestrians in the local area. Considering the local pedestrian interaction feature when predicting pedestrian trajectory can improve the accuracy of pedestrian trajectory prediction.

[0049] As an example, local pedestrian interaction features can be obtained by locally pooling the first hidden state features.

[0050] As an example, step 204 includes: dividing each pedestrian image into local regions according to the pedestrian position information in the pedestrian image at each time step to obtain the position information of each region; based on the position information of each region, extracting local pedestrian interaction features from the first hidden state features of multiple time steps to obtain local pedestrian interaction features of multiple time steps.

[0051] Step 206 : extracting global pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps.

[0052] Among them, the global pedestrian interaction feature is used to characterize the interaction influence information between pedestrians in the global area of ​​the pedestrian image.

[0053] As an example, global pedestrian interaction features can be extracted through GAT (Graph Neural Networks).

[0054] As an example, step 206 includes: normalizing the first hidden state features of multiple time steps respectively to obtain normalized hidden features of multiple time steps; extracting global pedestrian interaction features from the normalized hidden features of multiple time steps respectively to obtain global pedestrian interaction features of multiple time steps.

[0055] In step 208 , the local pedestrian interaction features of multiple time steps and the global pedestrian interaction features are fused to obtain target pedestrian interaction features of multiple time steps.

[0056] Among them, since the target pedestrian interaction feature integrates the local interactive pedestrian feature and the global pedestrian interaction feature, the target pedestrian interaction feature can be used to more accurately represent the interactive influence information between pedestrians.

[0057] As an example, step 208 includes: performing min-max normalization on the local interactive pedestrian features and the global pedestrian interaction features of multiple time steps respectively to obtain normalized local interactive pedestrian features of multiple time steps and normalized global pedestrian interaction features; and proportionally fusing the normalized local interactive pedestrian features of multiple time steps and the normalized global pedestrian interaction features to obtain target pedestrian interaction features of multiple time steps.

[0058] As an example, the feature fusion method in step 208 may be as follows:

[0059] C=λ*A+(1-λ)*B

[0060] Among them, A is the normalized local interactive pedestrian feature, B is the normalized global interactive pedestrian feature, C is the target interactive pedestrian feature, and λ∈(0,1) is the fusion coefficient.

[0061] Step 210 : predicting multiple pedestrian trajectories at the next time step based on the first hidden state features of multiple time steps and the target pedestrian interaction features of multiple time steps.

[0062] As an example, step 210 includes: concatenating and aggregating the first hidden state features of multiple time steps and the target pedestrian interaction features of multiple time steps to obtain the first fusion features of multiple time steps; concatenating and aggregating the noise features and the first fusion features of multiple time steps respectively to obtain the second fusion features; performing full connection processing on the target hidden state features obtained by performing hidden state feature extraction on the second fusion features to obtain the multi-pedestrian trajectories in the next time step.

[0063] In the above-mentioned pedestrian trajectory prediction method, while extracting the global features between pedestrians, the local pedestrian interaction features of the interactions between pedestrians in the local area are also considered, and the local pedestrian interaction features and the global pedestrian interaction features are fused to obtain the target pedestrian interaction features. Since the target pedestrian interaction features fuse the local pedestrian interaction features and the global pedestrian interaction features, the target pedestrian interaction features obtained in this way have higher accuracy. Therefore, based on the first hidden state features and the target pedestrian interaction features, the trajectories of multiple pedestrians are predicted, and the predicted multiple pedestrian trajectories have higher accuracy.

[0064] In one embodiment, Figure 2 As shown, multiple pedestrian trajectory sequence data are extracted from pedestrian images at multiple time steps; local pedestrian interaction features are extracted from the first hidden state features of multiple time steps respectively, and local pedestrian interaction features of multiple time steps are obtained, including:

[0065] Step 302: Slice the first hidden state feature to obtain hidden state sub-features corresponding to each pedestrian target in the pedestrian image.

[0066] Among them, the hidden state sub-feature is the hidden state feature of the pedestrian target in the pedestrian image, the first hidden state feature is the set of hidden state features of each pedestrian target in the pedestrian image, and the hidden state feature of each pedestrian target contains the position information of each pedestrian target in the pedestrian image. Slicing refers to extracting the hidden state feature of each pedestrian target from the first hidden state feature based on the position information of the pedestrian target in the pedestrian image.

[0067] As an example, there are n pedestrian targets in a pedestrian image, the first hidden state feature may be an n*32-dimensional feature, and the hidden state sub-feature may be a 1*32-dimensional feature.

[0068] As an example, step 302 includes: according to the position information of each pedestrian target in the pedestrian image, extracting the hidden state features that match the position information of each pedestrian target from the first hidden state features, and obtaining the hidden state sub-features corresponding to each pedestrian target in the pedestrian image.

[0069] Step 304 : Locate the pedestrian position of each pedestrian target in the pedestrian image, divide the pedestrian image into regions according to the pedestrian position, and obtain position information of each region.

[0070] Among them, the size of the area can be preset in advance when dividing the area. The more reasonable the area size is, the more accurate the local pedestrian interaction characteristics will be.

[0071] As an example, step 304 includes: locating the pedestrian position of each pedestrian target in the pedestrian image, dividing the pedestrian image into regions centered on each pedestrian position according to a preset region range, and obtaining position information of each region.

[0072] In step 306 , the hidden state sub-features of each pedestrian target in the image region corresponding to the position information of each region are respectively summarized to obtain the hidden features of each region.

[0073] As an example, step 306 includes: according to the positions of each pedestrian and the position information of each region, determining each pedestrian target in the image region corresponding to the region position information, and performing weighted summation on the hidden state sub-features of each pedestrian target to obtain the hidden features of each region.

[0074] Step 308 : extract pedestrian interaction features from the hidden features of each region to obtain pedestrian interaction features of each region.

[0075] As an example, step 308 includes: fully connecting the hidden features of each region to obtain each first cross hidden feature; performing nonlinear normalization processing on each first cross hidden feature through the ReLU activation function to obtain each second cross hidden feature; fully connecting each second cross-cross hidden feature to obtain each regional pedestrian interaction feature; wherein, each regional pedestrian interaction feature is used to describe the local pedestrian interaction features of each corresponding pedestrian target.

[0076] Step 310: Fusing the pedestrian interaction features of each region to obtain local pedestrian interaction features.

[0077] As an example, step 310 includes: concatenating and aggregating pedestrian interaction features in each region to obtain local pedestrian interaction features.

[0078] In the above embodiment, the pedestrian position of each pedestrian target in the pedestrian image is taken as the center, and the local area range of each pedestrian target is divided. The hidden state sub-features of each pedestrian in the local area are summarized to obtain the hidden features of each area. The pedestrian interaction features are extracted from the hidden features of each area to obtain the pedestrian interaction features of each area. The pedestrian interaction features of each area are fused to obtain the local pedestrian interaction features. Considering the local pedestrian interaction features for trajectory prediction of multiple pedestrian targets can improve the accuracy of the prediction.

[0079] In one embodiment, global pedestrian interaction features are extracted from the first hidden state features of multiple time steps respectively to obtain global pedestrian interaction features of multiple time steps, including:

[0080] The first hidden state features of multiple time steps are normalized respectively to obtain the first normalized features of multiple time steps; the first normalized features of multiple time steps are subjected to pedestrian interaction feature extraction respectively to obtain the first pedestrian interaction features of multiple time steps; the first pedestrian interaction features of multiple time steps are subjected to nonlinear processing respectively to obtain the second pedestrian interaction features of multiple time steps; the second pedestrian interaction features of multiple time steps are subjected to dimensionality reduction processing respectively to obtain the global pedestrian interaction features of multiple time steps.

[0081] Specifically, standard normalization is performed on the first hidden state features of multiple time steps to obtain the first normalized features of multiple time steps; based on the graph attention neural network, pedestrian interaction features are extracted from the first normalized features of multiple time steps to obtain the first pedestrian interaction features of multiple time steps; based on the Leaky ReLU activation function, nonlinear processing is performed on the first pedestrian interaction features of multiple time steps to obtain the second pedestrian interaction features of multiple time steps; and a Squeeze operation is performed on the second pedestrian interaction features of multiple time steps to obtain the global pedestrian interaction features of multiple time steps.

[0082] In the above embodiment, nonlinear processing is used to enable the global pedestrian interaction features to more accurately characterize the influence information between pedestrian targets, and dimensionality reduction processing is used to make the global pedestrian interaction features more streamlined, which helps to improve the accuracy of subsequent pedestrian target trajectory prediction.

[0083] In one embodiment, Figure 3 As shown in FIG, multiple pedestrian trajectory sequence data are extracted from pedestrian images at multiple time steps; the first normalized features of multiple time steps are respectively used to extract pedestrian interaction features to obtain the first pedestrian interaction features of multiple time steps, including:

[0084] In step 402 , the first normalized features of each time step are fully connected to obtain first fully connected features of multiple time steps.

[0085] As an example, step 402 includes: performing full connection processing on the first normalized features of each time step, mapping the first normalized features of each time step to the sample label space, and using each set in the sample label space as the first fully connected features of multiple time steps.

[0086] Step 404 : Generate attention weight information for multiple time steps based on the relative position information between pedestrian targets and the relative moving direction information between pedestrian targets in the pedestrian images for multiple time steps.

[0087] Among them, the attention weights between pedestrian targets are mainly affected by the relative positions between pedestrian targets and the relative travel directions between pedestrian targets. The closer the relative positions between pedestrian targets, the greater the attention weights, and the greater the travel direction conflicts between pedestrian targets, the greater the attention weights.

[0088] It is worth noting that the influence of pedestrian A on pedestrian B is not the same as the influence of pedestrian B on pedestrian A, that is, the attention weight of pedestrian A on pedestrian B is not equal to the attention weight of pedestrian B on pedestrian A.

[0089] As an example, step 404 includes: generating position weight coefficients and direction weight coefficients between pedestrian targets based on the relative position information between pedestrian targets and the relative moving direction information between pedestrian targets in pedestrian images of multiple time steps; aggregating the position weight coefficients and direction weight coefficients between pedestrian targets to generate attention weight information of multiple time steps.

[0090] As an example, the position weight coefficients and the direction weight coefficients may be aggregated by proportional superposition.

[0091] Step 406: Mask the attention weight information of multiple time steps to obtain mask weight information of multiple time steps.

[0092] Among them, mask processing can ignore the attention weights between pedestrians that have no influence, which helps to generate more accurate attention weight information between pedestrians. This mask is usually a binary matrix, where 1 means retention and 0 means ignore.

[0093] As an example, the relative position between pedestrian A and pedestrian B is 0.2 meters, but they are moving in opposite directions. Since the relative position between pedestrian A and pedestrian B is close, the attention weight between pedestrian A and pedestrian B is not 0. However, since they are moving in opposite directions, the next trajectory between pedestrian A and pedestrian B will not be affected by each other. At this time, the attention weight between pedestrian A and pedestrian B can be ignored by masking.

[0094] As an example, step 406 includes: setting a mask matrix for multiple time steps based on the position weight coefficients and direction weight coefficients between pedestrian targets at multiple time steps; masking the attention weight information of multiple time steps through the mask matrix for multiple time steps to obtain mask weight information of multiple time steps.

[0095] Step 408 : normalize the mask weight information of multiple time steps to obtain normalized weight information of multiple time steps.

[0096] As an example, step 408 includes: performing Softmax normalization processing on the mask weight information of multiple time steps to obtain normalized weight information of multiple time steps.

[0097] Step 410 : performing weighted processing on the first fully connected features of the multiple time steps according to the normalized weight information of the multiple time steps to obtain the first pedestrian interaction features of the multiple time steps.

[0098] Among them, weighted processing of the first fully connected feature according to the normalized weight information can enhance the features with large interaction effects between pedestrians, improve the accuracy of the first pedestrian interaction features at multiple time steps, and help generate more accurate global pedestrian interaction features.

[0099] As an example, step 410 includes: multiplying the normalized weight information of multiple time steps by the first fully connected features of the multiple time steps respectively to obtain the first pedestrian interaction features of the multiple time steps.

[0100] In the above embodiment, the generation of attention weight information not only takes into account the positional relationship between each pedestrian target, but also takes into account the travel direction between each pedestrian target, so the generated attention weight information is more accurate; by masking the attention weight, the accuracy of the attention weight information is further improved, which helps to generate more accurate global pedestrian interaction features, thereby helping to improve the accuracy of pedestrian trajectory prediction.

[0101] In one embodiment, predicting multiple pedestrian trajectories at the next time step based on first hidden state features at multiple time steps and target pedestrian interaction features at multiple time steps includes:

[0102] The first hidden state features of multiple time steps and the target pedestrian interaction features of multiple time steps are fused to obtain the first fused features of multiple time steps; the second hidden state features of multiple time steps are extracted from the first fused features of multiple time steps; the second hidden state features of each of the multiple time steps are fused with the preset noise features to obtain the second fused features of multiple time steps; the third hidden state features of multiple time steps are extracted from the second fused features of multiple time steps; and the trajectories of multiple pedestrians in the next time step are predicted based on the third hidden state features of multiple time steps.

[0103] In the process of predicting pedestrian trajectories, the surrounding environment of the pedestrian target may affect the next trajectory of the pedestrian target. In this embodiment, noise characteristics are used to simulate the impact of the surrounding environment on the next trajectory of the pedestrian target in an actual environment.

[0104] Specifically, the first hidden state features of multiple time steps and the target pedestrian interaction features of multiple time steps are spliced ​​and aggregated to obtain the first fusion features of multiple time steps; based on the LSTM neural network, the second hidden state features of multiple time steps are extracted from the first fusion features of multiple time steps; the second hidden state features of each of the multiple time steps are spliced ​​and aggregated with the preset noise features to obtain the second fusion features of multiple time steps; based on the LSTM neural network, the third hidden state features of multiple time steps are extracted from the second fusion features of multiple time steps; the third hidden state features of multiple time steps are fully connected to obtain the multi-pedestrian trajectories of the next time step.

[0105] In the above embodiment, the noise feature can characterize the factors that may affect the next trajectory of the pedestrian target in the actual environment. Since the noise feature is integrated, the second fusion feature can be more in line with the actual environment. Therefore, the trajectory of the next time step is predicted based on the second fusion feature that fuses the preset noise with the second state feature, which can improve the accuracy of the predicted trajectory.

[0106] In one embodiment, Figure 4 As shown, before obtaining multiple pedestrian trajectory sequence data of multiple time steps and extracting the first hidden state features of multiple time steps in the multiple pedestrian trajectory sequence data, the method further includes:

[0107] The training data of a multi-pedestrian trajectory sequence of multiple time steps and a multi-pedestrian trajectory prediction model to be trained are obtained, wherein the multi-pedestrian trajectory prediction model includes a first recurrent neural network, a second recurrent neural network, a first feature extraction module and a second feature extraction module; based on the first recurrent neural network, hidden state feature extraction is performed on the training data to obtain first training hidden state features of multiple time steps; based on the first feature extraction module, local pedestrian interaction feature extraction is performed on the first training hidden state features of multiple time steps to obtain training local pedestrian interaction features of multiple time steps; based on the second feature extraction module, the local pedestrian interaction features of the multiple time steps are extracted. The first training hidden state features are used to extract global pedestrian interaction features to obtain training global pedestrian interaction features for multiple time steps; the training local interaction pedestrian features and the training global pedestrian interaction features for multiple time steps are fused to obtain training pedestrian interaction features for multiple time steps; based on the second recurrent neural network and the training pedestrian interaction features, the pedestrian trajectory information for the next time step is predicted; according to the model loss determined by the pedestrian trajectory information for the next time step, the multi-pedestrian trajectory prediction model is iteratively optimized; wherein, the multi-pedestrian trajectory prediction model is used to predict the multi-pedestrian trajectories for the next time step based on the multi-pedestrian trajectory sequence data for multiple time steps.

[0108] Among them, the first recurrent neural network and the second recurrent neural network can be LSTM neural networks with different learning parameters, the first feature extraction module can be a pooling operation module, the second feature extraction module can be a graph attention neural network, and the pattern loss includes adversarial training loss, which can be calculated through the loss function.

[0109] Specifically, training data of a multi-pedestrian trajectory sequence of multiple time steps and a multi-pedestrian trajectory prediction model to be trained are obtained, wherein the multi-pedestrian trajectory prediction model includes a first LSTM neural network, a second LSTM neural network, a feature pooling module and a graph attention neural network; based on the first LSTM neural network, hidden state feature feature extraction is performed on the training data to obtain the first training hidden state features of multiple time steps; based on the feature pooling module, local pedestrian interaction feature extraction is performed on the first training hidden state features of multiple time steps to obtain the training local pedestrian interaction features of multiple time steps; based on the graph attention neural network, global pedestrian interaction feature extraction is performed on the first training hidden state features of multiple time steps to obtain the training global pedestrian interaction features of multiple time steps; the training local interaction Pedestrian features and trained global pedestrian interaction features are fused in proportion to obtain trained pedestrian interaction features of multiple time steps; the trained pedestrian interaction features of multiple time steps are spliced ​​and aggregated with preset noise features to obtain first fused training features of multiple time steps; based on the second LSTM neural network, hidden features are extracted from the first fused training features of multiple time steps to obtain second trained hidden state features; the second trained hidden state features are fully connected to obtain pedestrian trajectory information of the next time step; the multi-pedestrian trajectory prediction model is iteratively optimized based on the model loss determined by the pedestrian trajectory information of the next time step and the actual pedestrian trajectory information of the next time step; wherein, the multi-pedestrian trajectory prediction model is used to predict the multi-pedestrian trajectories of the next time step based on the multi-pedestrian trajectory sequence data of multiple time steps.

[0110] As an example, the specific calculation formula for model loss can be as follows:

[0111] L=L gan +αL1

[0112] Among them, L is the model loss, L gan is the adversarial loss, α is the weight coefficient, and L1 is the absolute loss;

[0113] The specific calculation formula of absolute loss L1 can be as follows:

[0114]

[0115] Among them, Y is the position of the true trajectory of the next time step, The position of the predicted trajectory for the next time step;

[0116] Adversarial loss L gan The specific calculation formula can be as follows:

[0117]

[0118] Among them, X is the trajectory data of multiple pedestrians, and Y is the position of the actual trajectory in the next time step. is the position of the predicted trajectory at the next time step, and D is the generative model.

[0119] In the above embodiment, the local pedestrian interaction characteristics and global pedestrian interaction characteristics of pedestrian targets are taken into account to predict the trajectory of each pedestrian target at the next time step. Based on the predicted trajectory data at the next time step and the actual trajectory data at the next time step, a model loss is generated through adversarial training. The multi-pedestrian trajectory prediction model is iteratively optimized based on the model loss. The optimized multi-pedestrian trajectory prediction model can be directly used to predict multiple pedestrian trajectories. Since the local pedestrian interaction characteristics and global pedestrian interaction characteristics of pedestrian targets are taken into account during the trajectory prediction process, the accuracy of pedestrian trajectory prediction can be improved. The model loss generated based on more accurate pedestrian prediction trajectories is more accurate. Iterative optimization of the multi-pedestrian trajectory prediction model based on the model loss helps to improve the performance of the multi-pedestrian trajectory prediction model, thereby improving the accuracy of the multiple pedestrian trajectories predicted and generated based on this multi-pedestrian trajectory prediction model.

[0120] In one embodiment, a pedestrian trajectory prediction method includes:

[0121] A plurality of pedestrian trajectory sequence data of a plurality of time steps are obtained, and first hidden state features of the plurality of time steps in the plurality of pedestrian trajectory sequence data are extracted, wherein the plurality of pedestrian trajectory sequence data are extracted from pedestrian images of the plurality of time steps.

[0122] After obtaining the first hidden state features of multiple time steps, the first hidden state features of the multiple time steps are sliced ​​respectively to obtain the hidden state sub-features corresponding to each pedestrian target in the multiple time steps pedestrian images; the pedestrian position of each pedestrian target is located in the pedestrian image, and the pedestrian image is divided into regions according to the pedestrian position to obtain the position information of each region; the hidden state sub-features of each pedestrian target in the image region corresponding to the position information of each region are summarized respectively to obtain the hidden features of each region; pedestrian interaction features are extracted from the hidden features of each region to obtain pedestrian interaction features of each region; the pedestrian interaction features of each region are fused to obtain local pedestrian interaction features. When performing trajectory prediction for multiple pedestrian targets, considering the local pedestrian interaction features helps to improve the accuracy of multi-pedestrian trajectory prediction.

[0123] After obtaining the local pedestrian interaction features, the first hidden state features of multiple time steps are normalized to obtain the first normalized features of multiple time steps; the first normalized features of each time step are fully connected to obtain the first fully connected features of multiple time steps; according to the relative position information between the pedestrian targets in the pedestrian images of multiple time steps and the relative moving direction information between the pedestrian targets, the attention weight information of multiple time steps is generated; the attention weight information of multiple time steps is masked to obtain the mask weight information of multiple time steps; the mask weight information of multiple time steps is normalized to obtain the normalized weight information of multiple time steps; according to the normalized weight information of multiple time steps, the attention weight information of multiple time steps is masked to obtain the mask weight information of multiple time steps. The first fully connected features of each time step are weighted to obtain the first pedestrian interaction features for multiple time steps. Nonlinear processing is performed on the first pedestrian interaction features of each time step to obtain the second pedestrian interaction features for each time step. Dimensionality reduction is performed on the second pedestrian interaction features of each time step to obtain the global pedestrian interaction features for each time step. The generation of attention weight information considers not only the positional relationship between pedestrian targets but also their direction of travel, resulting in more accurate attention weight information. Masking the attention weights further improves the accuracy of the attention weight information, facilitating the generation of more accurate global pedestrian interaction features and, consequently, improving the accuracy of pedestrian trajectory prediction.

[0124] Furthermore, the local interactive pedestrian features of multiple time steps and the global pedestrian interaction features are fused to obtain the target pedestrian interaction features of multiple time steps; the first hidden state features of multiple time steps and the target pedestrian interaction features of multiple time steps are fused to obtain the first fused features of multiple time steps; the second hidden state features of multiple time steps are extracted from the first fused features of multiple time steps; the second hidden state features of each of the multiple time steps are fused with the preset noise features to obtain the second fused features of multiple time steps; the third hidden state features of multiple time steps are extracted from the second fused features of multiple time steps; based on the third hidden state features of the multiple time steps, the multi-pedestrian trajectories of the next time step are predicted. Since the noise features are integrated, the second fused features can be made more relevant to the actual environment. Therefore, the trajectory of the next time step is predicted based on the second fused features that fuse the preset noise with the second state features, which can improve the accuracy of the predicted trajectory.

[0125] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0126] Based on the same inventive concept, the present application also provides a pedestrian trajectory prediction device for implementing the aforementioned pedestrian trajectory prediction method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more pedestrian trajectory prediction device embodiments provided below can be found in the above-mentioned limitations of the pedestrian trajectory prediction method and will not be repeated here.

[0127] In one embodiment, Figure 5 As shown, a pedestrian trajectory prediction device is provided, including: a hidden state feature extraction module 502, a local feature extraction module 504, a global feature extraction module 506, a feature fusion module 508 and a pedestrian trajectory prediction module 510, wherein:

[0128] A hidden state feature extraction module 502 is configured to obtain multiple pedestrian trajectory sequence data at multiple time steps and extract first hidden state features of the multiple time steps in the multiple pedestrian trajectory sequence data;

[0129] A local feature extraction module 504 is configured to extract local pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps;

[0130] A global feature extraction module 506 is configured to extract global pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps;

[0131] A feature fusion module 508 is configured to fuse the local pedestrian interaction features and the global pedestrian interaction features of the multiple time steps to obtain target pedestrian interaction features of the multiple time steps;

[0132] The pedestrian trajectory prediction module 510 is configured to predict the trajectories of multiple pedestrians at the next time step based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps.

[0133] In one embodiment, the local feature extraction module 504 is further configured to:

[0134] The first hidden state feature is sliced ​​to obtain hidden state sub-features corresponding to each pedestrian target in the pedestrian image; the pedestrian position of each pedestrian target is located in the pedestrian image, and the pedestrian image is divided into regions according to each pedestrian position to obtain position information of each region; the hidden state sub-features of each pedestrian target in the image region corresponding to each regional position information are respectively summarized to obtain hidden features of each region; pedestrian interaction features are extracted from the hidden features of each region to obtain pedestrian interaction features of each region; the pedestrian interaction features of each region are fused to obtain local pedestrian interaction features.

[0135] In one embodiment, the multiple pedestrian trajectory sequence data are extracted from pedestrian images at multiple time steps, and the global feature extraction module 506 is further configured to:

[0136] Normalizing the first hidden state features of the multiple time steps respectively to obtain the first normalized features of the multiple time steps; performing pedestrian interaction feature extraction on the first normalized features of the multiple time steps respectively to obtain the first pedestrian interaction features of the multiple time steps; performing nonlinear processing on the first pedestrian interaction features of the multiple time steps respectively to obtain the second pedestrian interaction features of the multiple time steps; performing dimensionality reduction processing on the second pedestrian interaction features of the multiple time steps respectively to obtain the global pedestrian interaction features of the multiple time steps.

[0137] In one embodiment, the multiple pedestrian trajectory sequence data are extracted from pedestrian images at multiple time steps, and the global feature extraction module 506 is further configured to:

[0138] The first normalized features of each time step are fully connected respectively to obtain the first fully connected features of multiple time steps; attention weight information of multiple time steps is generated according to the relative position information between the pedestrian targets in the pedestrian images of the multiple time steps and the relative moving direction information between the pedestrian targets; the attention weight information of the multiple time steps is masked to obtain mask weight information of the multiple time steps; the mask weight information of the multiple time steps is normalized to obtain normalized weight information of the multiple time steps; according to the normalized weight information of the multiple time steps, the first fully connected features of the multiple time steps are weighted respectively to obtain the first pedestrian interaction features of the multiple time steps.

[0139] In one embodiment, the pedestrian trajectory prediction module 510 is further configured to:

[0140] The first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps are fused to obtain the first fused features of the multiple time steps; the second hidden state features of the multiple time steps are extracted from the first fused features of the multiple time steps; the second hidden state features of each of the multiple time steps are fused with the preset noise features to obtain the second fused features of the multiple time steps; the third hidden state features of the multiple time steps are extracted from the second fused features of the multiple time steps; and the trajectories of multiple pedestrians in the next time step are predicted based on the third hidden state features of the multiple time steps.

[0141] Each module in the pedestrian trajectory prediction device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0142] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pedestrian trajectory data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a pedestrian trajectory prediction method is implemented.

[0143] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0145] Acquire multi-pedestrian trajectory sequence data of multiple time steps, and extract the first hidden state features of the multiple time steps in the multi-pedestrian trajectory sequence data; perform local pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps; perform global pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps; fuse the local interaction pedestrian features of the multiple time steps with the global pedestrian interaction features to obtain target pedestrian interaction features of the multiple time steps; predict the multi-pedestrian trajectory of the next time step based on the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps.

[0146] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0147] The first hidden state feature is sliced ​​to obtain hidden state sub-features corresponding to each pedestrian target in the pedestrian image; the pedestrian position of each pedestrian target is located in the pedestrian image, and the pedestrian image is divided into regions according to each pedestrian position to obtain position information of each region; the hidden state sub-features of each pedestrian target in the image region corresponding to each regional position information are respectively summarized to obtain hidden features of each region; pedestrian interaction features are extracted from the hidden features of each region to obtain pedestrian interaction features of each region; the pedestrian interaction features of each region are fused to obtain local pedestrian interaction features.

[0148] In one embodiment, the multiple pedestrian trajectory sequence data are extracted from pedestrian images at multiple time steps, and the processor further implements the following steps when executing the computer program:

[0149] Normalizing the first hidden state features of the multiple time steps respectively to obtain the first normalized features of the multiple time steps; performing pedestrian interaction feature extraction on the first normalized features of the multiple time steps respectively to obtain the first pedestrian interaction features of the multiple time steps; performing nonlinear processing on the first pedestrian interaction features of the multiple time steps respectively to obtain the second pedestrian interaction features of the multiple time steps; performing dimensionality reduction processing on the second pedestrian interaction features of the multiple time steps respectively to obtain the global pedestrian interaction features of the multiple time steps.

[0150] In one embodiment, the multiple pedestrian trajectory sequence data are extracted from pedestrian images at multiple time steps, and the processor further implements the following steps when executing the computer program:

[0151] The first normalized features of each time step are fully connected respectively to obtain the first fully connected features of multiple time steps; attention weight information of multiple time steps is generated according to the relative position information between the pedestrian targets in the pedestrian images of the multiple time steps and the relative moving direction information between the pedestrian targets; the attention weight information of the multiple time steps is masked to obtain mask weight information of the multiple time steps; the mask weight information of the multiple time steps is normalized to obtain normalized weight information of the multiple time steps; according to the normalized weight information of the multiple time steps, the first fully connected features of the multiple time steps are weighted respectively to obtain the first pedestrian interaction features of the multiple time steps.

[0152] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0153] The first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps are fused to obtain the first fused features of the multiple time steps; the second hidden state features of the multiple time steps are extracted from the first fused features of the multiple time steps; the second hidden state features of each of the multiple time steps are fused with the preset noise features to obtain the second fused features of the multiple time steps; the third hidden state features of the multiple time steps are extracted from the second fused features of the multiple time steps; and the trajectories of multiple pedestrians in the next time step are predicted based on the third hidden state features of the multiple time steps.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] Acquire training data of a multi-pedestrian trajectory sequence of multiple time steps and a multi-pedestrian trajectory prediction model to be trained, wherein the multi-pedestrian trajectory prediction model includes a first recurrent neural network, a second recurrent neural network, a first feature extraction module and a second feature extraction module; based on the first recurrent neural network, perform hidden state feature extraction on the training data to obtain first training hidden state features of multiple time steps; based on the first feature extraction module, perform local pedestrian interaction feature extraction on the first training hidden state features of the multiple time steps to obtain training local pedestrian interaction features of multiple time steps; based on the second feature extraction module, perform local pedestrian interaction feature extraction on the first training hidden state features of the multiple time steps to obtain training local pedestrian interaction features of the multiple time steps Hidden state features are trained to extract global pedestrian interaction features to obtain trained global pedestrian interaction features for multiple time steps; the trained local interaction pedestrian features for the multiple time steps are fused with the trained global pedestrian interaction features to obtain trained pedestrian interaction features for the multiple time steps; based on the second recurrent neural network and the trained pedestrian interaction features, pedestrian trajectory information for the next time step is predicted; according to the model loss determined based on the pedestrian trajectory information for the next time step, the multi-pedestrian trajectory prediction model is iteratively optimized; wherein, the multi-pedestrian trajectory prediction model is used to predict the multi-pedestrian trajectories for the next time step based on the multi-pedestrian trajectory sequence data for the multiple time steps.

[0156] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0157] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0158] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0159] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A pedestrian trajectory prediction method, characterized in that: The method comprises: Acquire multiple pedestrian trajectory sequence data at multiple time steps, and extract first hidden state features of the multiple time steps in the multiple pedestrian trajectory sequence data; Extracting local pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps; Performing global pedestrian interaction feature extraction on the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps; Fusing the local pedestrian interaction features of the multiple time steps with the global pedestrian interaction features to obtain target pedestrian interaction features of the multiple time steps; The first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps are fused to obtain the first fused features of the multiple time steps; the second hidden state features of the multiple time steps are extracted from the first fused features of the multiple time steps; the second hidden state features of each of the multiple time steps are fused with the preset noise features to obtain the second fused features of the multiple time steps; the third hidden state features of the multiple time steps are extracted from the second fused features of the multiple time steps; and the trajectories of multiple pedestrians in the next time step are predicted based on the third hidden state features of the multiple time steps.

2. The method according to claim 1, characterized in that The multiple pedestrian trajectory sequence data are extracted from pedestrian images at multiple time steps; and the local pedestrian interaction feature extraction is performed on the first hidden state features of the multiple time steps to obtain the local pedestrian interaction features of the multiple time steps, including: Slicing the first hidden state feature to obtain hidden state sub-features corresponding to each pedestrian target in the pedestrian image; Locating the pedestrian position of each pedestrian target in the pedestrian image, dividing the pedestrian image into regions according to the pedestrian positions, and obtaining position information of each region; Summarizing the hidden state sub-features of each pedestrian target in the image region corresponding to the position information of each region to obtain the hidden features of each region; Extracting pedestrian interaction features from hidden features of each region to obtain pedestrian interaction features of each region; The pedestrian interaction features of each region are fused to obtain local pedestrian interaction features.

3. The method according to claim 1, characterized in that The extracting global pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps includes: Normalizing the first hidden state features of the multiple time steps respectively to obtain first normalized features of the multiple time steps; Performing pedestrian interaction feature extraction on the first normalized features of the multiple time steps respectively to obtain first pedestrian interaction features of the multiple time steps; Performing nonlinear processing on the first pedestrian interaction features of the multiple time steps respectively to obtain second pedestrian interaction features of the multiple time steps; Dimensionality reduction processing is performed on the second pedestrian interaction features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps.

4. The method according to claim 3, characterized in that The multi-pedestrian trajectory sequence data is extracted from pedestrian images at multiple time steps; The extracting pedestrian interaction features from the first normalized features of the multiple time steps to obtain the first pedestrian interaction features of the multiple time steps includes: The first normalized features of each time step are fully connected to obtain the first fully connected features of multiple time steps; generating attention weight information for the multiple time steps according to the relative position information between the pedestrian targets and the relative moving direction information between the pedestrian targets in the pedestrian images for the multiple time steps; Performing masking on the attention weight information of the multiple time steps to obtain mask weight information of the multiple time steps; Normalizing the mask weight information of the multiple time steps to obtain normalized weight information of the multiple time steps; According to the normalized weight information of the multiple time steps, weighted processing is performed on the first fully connected features of the multiple time steps to obtain the first pedestrian interaction features of the multiple time steps.

5. The method according to claim 1, wherein Before acquiring multiple pedestrian trajectory sequence data of multiple time steps and extracting first hidden state features of the multiple time steps in the multiple pedestrian trajectory sequence data, the method further includes: Acquire training data of a multi-pedestrian trajectory sequence of multiple time steps and a multi-pedestrian trajectory prediction model to be trained, wherein the multi-pedestrian trajectory prediction model includes a first recurrent neural network, a second recurrent neural network, a first feature extraction module, and a second feature extraction module; Based on the first recurrent neural network, extract hidden state features from the training data to obtain first training hidden state features for multiple time steps; Based on the first feature extraction module, extract local pedestrian interaction features from the first training hidden state features of the multiple time steps to obtain training local pedestrian interaction features of the multiple time steps; Based on the second feature extraction module, performing global pedestrian interaction feature extraction on the first training hidden state features of the multiple time steps to obtain training global pedestrian interaction features of the multiple time steps; Fusing the trained local interactive pedestrian features of the multiple time steps with the trained global pedestrian interaction features to obtain the trained pedestrian interaction features of the multiple time steps; Predicting pedestrian trajectory information at the next time step based on the second recurrent neural network and the trained pedestrian interaction features; Iteratively optimizing the multi-pedestrian trajectory prediction model according to the model loss determined by the pedestrian trajectory information at the next time step; The multi-pedestrian trajectory prediction model is used to predict the multi-pedestrian trajectories of the next time step based on the multi-pedestrian trajectory sequence data of the multiple time steps.

6. A pedestrian trajectory prediction device, characterized in that: The device comprises: A hidden state feature extraction module is used to obtain multiple pedestrian trajectory sequence data at multiple time steps and extract the first hidden state features of the multiple time steps in the multiple pedestrian trajectory sequence data; A local feature extraction module is used to extract local pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain local pedestrian interaction features of the multiple time steps; A global feature extraction module is used to extract global pedestrian interaction features from the first hidden state features of the multiple time steps respectively to obtain global pedestrian interaction features of the multiple time steps; A feature fusion module is used to fuse the local interactive pedestrian features of the multiple time steps with the global pedestrian interaction features to obtain target pedestrian interaction features of the multiple time steps; The pedestrian trajectory prediction module is used to fuse the first hidden state features of the multiple time steps and the target pedestrian interaction features of the multiple time steps to obtain the first fused features of the multiple time steps; extract the second hidden state features of the multiple time steps from the first fused features of the multiple time steps; fuse the second hidden state features of each of the multiple time steps with the preset noise features to obtain the second fused features of the multiple time steps; extract the third hidden state features of the multiple time steps from the second fused features of the multiple time steps; and predict the multiple pedestrian trajectories of the next time step based on the third hidden state features of the multiple time steps.

7. The device according to claim 6, characterized in that The local feature extraction module is configured to slice the first hidden state feature to obtain hidden state sub-features corresponding to each pedestrian target in the pedestrian image; locate the pedestrian position of each pedestrian target in the pedestrian image, and divide the pedestrian image into regions according to each pedestrian position to obtain position information of each region; Summarizing the hidden state sub-features of each pedestrian target in the image region corresponding to the position information of each region to obtain the hidden features of each region; Pedestrian interaction features are extracted from the hidden features of each region respectively to obtain pedestrian interaction features of each region; and the pedestrian interaction features of each region are fused to obtain local pedestrian interaction features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images