A multi-scene adaptive pedestrian trajectory prediction method based on an attention mechanism

Abstract

The application discloses a multi-scene adaptive pedestrian trajectory prediction method based on an attention mechanism, realizes trajectory prediction in multiple scenes by constructing an adaptive pedestrian trajectory prediction model, and comprises the following steps: a multi-head attention mechanism is used to extract historical trajectory features, a social attention module is used to capture the interaction between pedestrians by using a graph convolution network, and a trajectory alignment module is used to further compare and align trajectory features in different scenes; finally, a sample weighted reinforcement training strategy is used to optimize the learning ability of the model for complex trajectories. The application can significantly improve the accuracy of pedestrian trajectory prediction and improve overall prediction performance and robustness.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of pedestrian trajectory prediction, and in particular to a multi-scene adaptive pedestrian trajectory prediction method based on an attention mechanism. BACKGROUND

[0002] With the rapid advancement of urbanization and the continuous increase in population density, urban traffic has become increasingly complex, and pedestrian safety issues have become increasingly prominent. In this context, pedestrian trajectory prediction technology is increasingly important in intelligent transportation systems, robot navigation, autonomous vehicles, and video surveillance. Pedestrian trajectory prediction technology aims to analyze and predict the future movement paths of pedestrians, thereby providing strong support for traffic management and safety measures. In addition, pedestrian trajectory prediction technology can significantly improve traffic safety by accurately predicting pedestrian movement paths and reducing the occurrence of traffic accidents. By improving traffic signal control and driving route planning, the overall traffic system's operational efficiency can also be improved. Furthermore, this technology supports intelligent transportation systems and autonomous driving technology, helping autonomous vehicles make more secure and effective decisions. In robot navigation and video surveillance, pedestrian trajectory prediction improves navigation accuracy and public safety. Pedestrian trajectory prediction methods not only promote smart city construction but also enhance the intelligent level of urban management, ultimately bringing significant social and economic benefits.

[0003] Existing pedestrian trajectory prediction techniques mainly include models based on recurrent neural networks (RNN), generative adversarial networks (GAN), and attention mechanisms. Pedestrian trajectory prediction methods based on LSTM (Long Short-Term Memory) such as S-LSTM, Group-LSTM, and StarNet perform well in specific scenarios by processing sequence data and capturing long-term dependencies. Prediction methods based on GAN such as SGAN, Sophie, and Social Way generate more realistic pedestrian trajectory samples through adversarial games. Although these models exhibit excellent performance in specific application scenarios, they still have limitations when dealing with diverse and complex scenarios. For example, existing models usually assume uniform pedestrian behavior, ignoring the spatiotemporal differences in crowd distribution, ultimately failing to fully capture the diversity of individual behavior in different scenarios. In addition, traditional methods rely on general feature representation of single samples, which easily overlooks key individual-level fine-grained features, resulting in insufficient adaptability and accuracy when dealing with trajectory prediction in complex environments.

[0004] Therefore, traditional trajectory prediction methods often rely on simple motion models, making it difficult to accurately capture complex spatiotemporal relationships and the interactive effects of multiple factors. Therefore, how to effectively integrate multi-source data and improve the accuracy and adaptability of prediction models has become an important technical problem that pedestrian trajectory prediction methods need to break through. SUMMARY

[0005] In order to overcome the problems of the prior art that the prediction accuracy of pedestrian trajectory in diversified scenes is limited, the spatio-temporal difference of crowd distribution is ignored, and the interaction information of pedestrians is not captured enough, the application provides a multi-scene adaptive pedestrian trajectory prediction method based on an attention mechanism. An adaptive pedestrian trajectory prediction model is constructed to realize trajectory prediction in multiple scenes. The model design includes using a multi-head attention mechanism to extract historical trajectory features, and a social attention module using a graph convolution network to capture pedestrian interaction, and a trajectory alignment module to further compare and align trajectory features in different scenes. Finally, a sample weighted reinforcement training strategy is used to optimize the model's learning ability for complex trajectories and improve the robustness and accuracy of the prediction.

[0006] The technical solution adopted by the application to solve its technical problems is:

[0007] A multi-scene adaptive pedestrian trajectory prediction method based on an attention mechanism, the method comprising the following steps:

[0008] (1) Pedestrian historical position information acquisition, image information in the scene is collected, and historical observation positions of a set time step are extracted as basic position information for pedestrian trajectory prediction;

[0009] (2) Design a social attention module to capture the interaction information of pedestrians, and use a graph convolution network to further learn the motion trend and output spatio-temporal features;

[0010] (3) Design a trajectory alignment module to fully capture the diversity of individual behavior in different scenes, calculate the difference and align to obtain the comparative difference features of the pedestrian trajectory, and finally integrate them into the backbone network to obtain the prediction result of the pedestrian trajectory;

[0011] (4) Pedestrian trajectory prediction backbone network model design: based on a multi-head attention mechanism to capture feature information of pedestrian historical trajectory sequence, and combined with a recurrent neural network to predict pedestrian trajectory position in future time steps, finally enabling the model to adapt to trajectory prediction tasks in different scenes;

[0012] (5) Pedestrian trajectory prediction model training: a sample weighted reinforcement training strategy is used to strengthen the learning of complex trajectories; each training sample is assigned a weight, and these weights are dynamically updated during training. For those samples with poor prediction accuracy of pedestrian trajectory, increase their training weight, so that they get more attention in subsequent training, thereby improving the prediction accuracy of the model for complex trajectory samples.

[0013] The technical concept of the present application is: first, by introducing the multi-head attention mechanism and the social attention module, the feature extraction and interaction information capture ability of the pedestrian trajectory are significantly improved. Second, the trajectory alignment module solves the challenge of trajectory comparison between different scenes, enhancing the adaptive ability of the model in diversified scenes. Finally, the sample weighted reinforcement training strategy is adopted, which effectively improves the learning ability of the model for complex trajectory samples and enhances the robustness and accuracy of the prediction.

[0014] The beneficial effects of the present application mainly include: the present application can significantly improve the accuracy of pedestrian trajectory prediction, accurately capture spatio-temporal features through the multi-head attention mechanism and social attention module, and adaptively compare the trajectory differences between different scenes through the trajectory alignment module; ultimately, the model only needs a small amount of sample data of the target scene to achieve high-precision pedestrian trajectory prediction in that scene, so the method proposed by the present application can better adapt to the trajectory prediction task in different scenes. At the same time, the sample weighted reinforcement training strategy enhances the learning ability of the model for complex trajectories, improves the overall prediction performance and robustness. The weighted fusion method in the model training and optimization process enables the prediction model to quickly adapt to complex scenes, which can effectively reduce the potential risk of collision. BRIEF DESCRIPTION OF DRAWINGS

[0015] Fig. 1 is an implementation flowchart of the multi-scene adaptive pedestrian trajectory prediction method based on the attention mechanism;

[0016] Fig. 2 is a block diagram of the overall model structure of the multi-scene adaptive pedestrian trajectory prediction method based on the attention mechanism;

[0017] Fig. 3 is a model reinforcement training flowchart of the multi-scene adaptive pedestrian trajectory prediction method based on the attention mechanism. DETAILED DESCRIPTION

[0018] The present application will be further described below with reference to the accompanying drawings.

[0019] Reference Figs. 1-3 A multi-scene adaptive pedestrian trajectory prediction method based on the attention mechanism, the method comprising the following steps:

[0020] (1) Pedestrian historical position information acquisition, collecting image information in the scene, extracting historical observation positions of a set time step as basic position information for pedestrian trajectory prediction;

[0021] The process of step (1) is as follows:

[0022] (1.1) Collect image information in the scene, extract time step t1 = [1, T obsthe historical observation positions of the pedestrians as basic position information for pedestrian trajectory prediction, that is,

[0023]

[0024] where T obs is set according to the actual situation, and the reference is 3.6 seconds, p i represents the basic position information of the ith pedestrian, including the two-dimensional plane position information of the pedestrian at time step t1∈[1,T obs ], as follows:

[0025]

[0026] where, and represent the two-dimensional horizontal and vertical position coordinate information of the ith pedestrian at time step t1, respectively;

[0027] (1.2) collect and extract the trajectory position information p i* of the ith pedestrian at time step t2∈[T obs +1,T pre ] :

[0028]

[0029] where T pre is set according to the actual situation and Tobs, and the reference is 4.8 seconds, and represent the two-dimensional horizontal and vertical position coordinate information of the ith pedestrian at time step t2, respectively;

[0030] (2) design a social attention module to capture the interaction information of pedestrians, and use a graph convolution network to further learn the motion trend and output the spatiotemporal features;

[0031] The process of step (2) is:

[0032] (2.1) node representation design for pedestrian trajectory prediction, in the social attention module, the motion trajectory of the pedestrian is described by a graph, and the interaction between pedestrians in each time step is abstractly defined as a graph representation G t1 :

[0033] G t1 =(V t1 ,E t1 ,F t1 )

[0034] where, is the vertex set of the graph, representing all pedestrians at time t1, that is, the vertex set of N pedestrians; the value of the ith pedestrian at time t1 Edge set The relationship between the ith and jth pedestrians is represented by an edge, and a feature set is defined as:

[0035]

[0036] where W is the weight, and σ(*) is the ReLU activation function.

[0037] (2.2) The social soft attention function is calculated. Since the interaction between pedestrians is complex and diverse, these interactions are roughly divided into three categories according to the actual situation, and the three types of interaction include walking towards each other, walking away from each other, and walking side by side. In order to better analyze and simulate the interaction mode of pedestrians, a new social soft attention function is proposed to construct the adjacency matrix The attention weight between the ith and jth pedestrians is represented by an edge, and a feature set

[0038]

[0039] where, and are the velocity vectors of pedestrians i and j at time t1, respectively; and α and β represent the velocity vectors and and the angle between the two pedestrian connection paths, and represent the position coordinates of pedestrians i and j at time t1, respectively.

[0040] In addition, when i = j, i.e., the two pedestrian nodes are the same, the self-attention value of the node is represented by γ, and the value of γ is set to 0.1 according to the maximum value of the element in the adjacency matrix A t1

[0041] ① When the two pedestrians walk towards each other, the angles α and β formed by the velocity vectors of the two people are both acute angles. At this time, cosα and cosβ are both greater than zero, the calculation result is greater than 0; it indicates that there is a significant influence between the two people, and the two pedestrians need to pay attention to each other to avoid head-on collision;

[0042] ② When the two pedestrians walk away from each other, the angles α and β formed by the velocity vectors of the two people are both obtuse angles. At this time, cosα and cosβ are both less than zero, the calculation result is limited to 0; it indicates that there is no direct mutual influence between the two pedestrians walking away from each other;

[0043] ​③ When two pedestrians walk side by side, i.e., pedestrian i is in front of pedestrian j, the included angle a formed by the speed vectors of the two pedestrians is obtuse, while the included angle β is acute; if the moving speed of pedestrian j is significantly higher than that of pedestrian i, there is a possibility of future collision, and at this time the output of the attention function is greater than 0, indicating that there is a potential collision risk; on the contrary, if the speed of pedestrian j is lower than that of pedestrian i, there is no collision risk, and the output of the attention function will be set to 0.

[0044] In order to better reflect the size of the collision risk, a natural function form is adopted to represent the nonlinear exponential changes between pedestrians affected by the Euclidean distance and relative speed; this processing method ensures that when the distance between pedestrians is close, the collision risk is higher, and the attention weight between pedestrians is also larger, so that the network can better extract and understand the characteristics of social interaction;

[0045] (2.3) Trajectory domain feature output, further extract spatio-temporal features by using graph convolution network (GCN), perform graph convolution operation at time step t1, i.e.,

[0046]

[0047] where A t1 is the adjacency matrix at time step t1, D t1 is the graph Laplacian matrix, and E is the identity matrix. The adjacency matrices from time steps T1 to T obs are stacked as:

[0048]

[0049] The Laplacian matrices are stacked as:

[0050]

[0051] At the same time, the feature matrices of the lth layer in the GCN layer are stacked as:

[0052]

[0053] Therefore, the GCN output F l+1 after the lth layer is calculated as:

[0054]

[0055] where W l is the learnable parameter of the lth layer, and three GCN network layers are used to extract the spatio-temporal feature representation of the observed trajectory, so l∈[1,3]; and the feature representation obtained after the last layer of GCN network is defined as where N is the number of pedestrians in the trajectory domain, D f and L obs are the data dimensions of the extracted features;

[0056] (3) The trajectory alignment module is designed to fully capture the diversity of individual behavior in different scenarios, calculate the differences and align the comparative difference features of the pedestrian trajectories, and finally integrate them into the backbone network to obtain the prediction results of the pedestrian trajectories;

[0057] Since there are differences between the trajectory domains in different scenarios, the trajectory alignment module is designed to fully capture the diversity of individual behavior in different scenarios, so that the model can better adapt to the trajectory prediction task in different scenarios.

[0058] Specifically, first, the pedestrian trajectory information in different scenarios is obtained through the social attention module designed in step (2) to obtain the trajectory features of the corresponding trajectory domain; then the trajectory alignment module is used for deep refinement, comparison and adaptation; finally, the pedestrian trajectory comparative difference feature dif is obtained i , which is integrated into the backbone network to alleviate the misalignment between different pedestrians and different sample spaces; the process of step (3) is as follows:

[0059] (3.1) Source trajectory and target trajectory information acquisition, collect pedestrian trajectory position information in source and target scenarios, and obtain the graph representation G t1 (source)={V t1 (source),E t1 (source),F t1 (source)} of the pedestrian trajectory in the source scenario, and the graph representation G t1 (target)=

[0060] {V t1 (target),E t1 (target),F t1 (target)} of the pedestrian trajectory in the target scenario;

[0061] (3.2) G t1 (source) and G t1 (target) are respectively taken as the input of the social attention module designed in step (2), and the feature representation F(source) of the source trajectory and the feature representation F(target) of the target trajectory are obtained, which contains the spatio-temporal information of each pedestrian in different trajectory domains;

[0062] (3.3) Trajectory domain attention score calculation, since the same feature vector may behave differently in different trajectory domains, in order to determine the representativeness of the feature vector of the pedestrian in the trajectory domain, the relative relevance between the feature vector and the trajectory domain is learned by using the attention mechanism, and the attention scores of the i-th pedestrian in the source trajectory domain and the target trajectory domain are respectively i(source) and p i (target), respectively, to measure the importance of each pedestrian feature in the specific trajectory domain, calculated as:

[0063]

[0064] where a and W h are learnable weight parameters, tanh(*) is the hyperbolic tangent function, f i (source) and f i (target) are the position features of the i-th pedestrian in the source trajectory domain and the target trajectory domain, respectively;

[0065] (3.4) Pedestrian trajectory contrast difference feature dif i is calculated based on the attention score value p i (source) and p i (target) calculated in step (3.3), to calculate the final feature representation c i (source) and c i (target) of the source trajectory domain and the target trajectory domain, respectively:

[0066] c i (source) = p i (source) · f i (source)

[0067] c i (target) = p i (target) · f i (target)

[0068] Then the individual-level difference between the final feature representations of the two trajectory domains is calculated, and the calculated difference is mapped to the data feature h i with the same dimension as the feature vector through a fully connected layer, i.e.:

[0069] dif i = FL(c i (source) - c i (target), W fl )

[0070] where W fl is a learnable weight matrix, and FL(*,*) is a fully connected layer. Finally, the calculated pedestrian trajectory contrast difference feature dif i is integrated into the backbone network model to realize pedestrian trajectory prediction with comprehensive feature analysis;

[0071] (4) Pedestrian trajectory prediction backbone network model design: based on the multi-head attention mechanism to capture the feature information of the pedestrian historical trajectory sequence, and combined with the recurrent neural network to predict the pedestrian trajectory position at the future time step, finally making the model adapt to the trajectory prediction task in different scenarios;

[0072] The processing process of the pedestrian trajectory prediction backbone network model is:

[0073] (4.1) For the pedestrian historical trajectory position First, the position is encoded by using the neural network embedding layer, and the encoded feature data is obtained

[0074]

[0075] Among them, is the embedding layer; W e represents the weight coefficient of the embedding layer, which is obtained by step-by-step iterative optimization through network training; and t1∈[1,T obs ], iv[1,N];

[0076] (4.2) Since there are differences in the motion patterns of pedestrians in different scenarios, the difference features dif i between the pedestrian trajectories in different scenarios obtained in step (3) are introduced into the backbone network to solve the shift problem of different trajectory domain comparisons, that is, the fused feature is:

[0077]

[0078] Among them, concat(*) represents the data channel splicing operation;

[0079] (4.3) The encoded position feature data is projected into the query matrix, the key matrix and the value matrix, and for the i-th pedestrian sequence, the and matrices are:

[0080]

[0081] Among them, f Q (*), f K (*) and f V (*) correspond to the query, key and value functions, respectively, and represent the query, key and value matrices of the i-th pedestrian under the j-th attention head, and the total number of multi-head attention heads is M, so j∈[1,M].

[0082] (4.4) Attention coefficient of multi-head attention and the feature vector calculation based on multi-head attention is:

[0083]

[0084] wherein, softmax(*) is the softmax normalization function, d k is the normalization factor; is the feature calculation result of the i-th pedestrian in the j-th attention head; h i is the feature vector result of the i-th pedestrian containing time dependence after multi-head attention processing, and concat(*) is the data channel splicing operation;

[0085] Therefore, the entire data operation based on the multi-head attention mechanism is:

[0086]

[0087] wherein, Multihead(*) is the multi-head attention operation function;

[0088] (4.5) Constructing a trajectory decoder based on a recurrent neural network, in order to enhance the robustness of pedestrian trajectory sequence prediction, a random vector noise i obeys a Gaussian distribution is integrated into the trajectory decoder, then the input feature vector v i of the trajectory decoder is:

[0089] v i = concat(h i ,noise i )

[0090] Secondly, based on the multi-head attention mechanism and the recurrent neural network, a trajectory decoder is constructed, and combined with the full connection layer and the ReLU activation function, the final prediction result of the pedestrian trajectory position is obtained, that is:

[0091] p′ i* = Decoder(v i ,W p )

[0092] wherein, p′ i* is the trajectory prediction result of the i-th pedestrian, Decoder(*) is the trajectory decoder processing function, and W p is the network weight coefficient of the trajectory decoder;

[0093] (5) Pedestrian trajectory prediction model training: a sample weighted reinforcement training strategy is adopted to strengthen the learning of complex trajectories; each training sample is assigned a weight, and these weights are dynamically updated during training. For those samples with poor pedestrian trajectory prediction accuracy, increase their training weight so that they receive more attention in subsequent training, thereby improving the prediction accuracy of the model for complex trajectory samples.

[0094] The process of step (5) is as follows:

[0095] (5.1) Trajectory prediction loss function design, the goal is to predict the trajectory position of pedestrians at future time step t2∈[T obs +1,T pre ], therefore, define the loss function loss as:

[0096]

[0097] Where p i* and p′ i* represent the true position vector and predicted position vector of pedestrian i at time step [T obs +1,T pre ], MSE(*,*) is the mean square error, and N is the total number of pedestrians;

[0098] (5.2) Parameter initialization, the total number of training data is M, then initialize the initial weight of all samples:

[0099]

[0100] Where m∈[1,M], and the initial network parameters of the pedestrian trajectory prediction model are initialized as θ0;

[0101] (5.3) Iterative training of pedestrian trajectory prediction model, and calculating the loss value of each sample in the training data set:

[0102]

[0103] Where, represents the initial weight of the mth training sample, loss m represents the direct loss value of the mth training sample calculated by step (5.1), L m represents the weighted loss value of the mth sample;

[0104] (5.4) Model parameter update, calculate the average loss Loss1 of the first iteration:

[0105]

[0106] Based on the loss calculation result Loss1, the model parameter optimization is performed by using the gradient descent method, and the parameter θ1 of the pedestrian trajectory prediction model after the first iteration is obtained.

[0107] (5.5) Update the sample training weight according to the loss function calculation result loss1, loss2,..., loss M of each sample in the first iteration, and update the weighted value of each set of training sample data, that is:

[0108]

[0109] Wherein, is the training weight of the mth sample after the first iteration update, μ is the preset learning rate of the sample training weight, and μ>0; softmax(*) is a softmax function; since the loss value of the complex sample is larger in the training process, the training weight of the complex sample can be further increased by using this method, and the training weight of the simple sample is reduced; this reinforcement training strategy enables the model to dynamically adjust the importance of each training sample during the iterative training process, so that the training sample with poor pedestrian trajectory prediction effect can obtain more attention. Finally, this improves the robustness of the pedestrian trajectory prediction model under complex conditions;

[0110] (5.6) Repeat steps (5.3) to (5.5) for a total of T iterations, and obtain the total loss calculation values Loss1, Loss2,..., Loss T and the network model parameters θ1, θ2,..., θ T of the pedestrian trajectory prediction model respectively;

[0111] (5.7) Based on the principle that the larger the loss function value is, the poorer the prediction performance of the sample is, and vice versa, the final pedestrian trajectory prediction model is obtained by using the model parameter fusion method. First, the fusion factor τ k of each model in the T training models is obtained:

[0112]

[0113] Wherein, τ k is the fusion factor of the pedestrian trajectory prediction model obtained by the kth iteration training, Loss k represents the total loss value of the kth iteration training, and k∈[1, T];

[0114] (5.8) Based on the fusion factor, the parameters of the T pedestrian trajectory prediction models are weighted and fused:

[0115]

[0116] where θ mix is the parameter of the fused pedestrian trajectory prediction model, θ k is the parameter of the model at the kth iteration. Finally, θ mix is applied to the network parameters of the pedestrian trajectory prediction model and is used as the final trained pedestrian trajectory prediction model.

[0117] The multi-scene adaptive pedestrian trajectory prediction method based on the attention mechanism of the embodiment is implemented as follows:

[0118] First step: deploy camera sensors above multiple target monitoring areas for real-time monitoring, which are respectively defined as source areas and target areas. A large number of dynamic videos of pedestrian walking in the source areas and a small amount of video data in the target areas are shot in real time, and image data is collected at a fixed frequency, with a time step of t1=[1,T obs ], to obtain pedestrian image history data in the two areas, denoted as:

[0119]

[0120] where T obs is set according to the actual situation, and the reference is 3.6 seconds, p i represents the basic position information of the ith pedestrian, including the two-dimensional plane position information of the pedestrian at time step t1∈[1,T obs ], specifically:

[0121]

[0122] where x and y represent the two-dimensional horizontal and vertical position coordinate information of the ith pedestrian at time step t1;

[0123] Second step: continue to sample image data in the two areas at the same frequency, and collect the trajectory position information p i* of the pedestrian at time step t2∈[T obs +1,T pre ] as the real trajectory data in the data set:

[0124]

[0125] where T pre is set according to the actual situation and Tobs, and the reference is 4.8 seconds, and y represent the two-dimensional horizontal and vertical position coordinate information of the ith pedestrian at time step t2;

[0126] Third step: collect the pedestrian history trajectory information pi The node representation is performed, and the node in each time step is defined as a graph representation G t1 :

[0127] G t1 = (V t1 , E t1 , F t1 )

[0128] wherein, is the vertex set of pedestrians in the graph, representing all pedestrians at time t1, i.e., the vertex set of N pedestrians; The value of the i-th pedestrian at time t1 observed coordinate position Edge set represents the relationship between the i-th and j-th pedestrians, and a relationship edge is constructed between each two pedestrians; and the feature set is defined as:

[0129]

[0130] wherein, W is the weight, and σ(*) is the ReLU activation function;

[0131] Fourth step: calculate the social soft attention function and construct the adjacency matrix The attention weight between the i-th and j-th pedestrians is represented as follows:

[0132]

[0133] wherein, and are the velocity vectors of pedestrians i and pedestrians j at time t1, respectively; and α and β represent the angles between the velocity vectors and and the connecting path between the two pedestrians, and represent the position coordinates of pedestrians i and pedestrians j at time t1, respectively, and the value of γ is set to 0.1;

[0134] Fifth step: integrate information through a graph convolutional neural network (GCN), and perform graph convolution operation at time step t1, i.e.,

[0135]

[0136] wherein, A t1 is the adjacency matrix at time step t1, D t1 is the submatrix of the graph, and E is the unit matrix. The adjacency matrices from time steps T1 to T obs are stacked as:

[0137]

[0138] The sub-matrix is stacked as:

[0139]

[0140] Meanwhile, the feature matrix of the l-th layer in the GCN hierarchy is stacked as:

[0141]

[0142] Therefore, the GCN output F l+1 after the l-th layer is calculated as:

[0143]

[0144] where W l is a learnable parameter of the l-th layer, and a three-layer GCN network is used to extract the spatio-temporal feature representation of the observed trajectory, so l ∈ [1, 3]; and the feature representation obtained after the last layer of the GCN network is defined as where N is the number of pedestrians in the trajectory domain, D f and L obs are the data dimensions of the extracted features, and the feature representation F(source) of the source trajectory and the feature representation F(target) of the target trajectory are obtained.

[0145] Step 6: Calculate the attention score for the source trajectory and the target trajectory, which is calculated as:

[0146]

[0147] where α and W h are learnable weight parameters, tanh(*) is the tangent function, f i (source) and f i (target) are the position features of the i-th pedestrian in the source trajectory domain and the target trajectory domain, respectively.

[0148] Step 7: Based on the attention score calculated in Step 6, the final feature representations c i (source) and c i (target) of the source trajectory domain and the target trajectory domain are calculated respectively:

[0149] c i (source) = ρ i (source) · f i (source)

[0150] c i (target) = ρ i(target)·f i (target)

[0151] Next, the individual-level differences between the final feature representations of the two trajectory domains are calculated, and the calculated differences are mapped to the feature vector h through a fully connected layer. i Data features with the same dimension, namely:

[0152] dif i =FL(c i (source)-c i (target),W fl )

[0153] Among them, W fl Let be a learnable weight matrix, and FL(*,*) be a fully connected layer;

[0154] Step 8: Collect pedestrian historical trajectory information from the source area The data is integrated into the backbone network for position encoding, and the encoded feature data is obtained.

[0155]

[0156] in, For embedding layer; W e The weight coefficients of the embedding layer are obtained through iterative optimization during network training; and t1∈[1,T] obs ], i∈[1,N];

[0157] Step 9: Compare the pedestrian trajectories obtained in Step 7 with the difference features in the .dif file. i The location encoding features obtained in step 8 Features after channel splicing and fusion for:

[0158]

[0159] Wherein, concat(*) represents the data channel splicing operation;

[0160] Step 10: Utilize the multi-head attention mechanism in the backbone network to perform multi-head attention feature fusion on the concatenated data. The operation is as follows:

[0161]

[0162] Among them, Multihead(*) is the multi-head attention operation function;

[0163] Step 10: Add extra noise to the multi-head attention fusion features obtained in step 9, and input them into the track decoder of the backbone network. Combine the full connection layer and the ReLU activation function to obtain the prediction result of the pedestrian track position, i.e.,

[0164] p' i* = Decoder(v i , W p )

[0165] where p' i* is the track prediction result of the i-th pedestrian, Decoder(*) is the track decoder processing function, and W p is the network weight coefficient of the track decoder.

[0166] Step 12: Based on the prediction result of the pedestrian future track in step 11 and the real pedestrian track in the training data set in step 2, design a loss function based on mean square error (MSE) to calculate the difference between the predicted value and the real value, and define the loss function loss as:

[0167]

[0168] where p i* and p' i* represent the real position vector and the predicted position vector of pedestrian i at time steps [T obs +1, T pre ], MSE(*, *) is the calculation of mean square error, and N is the total number of pedestrians.

[0169] Step 13: Initialize a fixed training weight for each training sample:

[0170]

[0171] where m ∈ [1, M], and the initial network parameters of the pedestrian track prediction model are initialized as θ0.

[0172] Step 14: Calculate the total loss value of the current iteration using the loss function designed in step 12 and the weighted training value of the sample;

[0173] Step 15: Update the network parameters of the pedestrian track prediction model, and update the weighted training weights of all samples;

[0174] Step 16: Repeat steps 14 to 15 until the preset number of iterations is reached. Then, according to the total loss calculation result of each iteration, perform model parameter weighted fusion to obtain the trained pedestrian track prediction model.

[0175] Seventeenth step: deploy the trained pedestrian trajectory prediction model to the server, and transmit the data collected by the camera sensors in each area to the server respectively, and finally realize the pedestrian trajectory prediction at future time in different area scenarios through the pedestrian trajectory prediction model.

[0176] The embodiments of the present specification are merely a list of implementation forms of the inventive concept, and are only for the purpose of illustration. The protection scope of the present application should not be regarded as being limited to the specific forms presented in the present embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A multi-scene adaptive pedestrian trajectory prediction method based on an attention mechanism, characterized in that, The method comprises the following steps: (1) Pedestrian historical position information acquisition, image information in the scene is collected, and historical observation positions in a set time step are extracted as basic position information for pedestrian trajectory prediction; (2) A social attention module is designed to capture the interaction information of pedestrians, and a graph convolution network is used to further learn the motion trend and output the spatio-temporal features; (3) A trajectory alignment module is designed to fully capture the diversity of individual behavior in different scenes, the difference is calculated and aligned to obtain the comparative difference features of the pedestrian trajectory, and finally the features are integrated into the backbone network to obtain the prediction result of the pedestrian trajectory; (4) Pedestrian trajectory prediction backbone network model design: based on the multi-head attention mechanism, the feature information of the pedestrian historical trajectory sequence is captured, and the recurrent neural network is combined to predict the pedestrian trajectory position in the future time step, finally the model can adapt to the trajectory prediction task in different scenes; (5) Pedestrian trajectory prediction model training: a sample weighted reinforcement training strategy is adopted to strengthen the learning of complex trajectories; a weight is assigned to each training sample, and the weights are dynamically updated during the training process; for those samples with poor pedestrian trajectory prediction accuracy, the training weight is increased, so that more attention is paid to them in subsequent training, thereby improving the prediction accuracy of the model for complex trajectory samples; The process of step (1) is as follows: (1.1) Collect image information in the scene, extract time step The historical observation position as the basic position information of pedestrian trajectory prediction, that is: ； Wherein, T obs According to the actual situation, The basic position information of the ith pedestrian is represented, including the two-dimensional plane position information of the pedestrian at the time step The two-dimensional plane position information of the pedestrian is as follows: ； wherein, and respectively represent the two-dimensional horizontal and vertical position coordinate information of the i-th pedestrian at time step t. (1.2) Collect and extract the trajectory position information of the i-th pedestrian at time step is : ； ； wherein T pre According to the actual situation and Tobs is set, and respectively represent the two-dimensional horizontal and vertical position coordinate information of the i-th pedestrian at time step t . The process of step (2) is as follows: (2.1) Node representation design for pedestrian trajectory prediction, the motion trajectory of pedestrians is described by a graph in the social attention module, and the interaction between pedestrians at each time step is abstractly defined as a graph representation : ； in, It is the vertex set of pedestrians in the graph, representing the... At any given time, there are all pedestrians, i.e., the vertex set of N pedestrians; The value is the value of the i-th pedestrian at time i. Observed coordinates edge set The relationship between the i-th and j-th pedestrians is represented by constructing a relationship edge between each pair of pedestrians; and the feature set... Defined as: ； wherein, is a weight, is a ReLU activation function; (2.2) Compute social soft attention function, the interaction between pedestrians includes walking towards each other, walking away from each other and walking side by side; a social soft attention function is proposed to construct the adjacency matrix , denotes the attention weight between the ith and jth pedestrians, which is calculated as follows: ； wherein, and are the velocity vectors of the pedestrian i and the pedestrian j at time respectively; and denote the angle between the velocity vectors and of the pedestrian i and the pedestrian j respectively, and denote the position coordinates of the pedestrian i and the pedestrian j at time respectively. In addition, when i.e. two row nodes are the same, the self-attention value of the node is set to denotes the self-attention value of the node, and the value of the element in the adjacency matrix is set to 0.1, the maximum value of the element ① When two pedestrians are walking towards each other, the angle formed by their velocity vectors. and All angles are acute; at this time and All are greater than zero. If the calculation result is greater than 0, it indicates that there is a significant influence between the two people, and both pedestrians need to pay attention to each other to avoid head-on collisions. ② When pedestrians are walking in opposite directions, the angle formed by their velocity vectors. and All angles are obtuse; at this time and All are less than zero. The calculation result is limited to 0; this indicates that there is no direct mutual influence between the pedestrians leaving in opposite directions. ③ When the two pedestrians walk side by side, i.e. pedestrian i is in front of pedestrian j, the included angle α formed by the speed vectors of the two pedestrians is obtuse, and β is acute; if the moving speed of pedestrian j is significantly higher than that of pedestrian i, there is a possibility of future collision, and the attention function output is greater than 0, indicating a potential collision risk; on the contrary, if the speed of pedestrian j is lower than that of pedestrian i, there is no collision risk, and the output of the attention function will be set to 0; The size of the collision risk adopts a natural function form to represent the nonlinear exponential change of the pedestrians with the influence of the Euclidean distance and the relative speed; (2.3) Trajectory domain feature output, further extract spatio-temporal features by using graph convolution network GCN, at time step graph convolution operation at time step ； ； wherein is the adjacency matrix at time step , is the submatrix of the graph, is the identity matrix, and time step is stacked into: ​ ； The second matrix is stacked as: ； Meanwhile, the feature matrix of the first layer in the GCN hierarchy is stacked as: layer. ； Thus, the computation of the GCN output through the i-th layer is layer is ​ ； wherein, is the number of layers of the GCN network, is the number of layers of the GCN network, and the feature representation obtained after the last layer of the GCN network is defined as wherein, is the number of layers of the GCN network, and is the data dimension of the extracted feature.

2. The multi-scene adaptive pedestrian trajectory prediction method based on attention mechanism according to claim 1, characterized in that, The process of step (3) is as follows: (3.1) Source trajectory and target trajectory information acquisition, collect the position information of the pedestrian trajectory under the source scene and the target scene, and respectively acquire the graph representation of the pedestrian trajectory under the source scene and the graph representation of the pedestrian trajectory under the target scene ; (3.2) obtaining the feature representation of the source trajectory and the feature representation of the target trajectory as the input of the social attention module designed in step (2), respectively, and obtaining the feature representation of the source trajectory and the feature representation of the target trajectory , which contains the spatiotemporal information of each pedestrian in different trajectory domains; (3.3) Trajectory domain attention score calculation, since the same feature vector may perform differently in different trajectory domains, in order to determine the representativeness of the feature vector of the pedestrian in the trajectory domain, the relative correlation between the feature vector and the trajectory domain is learned by using the attention mechanism, and the attention scores of the i th pedestrian in the source trajectory domain and the target trajectory domain are respectively and , for measuring the importance of each pedestrian feature in a specific trajectory domain, and the calculation is as follows: ； ； wherein, and are learnable weight parameters, is a tangent function, and are the position features of the i-th pedestrian in the source trajectory domain and the target trajectory domain, respectively. (3.4) pedestrian trajectory contrast difference features computing, based on the attention score values computed at step (3.3) and computing final feature representations for the source trajectory domain and the target trajectory domain, respectively and : ； ； The individual-level difference between the two trajectory domain final feature representations is then calculated, and the calculated difference is mapped to a feature vector Data features with the same dimension, i.e.: ； wherein, is a learnable weight matrix, is a fully connected layer, finally, the calculated pedestrian trajectory contrast difference features are integrated into the backbone network model to realize comprehensive feature analysis of pedestrian trajectory prediction.

3. The multi-scene adaptive pedestrian trajectory prediction method based on attention mechanism according to claim 1, characterized in that, In step (4), the processing process of the pedestrian trajectory prediction backbone network model is as follows: (4.1) For pedestrian historical trajectory positions First, the position of each historical trajectory position is encoded by a neural network embedding layer to obtain encoded feature data : ； wherein, is an embedding layer; represents a weight coefficient of the embedding layer, which is stepwise iteratively optimized by network training; and , ; (4.2) Due to the difference in the motion patterns of pedestrians in different scenarios, the pedestrian trajectories obtained in step (3) in different scenarios are compared with the difference features The backbone network is introduced to solve the shift problem of different trajectory domain comparison, that is, the fused features are: ； wherein represents a data lane stitching operation; (4.3) The encoded position feature data is projected into the query matrix, the key matrix, and the value matrix, and for the i-th pedestrian sequence, the following is calculated , , The matrix is: ； in, , and The corresponding query, key, and value functions are respectively. , These represent the i-th pedestrian in the i-th position. The query, key, and value matrix under each attention head, and the total number of multi-head attention heads is... ,but ; (4.4) Attention coefficients of multi-head attention and multi-head attention-based feature vector computation: ； ； ； wherein, is a softmax normalization function, is a normalization factor; is the feature computation result of the th pedestrian at the th attention head; is the feature vector result of the th pedestrian after multi-head attention processing, containing time dependency, is a data channel concatenation operation; Therefore, the entire data operation based on the multi-head attention mechanism is as follows: ； wherein, is a multi-head attention operation function; (4.5) Construct a trajectory decoder based on a recurrent neural network, in order to enhance the robustness of pedestrian trajectory sequence prediction, a random vector subject to a Gaussian distribution is integrated into the trajectory decoder, then the input feature vector of the trajectory decoder is: ​ ； Secondly, based on the multi-head attention mechanism and the recurrent neural network, a trajectory decoder is constructed, and a fully connected layer and a ReLU activation function are combined to finally obtain the prediction result of the pedestrian trajectory position, i.e. ； wherein, is the trajectory prediction result for the i-th pedestrian, is a trajectory decoder processing function, is a network weight coefficient of the trajectory decoder.

4. The multi-scene adaptive pedestrian trajectory prediction method based on attention mechanism of claim 1, wherein, The process of step (5) is as follows: (5.1) Trajectory prediction loss function design, the goal of which is to predict the pedestrian's steps in the future. The trajectory position is determined, therefore, the loss function is defined. for: ； in, and These represent pedestrian i at time step [i], respectively. The true position vector and the predicted position vector, To calculate the mean square error, Total number of employees; (5.2) Parameter initialization, total number in training data set Initialize initial weights of all samples: ； wherein, and initializing the initial network parameters of the pedestrian trajectory prediction model as ; (5.3) The pedestrian trajectory prediction model is iteratively trained, and the loss value of each sample in the training data set is calculated: ； wherein, denotes the initialized weight value of the mth training sample, denotes the direct loss value of the mth training sample calculated by step (5.1), denotes the weighted loss value of the mth sample; (5.4) Model parameter update, compute average loss for 1st iteration : ； Based on the loss calculation result , the model parameter optimization is performed by using a gradient descent method, and the parameters of the pedestrian trajectory prediction model after the first iteration are obtained ; (5.5) updating the sample training weights according to the loss function calculation results of each sample in the first iteration updating the weighted values of each set of training sample data, namely: ； wherein, is the training weight value of the mth sample after the 1st iteration update, is a preset learning rate of the sample training weight value, and ; is a softmax function; (5.6) repeating steps (5.3) to (5.5) for a total of iterations times, respectively, to obtain the total loss calculation value of the iterations , and the network model parameters of the pedestrian trajectory prediction model ; (5.7) obtaining the final pedestrian trajectory prediction model by using the model parameter fusion method, first obtaining the fusion factor of each model in the secondary trained model : ； wherein, is the fusion factor of the pedestrian trajectory prediction model obtained by the th iteration training, is the total loss value of the th iteration training, and ; (5.8) fusing factors based on weighting and fusing parameters of the individual pedestrian trajectory prediction models: ； wherein, is the parameter of the fused pedestrian trajectory prediction model, is the parameter of the model at the kth iteration, and finally is applied to the network parameters of the pedestrian trajectory prediction model and is taken as the final trained pedestrian trajectory prediction model.

Images