Pedestrian trajectory prediction method of multi-scale spatial-temporal feature fusion network

Abstract

The invention provides a pedestrian trajectory prediction method based on a multi-scale spatial-temporal feature fusion network, and belongs to the technical field of computer vision. The problem of how to effectively learn complex spatio-temporal characteristics and interrelation thereof in pedestrian trajectory prediction is solved. According to the technical scheme, the method comprises the following steps of S1, acquiring data of a data set; s2, using a self-attention mechanism to learn time dependence characteristics of each pedestrian; s3, converting the pedestrian coordinates into a space diagram; s4, cross-space-time interaction features are extracted through multi-head self-attention; s5, time features, space features and cross-space-time interaction features are acquired; and S6, performing multi-track prediction on the fused spatial-temporal features through a time extrapolation convolutional neural network (MTCN). The method has the beneficial effect that multi-track prediction is realized through the multi-layer time extrapolation convolutional neural network MTCN.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a pedestrian trajectory prediction method based on a multi-scale spatiotemporal feature fusion network. Background Art

[0002] Pedestrian trajectory prediction is a key technology in the fields of autonomous driving, intelligent monitoring, and traffic analysis. Accurately predicting the future movement trajectory of pedestrians is crucial to the safety and reliability of the system. With the acceleration of urbanization and the increase in traffic flow, the demand for pedestrian trajectory prediction is also increasing, which in turn promotes the development of related technologies. In recent years, with the rapid advancement of deep learning technology, many data-driven models have been proposed to improve the accuracy and effect of trajectory prediction.

[0003] Existing pedestrian trajectory prediction methods mainly use deep learning technology to predict the future movement trajectory of pedestrians. The following are some common technologies used for pedestrian trajectory prediction: based on long short-term memory network, based on generative adversarial network, based on spatiotemporal graph convolutional neural network, based on transformer model, and cross fusion model.

[0004] However, the models based on the above technologies still have certain technical problems. First, LSTM is prone to gradient vanishing or exploding when dealing with long-term dependencies. In addition, its modeling ability for spatial information is weak and it cannot effectively capture the spatial interactions between pedestrians and complex environmental information. In addition, since the LSTM model needs to iterate the data of each time step step by step, its computational efficiency on large-scale data sets is low and it cannot meet the real-time requirements well.

[0005] The pedestrian trajectory prediction model based on the generative adversarial network also faces some technical problems. First, it is prone to mode collapse during the training process, that is, the generator only generates certain trajectory patterns and ignores other possible trajectories, resulting in insufficient diversity in the generated results. Second, the training process of GAN is unstable, and a delicate balance is required between the generator and the discriminator, otherwise it will make it difficult for the model to converge or the generated results will be poor.

[0006] The application of spatiotemporal graph convolutional neural networks in pedestrian trajectory prediction also has technical limitations. The model requires multiple convolution operations on the spatiotemporal graph, which has high computational complexity and memory consumption, especially on large-scale datasets, affecting the scalability of the model. In addition, ST-GCN relies on the adjacency matrix to represent spatial relationships, but this approach may be difficult to effectively model indirect spatial interactions between pedestrians in complex scenes.

[0007] The Transformer-based pedestrian trajectory prediction model performs well in capturing long-term dependencies, but there are still some problems. The computational complexity of the Transformer is very high when processing long time series, especially when the self-attention mechanism needs to process a large number of time steps, the consumption of computing resources is significant. In addition, the standard Transformer mainly focuses on time series modeling, and its ability to capture spatial information and interactions between pedestrians is insufficient, resulting in limited performance of the model in complex spatial scenes.

[0008] The pedestrian trajectory prediction model based on cross-fusion captures spatiotemporal information by fusing multiple data streams, but it also faces some technical problems. First, the cross-fusion model usually needs to process multiple different information streams (such as time, space, individual behavior, etc.), which significantly increases the complexity and computational burden of the model, affecting its efficiency and feasibility of practical application. In addition, how to effectively fuse information at different levels to avoid information redundancy and information loss is a difficult problem that the cross-fusion model needs to solve. In some scenarios, the cross-fusion method may perform well, but its generalization ability is often poor. When applied to new environments or scenarios, the performance of the model may drop significantly, thus limiting its effectiveness in a wide range of applications.

[0009] In summary, these pedestrian trajectory prediction models have certain advantages in their respective fields, but they still face technical challenges such as low computational efficiency, insufficient spatiotemporal modeling capabilities, high model complexity, and weak generalization capabilities. How to reduce computational complexity while ensuring model prediction accuracy and improve the modeling capabilities of pedestrian interactions and spatiotemporal information are key issues that need to be urgently addressed in the field of pedestrian trajectory prediction. Summary of the invention

[0010] The purpose of the present invention is to provide a pedestrian trajectory prediction method based on a multi-scale spatiotemporal feature fusion network, which can capture the time-dependent characteristics and spatial interaction characteristics in pedestrian motion and effectively handle complex spatiotemporal dynamic changes; at the same time, through the sparse gating mechanism, the model can focus on key interaction relationships, thereby improving computational efficiency; finally, multi-trajectory prediction is achieved through a multi-layer time extrapolation convolutional neural network MTCN, and the two-dimensional Gaussian distribution parameters of future trajectories are jointly estimated by combining multi-layer prediction features. The network model is trained using the minimized negative log-likelihood loss function, which can better reflect the uncertainty of pedestrian motion and provide more diverse and accurate predictions.

[0011] The inventive concept of the present invention is to comprehensively utilize the time and space feature extraction technology, and realize efficient prediction of pedestrian trajectories through the time extrapolation convolutional neural network MTCN; first, on the time scale, the position encoding of Transformer is used to embed the time information of the historical trajectory data, and the time feature is combined with the position feature. On the spatial scale, the pedestrian coordinates are converted into a spatial graph, and the spatial feature weights are calculated using the self-attention mechanism, and then the weight matrix is ​​sparsely and normalized; at the same time, on the spatiotemporal scale, the spatiotemporal correlation features are extracted through the multi-head attention mechanism, and the weights are sparsely and normalized in combination with the sparse gating mechanism to obtain cross-spatiotemporal features; finally, the gated network module of adaptive feature fusion is used to dynamically integrate the time, space and spatiotemporal correlation features. Afterwards, the efficient prediction of pedestrian trajectories is realized through the time extrapolation convolutional neural network MTCN. This comprehensive method effectively improves the pedestrian trajectory prediction effect in complex scenarios.

[0012] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a pedestrian trajectory prediction model of a multi-scale spatiotemporal feature fusion network, comprising the following steps:

[0013] S1: Get the data set data, including the location coordinates of the pedestrians at each time Trajectory data is encoded using Transformer’s positional encoding.

[0014] S2: Use self-attention mechanism to learn the temporally dependent features of each pedestrian.

[0015] S3: The pedestrian coordinates are converted into a spatial graph, and then the spatial graph and weights are linearly transformed to obtain the graph embedding matrix. The spatial interaction matrix of pedestrians in space is obtained using the self-attention mechanism. Subsequently, the weights are sparsely and normalized in combination with the sparse gating mechanism to obtain spatial features.

[0016] S4: Cross-temporal and spatial interaction features are extracted through multi-head self-attention, and the sparse gating mechanism is used to perform weight sparsification and normalization to obtain cross-temporal and spatial interaction features.

[0017] S5: Obtain temporal features, spatial features, and cross-temporal and cross-spatial interaction features, and adaptively fuse temporal features, spatial features, and cross-temporal and cross-spatial correlation features through activation functions.

[0018] S6: The fused spatiotemporal features are passed through a temporal extrapolation convolutional neural network (MTCN) and combined with multi-layer prediction features to jointly estimate the two-dimensional Gaussian distribution parameters of future trajectories. The network model is trained using the minimized negative log-likelihood loss function to achieve multi-trajectory prediction.

[0019] S11: Get the data set data, including the location coordinates of the pedestrians at each time, expressed as Where T obs is the length of the observation period, i represents an individual pedestrian, is the two-dimensional coordinate of the pedestrian at time step t.

[0020] S12: Use Transformer positional encoding to temporally encode the trajectory data. Positional encoding generates a corresponding encoding vector for each time step t to represent the temporal information, so as to keep the timing information transmitted in the Transformer encoder. Each position point is encoded by the positional encoding function. Mapping into high-dimensional space:

[0021]

[0022] X=[E p1 ,E p2 ,…,E pobs ]

[0023] Among them, the PositionalEncoding function generates an embedding vector by calculating the sine and cosine functions of the time step. After encoding the trajectory points of all time steps, they are combined to form a trajectory representation matrix X, which is used as the input of subsequent modules. Among them, pos is the position index of the trajectory time step, i is the encoding dimension, and d is the hidden dimension of the model. In this way, the position of each time step in the trajectory is embedded in a high-dimensional space to facilitate the subsequent time-dependent feature extraction.

[0024] In step S2, the time-dependent features are learned through the Transformer’s self-attention mechanism, which includes the following steps:

[0025] S21: Extract the trajectory features of each pedestrian and perform linear projection, then perform self-attention operation on each pedestrian to obtain the feature matrix, which is then multiplied with the corresponding weight of each pedestrian to obtain the time feature

[0026]

[0027] Where Q m , K s are query and key matrices respectively, and d k Scaling factor for feature dimensions.

[0028] In step S3, the spatial states of all pedestrians at each time step are extracted from the input, which includes the following steps:

[0029] S31: Convert pedestrian coordinates into spatial graph Gt =(V t ,E t ), V t 、E t Respectively represent the nodes and edges between pedestrians, is the observed coordinate position Indicates whether there is a connection between nodes, 1 indicates connection, and 0 indicates no connection.

[0030]

[0031] S32: To better handle the spatial relationship of pedestrians, the spatial graph and weights are first linearly transformed to obtain the graph embedding matrix, and then the spatial interaction matrix W of pedestrians in space is calculated using the self-attention mechanism. s :

[0032]

[0033] in represents linear transformation, W s E , W s Q , W s K is the corresponding weight matrix, Q s , K s , respectively, the query and key matrices, d k is the scaling factor of the feature dimension.

[0034] S33: The calculated spatial attention matrix W s Perform sparse processing. First, sparse gating mechanism is used to reduce redundant connections. The specific sparse operation is as follows:

[0035] W s ′=Relu(W s -sigmoid(CNN(W s )))

[0036] This process uses convolutional neural networks to extract high-order information of spatial features and sparses the attention weights through a combination of ReLU and sigmoid.

[0037] S34: The spatial map matrix is ​​normalized by the normSoftmax function to ensure the numerical stability of the calculation.

[0038] The formula is as follows:

[0039]

[0040] where ∈ is a small constant that can be ignored to ensure numerical stability.

[0041] S35: Finally, the normalized weight matrix and pedestrian node feature V s

[0042] Multiply them together to get the final spatial features:

[0043]

[0044] Where V s is the weight matrix.

[0045] In step S4, cross-temporal and spatial correlation features are extracted through multi-head attention, which includes the following steps:

[0046] S41: For the extraction of spatiotemporal interaction features, we first use the transformer’s multi-head attention mechanism to construct the spatial and temporal interaction attention weight matrix W m .

[0047]

[0048] S42: The calculated cross-temporal and spatial attention matrix W m Perform sparse processing. First, sparse gating mechanism is used to reduce redundant connections. The specific sparse operation is as follows:

[0049] W m ′=Relu(W m -sigmoid(CNN(W m )))

[0050] This process uses convolutional neural networks to extract high-order information of cross-temporal and spatial features, and uses a combination of ReLU and sigmoid to sparse the attention weights to extract the most meaningful cross-temporal and spatial connections.

[0051] S43: Sparse weight matrix W m ′ is normalized to ensure numerical stability during the calculation process. The normalization formula is as follows:

[0052]

[0053]

[0054] The normalized sparse attention weight matrix is ​​combined with the feature matrix V m Multiply them together to get the cross-temporal and spatial feature representation:

[0055]

[0056] The adaptive fusion module contains the following steps:

[0057] S51: Obtaining time feature F t , spatial feature F s and the cross-temporal and spatial correlation features F m , for each branch feature, calculate the intermediate feature F e And the gate weight G, the intermediate feature calculation formula is:

[0058]

[0059] The weight calculation formula is:

[0060]

[0061] Among them, W F and W G is the learnable weight of the linear projection function, and different linear projection weights are used for temporal features, spatial features, and cross-temporal and spatial correlation features.

[0062] S52: The weight G of each branch is adjusted by the Softmax function t ,G s ,G m Normalize:

[0063]

[0064] S53: Multiply the intermediate features of each branch by the normalized weights, and then add them together to obtain the final fusion feature F:

[0065]

[0066] Among them, F et 、F es 、F em It is the intermediate transition feature when the time, space, and cross-time and space features pass through the adaptive fusion module.

[0067] S6: Input the spatiotemporal features into the temporal extrapolation convolutional neural network (MTCN), combine the multi-layer prediction features to jointly estimate the two-dimensional Gaussian distribution parameters of the future trajectory, and train the network model using the minimized negative log-likelihood loss function, which includes the following steps:

[0068] S61: Temporal Extrapolation Convolutional Neural Network (MTCN) captures the temporal dependency in spatiotemporal graph convolution features through multi-layer convolution operations. The first temporal convolution layer receives graph convolution features, performs preliminary convolution extraction of spatiotemporal features, and gradually stacks convolution layers. The features after each convolution layer are processed by the ReLU activation function.

[0069] S62: Finally, a fully connected layer (Linear Layer) is used to map the final convolutional features to the predicted values ​​of the pedestrian's future trajectory coordinates.

[0070] S63: Obtain the bivariate Gaussian distribution parameters of the predicted trajectory through MTCN. Then, the network model is trained by minimizing the negative log-likelihood loss function.

[0071]

[0072] in is the predicted mean, is the prediction standard deviation, is the predicted correlation coefficient.

[0073] S64: This model uses the Adam optimizer to optimize parameters, with a learning rate of 0.001, a batch size of 32, and 200 training rounds. The ETH and UCY datasets are used for training and testing. The evaluation indicators are ADE (AverageDisplacement Error) and FDE (FinalDisplacement Error) to evaluate the accuracy of the predicted trajectory:

[0074]

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] 1. Multi-scale spatiotemporal feature fusion module

[0077] One of the core innovations of the present invention is the introduction of a multi-scale spatiotemporal feature fusion module. This module combines time, space, and cross-spatiotemporal correlation features to enhance the ability to predict pedestrian trajectories. Traditional trajectory prediction models often deal with time and space features separately, lacking effective modeling of the interaction between the two. The present invention, by adaptively fusing these features, not only improves the model's adaptability to complex dynamic scenes, but also can more accurately capture the subtle interactions and dependencies between spatiotemporal features. This technology can effectively improve the prediction effect of traditional methods in multi-target scenarios, especially in the case of complex interactions between pedestrians, and can provide more accurate trajectory predictions.

[0078] 2. Sparse gating mechanism

[0079] In order to avoid redundant connections between nodes in the spatial graph, the present invention introduces a sparse gating mechanism to perform sparse processing on the spatial and cross-temporal interaction matrices. By using a convolutional neural network to extract high-order spatial and spatiotemporal cross features, and combining ReLU and sigmoid activation functions, the model can filter out unimportant connections and extract the most useful spatial and cross-temporal interaction information for trajectory prediction. This technology can effectively reduce the complexity of calculations in scenes with complex spatial relationships, while avoiding redundant parts in spatiotemporal features, greatly improving the real-time performance and efficiency of the model. In addition, normalization after sparseness ensures the stability of numerical calculations, avoids the problem of gradient explosion or gradient disappearance that may occur during training, and thus further improves the training efficiency and robustness of the model.

[0080] 3. Adaptive fusion mechanism

[0081] The present invention optimizes the fusion process of temporal features, spatial features and cross-temporal and spatial features through an adaptive fusion mechanism. By calculating the gated weight of each branch feature and normalizing it using the Softmax function, the model can dynamically adjust the weights of each branch according to the importance of different features. This adaptive fusion mechanism can not only automatically select the most representative features for prediction, but also flexibly adjust the fusion strategy according to scene changes, thereby improving the generalization ability and prediction accuracy of the model. Compared with the traditional static feature fusion method, this technology can better cope with diverse pedestrian trajectory prediction tasks.

[0082] 4. Temporal Extrapolation Convolutional Neural Network (MTCN) Module

[0083] The present invention adopts a temporal extrapolation convolutional neural network (MTCN) to jointly estimate the spatiotemporal features to predict the future pedestrian trajectories. Through multi-layer convolution operations, MTCN can extract and integrate the temporal dependencies in the spatiotemporal graph convolution features, and output the prediction results through the final fully connected layer. The use of MTCN effectively solves the problem that traditional methods are prone to failure on time series data, especially in multi-target dynamic scenes, and can accurately predict the future positions of pedestrians.

[0084] In summary, the innovations of the present invention in spatiotemporal feature modeling, feature fusion, sparsification and normalization, and finally multi-trajectory prediction have significantly improved the accuracy, efficiency and stability of pedestrian trajectory prediction, solved many challenges in the prior art, and provided an efficient and accurate solution for the field of pedestrian trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0086] Figure 1 It is the overall structure flow chart of the present invention.

[0087] Figure 2 This is a diagram of the space-time fusion model used in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0089] Example 1

[0090] See also Figure 1 to Figure 2 The technical solution provided by this embodiment is: a pedestrian trajectory prediction method based on spatiotemporal feature fusion, comprising the following steps:

[0091] S1: Get the data set data, including the location coordinates of the pedestrians at each time Trajectory data is encoded using Transformer’s positional encoding.

[0092] The specific steps include:

[0093] S11: The trajectory data of pedestrians in the dataset in the past time period, including the position coordinates of pedestrians at each time, expressed as Where T obs is the length of the observation period, i represents an individual pedestrian, is the two-dimensional coordinate of the pedestrian at time step t.

[0094] S12: Use Transformer's positional encoding to temporally encode the trajectory data. Positional encoding generates a corresponding encoding vector for each time step t to represent the temporal information, so as to keep the timing information transmitted in the Transformer encoder. The positional encoding formula is:

[0095]

[0096] Among them, pos is the position index of the trajectory time step, i is the encoding dimension, and d is the hidden dimension of the model. In this way, the position of each time step in the trajectory is embedded in a high-dimensional space to facilitate subsequent time-dependent feature extraction.

[0097] Use the self-attention mechanism to learn the time-dependent features of each pedestrian:

[0098] S21: The input time series data is reshaped into a tensor of shape [T, N*sample_num, d_model], where T is the time step, N is the number of agents, sample_num is the number of samples, and d_model is the feature dimension. Then a self-attention operation is performed on each pedestrian to obtain a feature matrix, which is multiplied with the weights to obtain the temporal features. The number of multi-head attention heads used is 8 and the feature dimension is 512.

[0099]

[0100] S3: Extract the spatial state of all pedestrians at each time step from the input. The pedestrian coordinates are converted into a spatial graph, and then a sparse weight matrix is ​​extracted using multi-head self-attention. After that, the sparse gating mechanism is used to sparse and normalize it to obtain the spatial features. It includes the following steps:

[0101] S31: Convert pedestrian coordinates into spatial graph G t ={(V t ,E t ). G t =(V t ,E t ), V t 、E t Respectively represent the nodes and edges between pedestrians, is the observed coordinate position Indicates whether there is a connection between nodes, 1 indicates connection, and 0 indicates no connection. Reshape the input data into a tensor of shape [N, T*sample_num, d_model].

[0102] S32: To better handle the spatial relationship of pedestrians, the spatial graph and weights are first linearly transformed to obtain the graph embedding matrix, and then the spatial interaction matrix W of pedestrians in space is calculated using the self-attention mechanism. s :

[0103]

[0104] in represents linear transformation, W s E , W s Q , W s K is the corresponding weight matrix, Q s , K s , respectively, the query and key matrices, d k is the scaling factor of the feature dimension.

[0105] S33: The calculated spatial attention matrix W s Perform sparse processing. First, sparse gating mechanism is used to perform sparse processing to reduce redundant connections. Multiple convolutional layers are mainly used to extract high-order features to further enhance the model's ability to model complex relationships. Sparse depth: 3 convolutional layers, filter size is 3×3, and the number of channels in each layer is 64, 128, and 256 respectively. The specific sparse operations are as follows:

[0106] W s ′=Relu(W s -sigmoid(CNN(W s )))

[0107] This process uses convolutional neural networks to extract high-order information of spatial features and sparses the attention weights through a combination of ReLU and sigmoid.

[0108] S34: The spatial map matrix is ​​normalized by the normSoftmax function to ensure the numerical stability of the calculation. The formula is as follows:

[0109]

[0110] where ∈ is a small constant that can be ignored to ensure numerical stability.

[0111] S35: Finally, the normalized weight matrix and pedestrian node feature V s

[0112] Multiply them together to get the final spatial features:

[0113]

[0114] Where V m is the weight matrix.

[0115] S4: The cross-temporal and spatial correlation features are extracted through multi-head self-attention, and the weights are sparsely and normalized by combining the sparse gating mechanism to obtain cross-temporal and spatial features. It includes the following steps:

[0116] S41: Reshape the time series data into a tensor of shape [T*N, sample_num, d_model] for spatiotemporal interaction attention calculation. For the extraction of cross-spatiotemporal features, first construct the spatiotemporal interaction attention weight matrix W m , calculating cross-temporal and spatial weights through the self-attention mechanism.

[0117]

[0118] Where Q m , K s, V m are query, key and value matrices respectively, d k Scaling factor for feature dimensions.

[0119] S42: The calculated cross-temporal and spatial attention matrix W m Perform sparse processing. First, sparse gating mechanism is used to reduce redundant connections. The specific sparse operation is as follows:

[0120] W m ′=Relu(W m -sigmoid(CNN(W m )))

[0121] This process uses convolutional neural networks to extract high-order information of cross-temporal and spatial features, and uses a combination of ReLU and sigmoid to sparse the attention weights to extract the most meaningful cross-temporal and spatial connections.

[0122] S43: Sparse weight matrix W m ′ is normalized to ensure numerical stability during the calculation process. The normalization formula is as follows:

[0123]

[0124]

[0125] The normalized sparse attention weight matrix is ​​combined with the feature matrix V m Multiply them together to get the cross-temporal and spatial feature representation:

[0126]

[0127] S5: The adaptive fusion module includes the following steps:

[0128] S51: Obtaining time feature F t , spatial feature F s and the cross-temporal and spatial correlation features F m , for each branch feature, calculate the intermediate feature F e And the gate weight G, the intermediate feature calculation formula is:

[0129]

[0130] The weight calculation formula is:

[0131]

[0132] Among them, W F and W Gis the learnable weight of the linear projection function, and different linear projection weights are used for temporal features, spatial features, and cross-temporal and spatial correlation features.

[0133] S52: The weight G of each branch is adjusted by the Softmax function t ,G s ,G m Normalize:

[0134]

[0135] S53: Multiply the intermediate features of each branch by the normalized weights, and then add them together to obtain the final fusion feature:

[0136]

[0137] S6 By introducing Gaussian latent variables into the CVAE framework, this embodiment realizes multimodal prediction and generates diverse future trajectories. It includes the following steps:

[0138] S61: The multi-layer temporal extrapolation convolutional neural network MTCN module defines 4 layers of TCN, each layer accepts the output of the previous layer as input, and captures the temporal dependency in the spatiotemporal graph convolution features through multi-layer convolution operations. The first layer of temporal convolution layer receives the graph convolution features, performs preliminary convolution extraction of spatiotemporal features, and gradually stacks the convolution layers. The output of each layer of TCN is fused with the input of the previous layer through residual connection, and the features after each layer of convolution are processed by ReLU activation function;

[0139] S62: Finally, the final convolutional features are mapped to a predicted trajectory through a fully connected layer, which is a sequence of positions in the future. Before prediction, the features output by TCN are subjected to torch.mean operation to reduce them in the time dimension to ensure that the feature dimension is consistent with the expected output.

[0140] S63: Obtain the bivariate Gaussian distribution parameters of the predicted trajectory through a multi-layer temporal extrapolation convolutional neural network (MTCN), including the predicted mean t, the predicted standard deviation t, and the predicted correlation coefficient t. Finally, the network model is trained by minimizing the negative log-likelihood loss function;

[0141]

[0142] in is the predicted mean, is the prediction standard deviation, is the predicted correlation coefficient.

[0143] S64: This model uses the Adam optimizer to optimize parameters, with a learning rate of 0.001, a batch size of 32, and 200 training rounds. The ETH and UCY datasets are used for training and testing. The evaluation indicators are ADE (AverageDisplacement Error) and FDE (FinalDisplacement Error) to evaluate the accuracy of the predicted trajectory:

[0144]

[0145] Finally, the model of this embodiment is compared with the most advanced model in recent years in an ADE / FDE comparative test. The experimental results are shown in Table 1 below. It can be seen from the data in Table 1 that our model has achieved the best performance overall and is superior to other models.

[0146] Table 1 Comparison of ADE / FDE between the proposed model and other models in ETH and UCY datasets

[0147]

[0148] Example 2

[0149] On the basis of Example 1, in order to further verify the feasibility of the technical solution of the present invention and its superiority in index performance, Example 2 compares this model with the most advanced model in recent years. ADE / FDE comparison test, different from Example 1, the data set of Example 2 is SDD, SDD (Stanford Drone Dataset) is a pedestrian trajectory prediction data set released by Stanford University, which specifically records the trajectories of pedestrians, vehicles, bicycles and other moving targets in high-density traffic scenes on campus captured from the perspective of drones. The data set is used to analyze the motion behavior of various objects in complex environments, especially the interaction between pedestrians and between pedestrians and other objects. The environment of the SDD data set is complex, including complex interactive behaviors between pedestrians and between pedestrians and other objects, while ETH and UCY are mainly shot from a ground perspective, mainly focusing on urban open environments and campus blocks, with less interaction between pedestrians and simpler scenes.

[0150] From the results in Table 2, we can see that our method significantly outperforms all existing models in the table, with ADE and FDE of 9.31 and 14.56 respectively on the SDD dataset. In terms of ADE, it is reduced by about 15% compared to SGTN, and in terms of FDE, it is reduced by about 5.6% compared to STAR. This shows that our model strengthens the spatiotemporal dependencies that have a decisive influence on the prediction results by more finely modeling trajectory features at different time scales, especially in more complex pedestrian behavior patterns, and adapts to the dynamic changes of different scenarios. In addition, the reduction in FDE is also due to the fact that the multi-layer convolution of MTCN can effectively capture the temporal relationship in the historical trajectory, which enables the model to achieve improvements in predictions of changing scenarios and long time steps, and accurately predict future trajectories.

[0151] Table 2 Comparison of ADE / FDE between the proposed model and other models in the SDD dataset

[0152]

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A pedestrian trajectory prediction method based on a multi-scale spatiotemporal feature fusion network, characterized in that: The following steps are involved: S1: Get the data set data, including the location coordinates of the pedestrians at each time in is the two-dimensional coordinate of the pedestrian at time step t, i represents the pedestrian individual, T obs To observe the length of the time period, the trajectory data is encoded using Transformer's position encoding; S2: Use self-attention mechanism to learn the time-dependent features of each pedestrian; S3: The pedestrian coordinates are converted into a spatial graph, and then the spatial graph and weights are linearly transformed to obtain a graph embedding matrix. The spatial interaction matrix of pedestrians in space is obtained by using the self-attention mechanism. Subsequently, the weights are sparsely and normalized in combination with the sparse gating mechanism to obtain spatial features. S4: The cross-temporal and spatial interaction features are extracted through multi-head self-attention, and the weights are sparsely and normalized by combining the sparse gating mechanism to obtain the cross-temporal and spatial interaction features; S5: Obtain temporal features, spatial features and cross-temporal and spatial interaction features, and adaptively fuse temporal features, spatial features and cross-temporal and spatial correlation features through activation functions; S6: The fused spatiotemporal features are passed through the time extrapolation convolutional neural network (MTCN) and combined with multi-layer prediction features to jointly estimate the two-dimensional Gaussian distribution parameters of future trajectories. The network model is trained using the minimized negative log-likelihood loss function to achieve multi-trajectory prediction.

2. The pedestrian trajectory prediction method based on the multi-scale spatiotemporal feature fusion network according to claim 1 is characterized in that: The step S1 comprises the following steps: S11: Get the data set data, including the location coordinates of the pedestrians at each time, expressed as S12: Use Transformer's position encoding to time-encode the trajectory data. Position encoding generates a corresponding encoding vector for each time step t to represent the time information, so as to keep the transmission of the timing information in the Transformer encoder. Each position point is encoded by the position encoding function. Mapping into high-dimensional space: Among them, the PositionalEncoding function generates an embedding vector by calculating the sine and cosine functions of the time step, encodes the trajectory points of all time steps, and combines them to form a trajectory representation matrix X for the input of subsequent modules, where pos is the position index of the trajectory time step, i is the encoding dimension, and d is the hidden dimension of the model.

3. The pedestrian trajectory prediction method based on the multi-scale spatiotemporal feature fusion network according to claim 1 is characterized in that: In step S2, the time-dependent features are learned through the self-attention mechanism of Transformer, which includes the following steps: S21: Extract the trajectory features of each pedestrian and perform linear projection, then perform self-attention operation on each pedestrian to obtain the feature matrix, which is then multiplied with the corresponding weight of each pedestrian to obtain the time feature Where Q m , K s are query and key matrices respectively, and d k Scaling factor for feature dimensions.

4. The pedestrian trajectory prediction method based on the multi-scale spatiotemporal feature fusion network according to claim 1 is characterized in that: In step S3, the spatial states of all pedestrians at each time step are extracted from the input, including the following steps: S31: Convert pedestrian coordinates into spatial graph G t =(V t ,E t ), V t 、E t Respectively represent the nodes and edges between pedestrians, is the observed coordinate position Indicates whether there is a connection between nodes, 1 indicates connection, and 0 indicates no connection; S32: First, the spatial graph and weights are linearly transformed to obtain the graph embedding matrix, and then the self-attention mechanism is used to calculate the spatial interaction matrix W of pedestrians in space s : in represents a linear transformation, is the corresponding weight matrix, Q s , K s , respectively, the query and key matrices, d k is the scaling factor of the feature dimension; S33: The calculated spatial attention matrix W s Perform sparse processing and use sparse gating mechanism to reduce redundant connections. The specific sparse operations are as follows: IN s ′=Relu(W s -sigmoid(CNN(W s ))) This process uses convolutional neural networks to extract high-order information of spatial features and sparses the attention weights through a combination of activation functions ReLU and Sigmoid. S34: The spatial map matrix is ​​normalized by the normSoftmax function to ensure the numerical stability of the calculation. The formula is as follows: Where,∈ is a small constant that is ignored to ensure numerical stability; S35: Finally, the normalized weight matrix and pedestrian node feature V s Multiply them together to get the final spatial features: Among them, V m is the weight matrix.

5. The pedestrian trajectory prediction method based on the multi-scale spatiotemporal feature fusion network according to claim 1 is characterized in that: In step S4, cross-temporal and spatial correlation features are extracted through a multi-head attention mechanism, which includes the following steps: S41: For the extraction of spatiotemporal interaction features, we first use the transformer’s multi-head attention mechanism to construct the spatial and temporal interaction attention weight matrix W m ; S42: The calculated cross-temporal and spatial attention matrix W m For sparse processing, first, sparse gating mechanism is used to reduce redundant connections. The specific sparse operation is as follows: IN m ′=Relu(W m -sigmoid(CNN(W m ))) This process uses convolutional neural networks to extract high-order information of cross-temporal and spatial features, and uses a combination of ReLU and sigmoid to sparse the attention weights to extract the most meaningful cross-temporal and spatial connections; S43: Sparse weight matrix W m ′ is normalized to ensure numerical stability during the calculation process. The normalization formula is as follows: The normalized sparse attention weight matrix is ​​combined with the feature matrix V m Multiply them together to get the cross-temporal and spatial feature representation:

6. The pedestrian trajectory prediction method based on the multi-scale spatiotemporal feature fusion network according to claim 1 is characterized in that: The step S5 adaptive fusion module comprises the following steps: S51: Obtaining time feature F t , spatial feature F s and the cross-temporal and spatial correlation features F m , for each branch feature, calculate the intermediate feature F e And the gate weight G, the intermediate feature calculation formula is: The weight calculation formula is: Among them, W F and W G The linear projection function is a learnable weight, and different linear projection weights are used for temporal features, spatial features, and cross-temporal and spatial correlation features; S52: The weight G of each branch is adjusted by the Softmax function t ,G s ,G m Normalize: S53: Multiply the intermediate features of each branch by the normalized weights and add them together to get the final fusion features: Among them, F et 、F es 、F em It is the intermediate transition feature when the time, space, and cross-time and space features pass through the adaptive fusion module.

7. The pedestrian trajectory prediction method based on the multi-scale spatiotemporal feature fusion network according to claim 1 is characterized in that: The step S6 comprises the following steps: S61: The multi-layer time extrapolation convolutional neural network module captures the temporal dependency in the spatiotemporal graph convolution features through multi-layer convolution operations. The first layer of temporal convolution layer receives the graph convolution features, performs preliminary convolution extraction of spatiotemporal features, and gradually stacks the convolution layers. The features after each convolution layer are processed by the ReLU activation function. S62: Map the final convolutional features to the predicted values ​​of the pedestrian's future trajectory coordinates through a fully connected layer; S63: The bivariate Gaussian distribution parameters of the predicted trajectory are obtained through a multi-layer temporal extrapolation convolutional neural network (MTCN). Subsequently, the network model is trained by minimizing the negative log-likelihood loss function. in, is the predicted mean, is the prediction standard deviation, is the predicted correlation coefficient; S64: This model uses the Adam optimizer to optimize the parameters, with a learning rate of 0.001, a batch size of 32, and a training round number of 200 rounds. The ETH and UCY datasets are used for training and testing to evaluate pedestrian trajectory prediction and group behavior analysis. The dataset contains pedestrian trajectory data in different scenarios. The UCY dataset contains pedestrian trajectory data in multiple complex environments, especially in dynamic and complex urban environments, including interactions between multiple pedestrians. The evaluation indicators are ADE and FDE to evaluate the accuracy of the predicted trajectory:

Images