A pedestrian trajectory prediction method of a multi-scale spatio-temporal feature fusion network

Abstract

The application provides a kind of multi-scale spatiotemporal feature fusion network pedestrian trajectory prediction method, belong to computer vision technical field.The problem of how to effectively learn complex spatiotemporal features and their mutual relations in pedestrian trajectory prediction is solved.The technical scheme is as follows: including the following steps: S1: obtaining dataset data;S2: learning the time dependence feature of each pedestrian using self-attention mechanism;S3: converting pedestrian coordinates into a spatial graph;S4: cross-spatiotemporal interaction features are extracted using multi-head self-attention;S5: obtain time features, spatial features and cross-spatiotemporal interaction features;S6: the fused spatiotemporal features are passed through a temporal extrapolation convolutional neural network (MTCN) to achieve multi-trajectory prediction.The application has the beneficial effects of achieving multi-trajectory prediction through a multi-layer temporal extrapolation convolutional neural network (MTCN).

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a pedestrian trajectory prediction method based on a multi-scale spatio-temporal feature fusion network. BACKGROUND

[0002] In the fields of autonomous driving, intelligent surveillance, and traffic analysis, pedestrian trajectory prediction is a key technology. Accurate prediction of future pedestrian motion trajectories is crucial for the safety and reliability of systems. With the acceleration of urbanization and the increase in traffic flow, the demand for pedestrian trajectory prediction has also increased, thereby driving the development of related technologies. In recent years, with the rapid progress of deep learning technology, many data-driven models have been proposed to improve the accuracy and effectiveness of trajectory prediction.

[0003] Existing pedestrian trajectory prediction methods mainly use deep learning techniques to predict the future motion trajectories of pedestrians. Some common techniques for pedestrian trajectory prediction include LSTM-based models, GAN-based models, ST-GCN-based models, transformer-based models, and cross-fusion models.

[0004] However, models based on the above techniques still have certain technical problems. First, LSTM is prone to gradient vanishing or explosion when dealing with long-term dependencies, and its modeling ability for spatial information is weak, making it difficult to effectively capture spatial interactions between pedestrians and complex environmental information. In addition, since the LSTM model needs to process the data of each time step iteratively, its computational efficiency is low on large-scale datasets, making it difficult to meet real-time requirements.

[0005] Pedestrian trajectory prediction models based on GAN also face some technical problems. First, they are prone to mode collapse during training, i.e., the generator only generates certain trajectory patterns while ignoring other possible trajectories, resulting in insufficient diversity of generated results. Second, the training process of GAN is unstable, and fine balance between the generator and the discriminator is required, otherwise the model may not converge or the generated results may be poor.

[0006] The application of ST-GCN in pedestrian trajectory prediction also has technical limitations. This model requires multiple convolution operations on the spatio-temporal graph, especially on large-scale datasets, which has high computational complexity and memory consumption, affecting the scalability of the model. In addition, ST-GCN relies on an adjacency matrix to represent spatial relationships, but this approach may not effectively model non-direct spatial interactions between pedestrians in complex scenarios.

[0007] Transformer-based pedestrian trajectory prediction models perform well in capturing long-term dependencies, but still have some problems. The computational complexity of the Transformer is high when processing long time series, especially when the self-attention mechanism needs to process a large number of time steps, the consumption of computing resources is significant. In addition, the standard Transformer mainly focuses on time series modeling, and the ability to capture spatial information and interactions between pedestrians is insufficient, resulting in limited performance of the model in complex spatial scenarios.

[0008] Pedestrian trajectory prediction models based on cross-fusion capture spatio-temporal information by fusing multiple data streams, but also face some technical problems. First, cross-fusion models usually need to process multiple different information streams (such as time, space, individual behavior, etc.), resulting in a significant increase in model complexity and computational burden, affecting its efficiency and the feasibility of its practical application. In addition, how to effectively fuse information at different levels, avoiding information redundancy and loss, is a difficult problem that cross-fusion models need to solve. In some scenarios, the cross-fusion method may perform well, but its generalization ability is often poor, and when applied to new environments or scenarios, the performance of the model may decrease significantly, thereby limiting its effectiveness in a wide range of applications.

[0009] In summary, these pedestrian trajectory prediction models have certain advantages in their respective fields, but still face technical challenges such as low computational efficiency, insufficient spatio-temporal modeling ability, high model complexity, and weak generalization ability. How to reduce computational complexity while ensuring the accuracy of model prediction and improve the modeling ability of pedestrian interactions and spatio-temporal information is a key problem in the field of pedestrian trajectory prediction that needs to be solved. SUMMARY

[0010] The purpose of the present application is to provide a pedestrian trajectory prediction method based on a multi-scale spatio-temporal feature fusion network, which can capture the time-dependent features and spatial interaction features in pedestrian motion, effectively handle complex spatio-temporal dynamic changes; at the same time, through the sparse gating mechanism, the model can focus on key interaction relationships, improving the computational efficiency; finally, through the multi-layer time extrapolation convolutional neural network MTCN, multi-trajectory prediction is realized, combining multi-layer prediction features to jointly estimate the two-dimensional Gaussian distribution parameters of the future trajectory, using the minimum negative log-likelihood loss function to train the network model, which can better reflect the uncertainty of pedestrian motion and provide more diverse and accurate predictions.

[0011] The application idea of the application is: comprehensively utilizing time and space feature extraction technology, realizing efficient prediction of pedestrian trajectory through time extrapolation convolutional neural network MTCN; first, in the time scale, embedding time information of historical trajectory data by using position encoding of the Transformer, combining time features and position features, in the space scale, converting pedestrian coordinates into a space graph, calculating spatial feature weights by using a self-attention mechanism, and then sparsifying and normalizing the weight matrix; meanwhile, in the time-space scale, extracting time-space correlation features through a multi-head attention mechanism, and performing weight sparsification and normalization processing by combining a sparse gating mechanism, to obtain cross-time-space features; finally, dynamically integrating time, space and time-space correlation features by using a gated network module with adaptive feature fusion. Then, efficient prediction of pedestrian trajectory is realized through the time extrapolation convolutional neural network MTCN, and this comprehensive method effectively improves the pedestrian trajectory prediction effect in a complex scene.

[0012] In order to achieve the above application purpose, the technical scheme adopted by the application is specifically as follows: a pedestrian trajectory prediction model of a multi-scale space-time feature fusion network, comprising the following steps:

[0013] S1: acquiring dataset data, including position coordinates of each time step pedestrian The trajectory data is encoded by using the position encoding of the Transformer.

[0014] S2: learning time dependency features of each pedestrian by using a self-attention mechanism.

[0015] S3: converting pedestrian coordinates into a space graph, then linearly transforming the space graph and the weight to obtain a graph embedding matrix, obtaining a spatial interaction matrix of the pedestrian in the space by using a self-attention mechanism, and then performing weight sparsification and normalization processing by combining a sparse gating mechanism, to obtain spatial features.

[0016] S4: cross-time-space interaction features are extracted through a multi-head self-attention mechanism, and weight sparsification and normalization processing are performed by combining a sparse gating mechanism, to obtain cross-time-space interaction features.

[0017] S5: obtaining time features, space features and cross-time-space interaction features, and adaptively fusing the time features, space features and cross-time-space correlation features through an activation function.

[0018] S6: combining the fused space-time features through a time extrapolation convolutional neural network (MTCN), combining multi-layer prediction features to jointly estimate two-dimensional Gaussian distribution parameters of the future trajectory, using a minimum negative log-likelihood loss function to train the network model, and realizing multi-trajectory prediction.

[0019] S11: Obtain dataset data, including the position coordinates of each pedestrian at each time step, denoted as where T obs is the length of the observation period, i represents the individual pedestrian, is the two-dimensional coordinate of the pedestrian at time step t.

[0020] S12: Time-encode the trajectory data using the positional encoding of the Transformer. The positional encoding generates a corresponding encoding vector for each time step t, representing the time information, to maintain the transmission of time information in the Transformer encoder. Each position point is mapped into a high-dimensional space by a positional encoding function:

[0021]

[0022] X = [E p1 ,E p2 ,…,E pobs ]

[0023] where the PositionalEncoding function generates an embedding vector by calculating the sine and cosine functions of the time step length. After encoding all the trajectory points of the time steps, they are combined to form a trajectory representation matrix X, which is used as input for subsequent modules. Here, pos is the position index of the trajectory time step, i is the encoding dimension, and d is the hidden layer dimension of the model. In this way, the position of each time step in the trajectory is embedded in a high-dimensional space to facilitate subsequent time-dependent feature extraction.

[0024] In step S2, the time-dependent features are learned through the self-attention mechanism of the Transformer, including the following steps:

[0025] S21: Extract the trajectory features of each pedestrian and perform linear projection, then perform self-attention operation on each pedestrian to obtain a feature matrix, and then multiply it by the corresponding weight of each pedestrian to obtain the time feature

[0026]

[0027] where Q m , K s are the query and key matrices, respectively, and d k is the scaling factor of the feature dimension.

[0028] In step S3, the spatial state of all pedestrians at each time step is extracted from the input, including the following steps:

[0029] S31: Convert the pedestrian coordinates into a spatial graph Gt = (V t , E t ), V t , E t represent the node and edge set between pedestrians respectively, is the observed coordinate position represents whether there is a connection between nodes, 1 represents connection, 0 represents no connection.

[0030]

[0031] S32: In order to better handle the spatial relationship of pedestrians, first linearly transform the spatial graph and the weight to get the graph embedding matrix, and then use the self-attention mechanism to calculate the spatial interaction matrix W s of pedestrians in space.

[0032]

[0033] where represents linear transformation, W s E , W s Q , W s K is the corresponding weight matrix, Q s , K s , are the query, key matrix respectively, d k is the scaling factor of feature dimension.

[0034] S33: Sparse processing is performed on the calculated spatial attention matrix W s . First, sparse gating mechanism is used for sparse processing to reduce redundant connections. The specific sparse processing operation is as follows:

[0035] W s '= Relu(W s -sigmoid(CNN(W s )))

[0036] This process uses convolutional neural network to extract high-order information of spatial features, and uses the combination of ReLU and sigmoid to sparse the attention weight.

[0037] S34: The spatial graph matrix is normalized by the normSoftmax function to ensure the numerical stability of the calculation,

[0038] The formula is as follows:

[0039]

[0040] where ∈ is a small constant that can be ignored to ensure numerical stability.

[0041] S35: Finally, the normalized weight matrix W is multiplied with the pedestrian node feature V s to obtain the final spatial feature:

[0042]

[0043]

[0044] where V s is the weight matrix.

[0045] In step S4, the cross-spatiotemporal correlation feature is extracted through multi-head attention, including the following steps:

[0046] S41: For the extraction of spatiotemporal interaction features, first use the multi-head attention mechanism of the transformer to construct the interaction attention weight matrix W m .

[0047]

[0048] S42: The calculated cross-spatiotemporal attention matrix W m is processed by sparse gating mechanism to reduce redundant connections. The specific sparse operation is as follows:

[0049] W m ′ = Relu(W m -sigmoid(CNN(W m )))

[0050] This process uses a convolutional neural network to extract high-order information of cross-spatiotemporal features, and through the combination of ReLU and sigmoid, the most meaningful cross-spatiotemporal connections are extracted.

[0051] S43: The sparse weight matrix W m ′ is normalized to ensure numerical stability in the calculation process. The normalization formula is as follows:

[0052]

[0053]

[0054] The normalized sparse attention weight matrix is multiplied with the feature matrix V m to obtain the cross-spatiotemporal feature representation:

[0055]

[0056] The adaptive fusion module comprises the following steps:

[0057] S51: Obtain the time feature F t , the space feature F s , and the cross-temporal-spatial correlation feature F m . For each branch feature, calculate the intermediate feature F e and the gating weight G. The intermediate feature calculation formula is:

[0058]

[0059] The weight calculation formula is:

[0060]

[0061] wherein, W F and W G are the learnable weights of the linear projection function, and the time feature, the space feature, and the cross-temporal-spatial correlation feature use different linear projection weights.

[0062] S52: Normalize each branch weight G t , G s , and G m by using the Softmax function:

[0063]

[0064] S53: Multiply the intermediate feature of each branch with the normalized weight, and then add them to obtain the final fusion feature F:

[0065]

[0066] wherein, F et , F es , and F em are the intermediate transition features of the time feature, the space feature, and the cross-temporal-spatial correlation feature after passing through the adaptive fusion module.

[0067] S6: Input the spatio-temporal feature into a time extrapolation convolutional neural network (MTCN), combine multiple layers of prediction features to jointly estimate the two-dimensional Gaussian distribution parameters of the future trajectory, and train the network model using a minimum negative log-likelihood loss function, comprising the following steps:

[0068] S61: The time extrapolation convolutional neural network (MTCN) captures the time dependence in the spatio-temporal graph convolutional feature through multiple convolutional operations. The first layer of time convolutional layer receives the graph convolutional feature, performs preliminary convolution extraction of the spatio-temporal feature, and gradually stacks the convolutional layers. The feature after each convolutional layer is processed by a ReLU activation function.

[0069] S62: Finally, the final convolutional features are mapped to the predicted values of the pedestrian future trajectory coordinates through a fully connected layer (Linear Layer).

[0070] S63: Obtain the bivariate Gaussian distribution parameters of the predicted trajectory through MTCN. Then, train the network model by minimizing the negative log-likelihood loss function,

[0071]

[0072] wherein is the predicted mean, is the predicted standard deviation, is the predicted correlation coefficient.

[0073] S64: The model uses the Adam optimizer to optimize the parameters, the learning rate is 0.001, the batch size is 32, and the training rounds are 200 rounds. The ETH, UCY data set is used for training and testing. The evaluation index is ADE (Average Displacement Error) and FDE (Final Displacement Error) to evaluate the accuracy of the predicted trajectory:

[0074]

[0075] Compared with the prior art, the beneficial effects of the present application are:

[0076] 1. Multi-scale spatio-temporal feature fusion module

[0077] One of the core innovations of the present application is the introduction of a multi-scale spatio-temporal feature fusion module. This module combines time, space and cross-spatio-temporal correlation features to enhance the prediction ability of pedestrian trajectories. Traditional trajectory prediction models often process time and space features separately, lacking effective modeling of the interaction between the two. By adaptively fusing these features, the present application not only improves the model's ability to adapt to complex dynamic scenarios, but also more accurately captures the subtle interactions and dependencies between spatio-temporal features. This technology can effectively improve the prediction effect of traditional methods in multi-target scenarios, especially in the presence of complex interactions between pedestrians, providing more accurate trajectory prediction.

[0078] 2. Sparse gating mechanism

[0079] To avoid redundant connections between nodes in the spatial graph, the present application introduces a sparse gating mechanism to sparsify the spatial and cross-spatiotemporal interaction matrices. By utilizing convolutional neural networks to extract high-order spatial and spatiotemporal cross features, and combining ReLU and sigmoid activation functions, the model can filter out unimportant connections and extract the most useful spatial and cross-spatiotemporal interaction information for trajectory prediction. This technology can effectively reduce the complexity of computation in complex spatial relationship scenarios, while avoiding redundant parts of spatiotemporal features, greatly improving the real-time performance and efficiency of the model. In addition, normalization after sparsification ensures the stability of numerical calculation, avoiding the problem of gradient explosion or gradient disappearance that may occur during training, thereby further improving the training efficiency and robustness of the model.

[0080] 3. Adaptive fusion mechanism

[0081] The present application optimizes the fusion process of time features, spatial features, and cross-spatiotemporal features through an adaptive fusion mechanism. By calculating the gating weights for each branch feature and using the Softmax function for normalization, the model can dynamically adjust the weights of each branch according to the importance of different features. This adaptive fusion mechanism not only automatically selects the most representative features for prediction, but also flexibly adjusts the fusion strategy according to scene changes, thereby improving the generalization ability and prediction accuracy of the model. Compared with traditional static feature fusion methods, this technology can better handle diverse pedestrian trajectory prediction tasks.

[0082] 4. Time extrapolation convolutional neural network (MTCN) module

[0083] The present application uses a time extrapolation convolutional neural network (MTCN) to jointly estimate spatiotemporal features to predict future pedestrian trajectories. Through multiple convolution operations, MTCN can extract and integrate the temporal dependencies in spatiotemporal graph convolution features, and output the prediction results through the final fully connected layer. The use of MTCN effectively solves the problem that traditional methods are prone to failure on time series data, especially in multi-target dynamic scenarios, and can accurately predict the future positions of pedestrians.

[0084] In summary, the innovations of the present application in spatiotemporal feature modeling, feature fusion, sparsification and normalization, and finally multi-trajectory prediction significantly improve the accuracy, efficiency, and stability of pedestrian trajectory prediction, solving multiple challenges in the prior art and providing an efficient and accurate solution for the field of pedestrian trajectory prediction. BRIEF DESCRIPTION OF DRAWINGS

[0085] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation on the present application.

[0086] Fig. 1 This is a flowchart illustrating the overall structure of the present invention.

[0087] Fig. 2 This is a diagram of the spatiotemporal fusion model used in an embodiment of the present invention. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0089] Example 1

[0090] See Figs. 1-2 The technical solution provided in this embodiment is: a pedestrian trajectory prediction method based on spatiotemporal feature fusion, comprising the following steps:

[0091] S1: Obtain the dataset, including the location coordinates of pedestrians at each time point. The trajectory data is encoded using Transformer position encoding.

[0092] Specifically, it includes the following steps:

[0093] S11: Trajectory data of pedestrians in the dataset over past time periods, including the location coordinates of each pedestrian at each time point, represented as... Where T obs Let i represent the length of the observation period, and i represent the individual pedestrian. Let be the two-dimensional coordinates of the pedestrian at time step t.

[0094] S12: Use Transformer Positional Encoding to perform temporal encoding on the trajectory data. Positional encoding generates a corresponding encoding vector for each time step t, representing temporal information to maintain the transmission of temporal information within the Transformer encoder. The positional encoding formula is:

[0095]

[0096] Where pos is the position index of the trajectory time step, i is the encoding dimension, and d is the hidden layer dimension of the model. In this way, the position of each time step in the trajectory is embedded in a high-dimensional space to facilitate subsequent extraction of temporal dependency features.

[0097] Learn the time-dependent features of each pedestrian using a self-attention mechanism:

[0098] S21: The input time series data is reshaped into a tensor of shape [T, N*sample_num, d_model], where T is the time step, N is the number of agents, sample_num is the number of samples, and d_model is the feature dimension. Then, a self-attention operation is performed on each pedestrian to obtain a feature matrix, which is multiplied by a weight to obtain a time feature. The number of multi-head attention heads used is 8, and the feature dimension is 512.

[0099]

[0100] S3: Extract the spatial state of all pedestrians at each time step from the input. Convert the pedestrian coordinates into a spatial graph, then use multi-head self-attention to extract a sparse weight matrix. Then, use a sparse gating mechanism to sparsify and normalize it to obtain spatial features. The following steps are included:

[0101] S31: Convert the pedestrian coordinates into a spatial graph G t = {(V t , E t ). G t = (V t , E t ), V t , E t represent the node and edge sets between pedestrians, is the observed coordinate position indicates whether there is a connection between nodes, 1 indicates a connection, and 0 indicates no connection. The input data is reshaped into a tensor of shape [N, T*sample_num, d_model].

[0102] S32: To better handle the spatial relationship of pedestrians, first perform a linear transformation on the spatial graph and the weight to obtain a graph embedding matrix, then use a self-attention mechanism to calculate the spatial interaction matrix W s of pedestrians in space.

[0103]

[0104] where represents linear transformation, W s E , W s Q , W s K is the corresponding weight matrix, Q s , K s , are the query and key matrices respectively, and d k is the scaling factor of the feature dimension.

[0105] S33: The calculated spatial attention matrix W s is sparsified. First, sparsification is performed through a sparse gating mechanism to reduce redundant connections. Multiple convolutional layers are mainly used to extract high-order features, further enhancing the model's ability to model complex relationships. The sparsification depth is 3 convolutional layers with a filter size of 3x3, and the number of channels for each layer is 64, 128, and 256, respectively. The specific sparsification operation is as follows:

[0106] W s ′=Relu(W s -sigmoid(CNN(W s )))

[0107] This process extracts high-order information of spatial features using a convolutional neural network and sparsifies attention weights through a combination of ReLU and sigmoid.

[0108] S34: The spatial graph matrix is normalized by the normSoftmax function to ensure numerical stability in the calculation. The formula is as follows:

[0109]

[0110] where ∈ is a small constant that can be ignored to ensure numerical stability.

[0111] S35: Finally, multiply the normalized weight matrix with the pedestrian node feature V s

[0112] to obtain the final spatial feature:

[0113]

[0114] where V m is the weight matrix.

[0115] S4: Cross-temporal correlation features are extracted through multi-head self-attention, combined with sparse gating mechanism for weight sparsification and normalization processing to obtain cross-temporal features. The following steps are included:

[0116] S41: Reshape the time series data into a tensor of shape [T*N, sample_num, d_model] to perform spatio-temporal interaction attention calculation. For cross-temporal feature extraction, first construct the spatio-temporal interaction attention weight matrix W m , which is calculated through self-attention mechanism.

[0117]

[0118] where Q m , K s, V m are query, key and value matrices respectively, d k is the scaling factor of feature dimension.

[0119] S42: Sparse the calculated cross-temporal-spatial attention matrix W m . First, sparse the W m by sparse gating mechanism to reduce redundant connections. The specific sparse operation is as follows:

[0120] W m ′=Relu(W m -sigmoid(CNN(W m ))

[0121] This process extracts high-order information of cross-temporal-spatial features by using convolutional neural network, and extracts the most meaningful cross-temporal-spatial connections by combining ReLU and sigmoid to sparse the attention weights.

[0122] S43: Normalize the sparse weight matrix W m ′ to ensure numerical stability in the calculation process. The normalization formula is as follows:

[0123]

[0124]

[0125] Multiply the normalized sparse attention weight matrix with the feature matrix V t to get the cross-temporal-spatial feature representation:

[0126]

[0127] S5: The adaptive fusion module includes the following steps:

[0128] S51: Obtain the temporal feature F s , the spatial feature F m and the cross-temporal-spatial correlation feature F e , and calculate the intermediate feature F F and the gating weight G for the feature of each branch. The intermediate feature calculation formula is:

[0129]

[0130] The weight calculation formula is:

[0131]

[0132] where W GThe learnable weights of the linear projection function, the temporal feature, the spatial feature and the cross-temporal-spatial correlation feature use different linear projection weights.

[0133] S52: Normalize the weight G of each branch by the Softmax function t ,G s ,G m Normalization is performed:

[0134]

[0135] S53: Multiply the intermediate feature of each branch by the normalized weight, and then add them to obtain the final fused feature:

[0136]

[0137] S6: By introducing a Gaussian latent variable in the CVAE framework, the embodiment realizes multi-modal prediction and generates diversified future trajectories. The following steps are included:

[0138] S61: The multi-layer time extrapolation convolutional neural network MTCN module defines 4 layers of TCN, each of which accepts the output of the previous layer as input, captures the temporal dependence in the spatio-temporal graph convolutional feature through multi-layer convolution operation. The first layer of time convolution layer receives the graph convolutional feature, performs preliminary convolution extraction of the spatio-temporal feature, and gradually stacks the convolutional layers. The output of each layer of TCN is fused with the input of the previous layer through residual connection, and the feature after each layer of convolution is processed through the ReLU activation function;

[0139] S62: Finally, a fully connected layer is used to map the final convolutional feature to a predicted trajectory, which is a sequence of positions at future time steps. The torch.mean operation is performed on the TCN output feature before prediction to reduce in the time dimension, ensuring that the feature dimension is consistent with the expected output.

[0140] S63: The bivariate Gaussian distribution parameters of the predicted trajectory are obtained through the multi-layer time extrapolation convolutional neural network (MTCN), including the prediction mean t, the pre-prediction standard deviation t and the prediction correlation coefficient t. Finally, the network model is trained by minimizing the negative log-likelihood loss function;

[0141]

[0142] where is the prediction mean, is the prediction standard deviation, is the prediction correlation coefficient.

[0143] S64: The model uses the Adam optimizer to optimize the parameters, the learning rate is 0.001, the batch size is 32, and the training rounds are 200 rounds. The ETH, UCY data set is used for training and testing. The evaluation index is ADE (Average Displacement Error) and FDE (Final Displacement Error) to evaluate the accuracy of the predicted trajectory:

[0144]

[0145] Finally, the model of the embodiment is compared with the most advanced model in recent years for ADE / FDE comparison test, and the experimental results are shown in Table 1. As can be seen from the data in Table 1, the model of the embodiment has the best performance and is better than other models.

[0146] Table 1: Comparison of ADE / FDE of the model of the embodiment and other models in the ETH, UCY data set

[0147]

[0148] Example 2

[0149] On the basis of Example 1, in order to further verify the feasibility of the technical scheme of the application and the superiority in the index performance, Example 2 compares the model with the most advanced model in recent years for ADE / FDE comparison test. Unlike Example 1, the data set of Example 2 is SDD, SDD (Stanford Drone Dataset) is a pedestrian trajectory prediction data set released by Stanford University, which records the trajectories of moving targets such as pedestrians, vehicles and bicycles in high-density traffic scenes in the campus captured from the perspective of a drone. The data set is used to analyze the motion behavior of various objects in complex environments, especially the interaction between pedestrians and other objects. The data set of SDD is complex, including the complex interaction between pedestrians and other objects, while ETH and UCY mainly use ground view shooting, mainly focusing on urban open environment and campus street, with less interaction between pedestrians and simpler scenes.

[0150] From the observation of the results in Table 2, our method significantly outperforms all existing models in the table, with ADE and FDE of 9.31 and 14.56 on the SDD dataset. In terms of ADE, it reduces by about 15% compared to SGTN, and in terms of FDE, it reduces by about 5.6% compared to STAR. This indicates that our model, by modeling trajectory features at different time scales more finely, especially in more complex pedestrian behavior patterns, strengthens the spatiotemporal dependencies that have a decisive influence on the prediction results, and adapts to the dynamic changes of different situations. In addition, the decline in FDE also benefits from the multi-layer convolution of MTCN, which can effectively capture the temporal relationships in historical trajectories, enabling the model to make improvements in predicting multiple scenarios and long time steps, and accurately predicting future trajectories.

[0151] Table 2 Comparison of ADE / FDE of the model of the present application and other models in the SDD dataset

[0152]

[0153] The above description is merely preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for predicting pedestrian trajectories using a multi-scale spatiotemporal feature fusion network, characterized in that, Includes the following steps: S1: Obtain the dataset, including the location coordinates of pedestrians at each time point. ,in For pedestrians in time steps Two-dimensional coordinates, Representing individual pedestrians, To observe the length of the time period, the trajectory data is encoded using Transformer's position encoding. S2: Use a self-attention mechanism to learn the temporal dependence features of each pedestrian; S3: Transform pedestrian coordinates into a spatial graph, then perform a linear transformation on the spatial graph and weights to obtain a graph embedding matrix. Use a self-attention mechanism to obtain the spatial interaction matrix of pedestrians in space. Then, combine a sparse gating mechanism to perform weight sparsification and normalization to obtain spatial features. S4: Cross-temporal interaction features are extracted through multi-head self-attention, and weight sparsification and normalization are performed by combining sparse gating mechanism to obtain cross-temporal interaction features. In step S4, the cross-temporal correlation features are extracted using a multi-head attention mechanism, which includes the following steps: S41: For the extraction of spatiotemporal interaction features, the multi-head attention mechanism of the transformer is first used to construct the spatial and temporal interaction attention weight matrix. ; ； in, , For query matrix and key matrix across spatiotemporal dimensions, , which is the scaling factor for the feature dimension; S42: The calculated cross-temporal attention matrix To perform sparsity reduction, firstly, a sparse gating mechanism is used to reduce redundant connections. The specific sparsity reduction operation is as follows: ； This process utilizes convolutional neural networks to extract high-order information of cross-temporal and spatial features, and sparsifies the attention weights through a combination of ReLU and sigmoid to extract the most meaningful cross-temporal and spatial connections. S43: The sparsified weight matrix To ensure numerical stability during the calculation process, normalization is performed. The normalization formula is as follows: ； ； The normalized sparse attention weight matrix and feature matrix are compared. Multiplying them together yields a spatiotemporal feature representation: ； S5: Obtain temporal features, spatial features, and cross-temporal-spatial interaction features, and adaptively fuse temporal features, spatial features, and cross-temporal-spatial correlation features through activation functions; S6: The fused spatiotemporal features are combined with the temporal extrapolation convolutional neural network MTCN to jointly estimate the two-dimensional Gaussian distribution parameters of future trajectories. The network model is trained using the minimized negative log-likelihood loss function to achieve multi-trajectory prediction.

2. The pedestrian trajectory prediction method using a multi-scale spatiotemporal feature fusion network according to claim 1, characterized in that, Step S1 includes the following steps: S11: Obtain the dataset, including the location coordinates of pedestrians at each time point, represented as... ; S12: Use Transformer's position encoding to perform time encoding on the trajectory data. Position encoding generates a corresponding encoding vector for each time step t to represent time information, thus maintaining the transmission of temporal information in the Transformer encoder. The position encoding function encodes each position point... Mapped to a higher-dimensional space: ； ； ； ； in, The function generates an embedding vector by calculating the sine and cosine functions of the time step, and after encoding the trajectory points of all time steps, combines them to form a trajectory representation matrix. This is used as input for subsequent modules, where... is the position index of the trajectory time step, i is the encoding dimension, and d is the hidden layer dimension of the model.

3. The pedestrian trajectory prediction method using a multi-scale spatiotemporal feature fusion network according to claim 1, characterized in that, In step S2, the time-dependent features are learned through the self-attention mechanism of the Transformer, which includes the following steps: S21: Extract the trajectory features of each pedestrian and perform linear projection. Then, perform a self-attention operation on each pedestrian to obtain a feature matrix, which is then multiplied by the corresponding weights of each pedestrian to obtain the temporal features. ； in For features that include time dependencies in the time dimension. , and These are the query matrix, key matrix, and value matrix in the time dimension, respectively. is the scaling factor for the feature dimension.

4. The pedestrian trajectory prediction method using a multi-scale spatiotemporal feature fusion network according to claim 1, characterized in that, In step S3, extracting the spatial state of all pedestrians at each time step from the input includes the following steps: S31: Convert pedestrian coordinates into a spatial map , Let these represent the sets of nodes and edges between pedestrians, respectively. The observed coordinates , This indicates whether there is a connection between nodes; 1 indicates a connection, and 0 indicates no connection. S32: First, perform a linear transformation on the spatial graph and weights to obtain the graph embedding matrix. Then, use a self-attention mechanism to calculate the spatial interaction matrix of pedestrians in the space. : ； ； ； ； in Represents a linear transformation. , , For the corresponding weight matrix, , These are the query and the key matrix, respectively. , which is the scaling factor for the feature dimension; S33: The calculated spatial attention matrix Sparsity processing is performed using a sparse gating mechanism to reduce redundant connections; the specific sparsity operations are as follows: ； This process utilizes convolutional neural networks to extract high-order information of spatial features and sparsifies the attention weights through a combination of activation functions ReLU and Sigmoid. S34: Spatial graph matrix passed The function is normalized to ensure numerical stability of the calculation, as shown in the following formula: ; ； in, ; S35: Finally, the normalized weight matrix Pedestrian node features Multiplying them together yields the final spatial characteristics: ； in, This is the weight matrix.

5. The pedestrian trajectory prediction method using a multi-scale spatiotemporal feature fusion network according to claim 1, characterized in that, The adaptive fusion module in step S5 includes the following steps: S51: Obtaining Time Features Spatial features and cross-temporal correlation characteristics For the features of each branch, calculate the intermediate features. and gating weights The formula for calculating intermediate features is: ； The formula for calculating the weight is: ； in, and Learnable weights for linear projection functions are assigned to temporal features, spatial features, and cross-temporal correlation features, using different linear projection weights. S52: Through Weight of the function for each branch Normalize: ； S53: Multiply the intermediate features of each branch by the normalized weights, and then add them together to obtain the final fused features: ； in, , , These are intermediate transitional features when temporal, spatial, and cross-temporal features pass through the adaptive fusion module.

6. The pedestrian trajectory prediction method using a multi-scale spatiotemporal feature fusion network according to claim 1, characterized in that, Step S6 includes the following steps: S61: The multi-layer temporal extrapolation convolutional neural network module captures the temporal dependence in the spatiotemporal graph convolutional features through multi-layer convolutional operations. The first temporal convolutional layer receives the graph convolutional features, performs preliminary convolutional extraction of spatiotemporal features, and then stacks convolutional layers step by step. The features after each convolutional layer are processed by the ReLU activation function. S62: The final convolutional features are mapped onto the predicted values ​​of the pedestrian's future trajectory coordinates through a fully connected layer; S63: The bivariate Gaussian distribution parameters of the predicted trajectory are obtained through a multi-layer temporal extrapolation convolutional neural network (MTCN). Subsequently, the network model is trained by minimizing the negative log-likelihood loss function. ； in, To predict the mean, To predict standard deviation, Predictive correlation coefficient; S64: This model uses the Adam optimizer to optimize the parameters, with a learning rate of 0.001, a batch size of 32, and 200 training epochs. It is trained and tested using the ETH and UCY datasets to evaluate pedestrian trajectory prediction and group behavior analysis. The ETH dataset contains pedestrian trajectory data in different scenarios, while the UCY dataset contains pedestrian trajectory data in multiple complex environments, including interactions between multiple pedestrians in dynamic and complex urban environments. The evaluation metrics are ADE and FDE, used to assess the accuracy of the predicted trajectory: ； 。

Images