An automatic driving-oriented multi-dimensional space-time feature fusion trajectory prediction method

Abstract

The application provides a trajectory prediction method for automatic driving based on multi-dimensional space-time feature fusion, in the spatial dimension, a graph convolution network is used to capture the local spatial features of the traffic subjects, a spatial attention mechanism is used to capture the global spatial features on the time sequence, and an LSTM is used to capture the overall spatial features on the entire historical time sequence, and the captured spatial features are fused through a gating fusion mechanism. In the time dimension, a Transformer-based encoder is used to mine the long-term time dependence of the traffic trajectory sequence and fuse the short-term time dependence obtained by the LSTM. Finally, two time convolution networks (TCN) with different convolution kernel sizes are used to predict the future trajectory of the traffic subject. By accurately predicting the trajectories of other traffic participants, the automatic driving system can better plan and execute safe driving decisions, reduce the risk of traffic accidents, and improve the overall safety of road traffic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of information technology, deep learning, automatic driving and the like, and particularly relates to a trajectory prediction method for automatic driving based on multi-dimensional spatio-temporal feature fusion. BACKGROUND

[0002] Automatic driving is attracting more and more attention from the academic and industrial circles, because it has broad prospects in solving many long-term traffic challenges related to safety, congestion, energy saving, etc. In recent years, we have witnessed the rapid development of automatic driving vehicle (AV) perception, planning and control systems. In order to further improve safety, one of the key technologies is that the automatic driving vehicle should be able to predict the future state of the surrounding traffic subjects (including vehicles, pedestrians, bicycles and motorcycles, etc.) in real time like human drivers. Improving the accuracy of trajectory prediction is of great significance to improving traffic efficiency and reducing environmental pollution. Trajectory prediction is to predict the driving trajectory of a vehicle in a future period of time according to the current state of the vehicle and the surrounding environment information. However, in practice, it is a great challenge to predict the future trajectory of a traffic subject, because the traffic subject has different characteristics at different times and in different scenarios, and because the future trajectory of the traffic subject is also affected by the actual operation of different drivers.

[0003] Early trajectory prediction methods are physics-based methods. They use vehicle dynamics models or kinematics models for prediction, but fail to consider road-related factors, resulting in low prediction accuracy. Later researchers use machine learning models to predict traffic agents, providing a new approach to trajectory prediction. However, machine learning usually requires pre-provided or identified operations, and it is difficult to capture abstract and implicit cross features. To effectively capture the rich spatio-temporal relationships contained in traffic trajectory information, most current trajectory prediction methods are based on deep learning models to model spatio-temporal features. Deep learning methods not only consider road-related factors, but also consider interaction-related factors. In the spatial dimension, early researchers use convolutional neural networks (CNN) to understand the spatial relationships in the scene and have made great progress in extracting Euclidean spatial features. With the development of graph neural networks (GNN), researchers attempt to model the spatial interaction between traffic agents as a graph and use graph neural networks to learn their dependencies. Since the node information of GCN is obtained by weighted averaging of neighboring nodes, traffic agents may not be able to accurately integrate the spatial information of their surrounding traffic agents. GAT introduces an attention mechanism, allowing each traffic agent to assign different weights to surrounding traffic agents, better capturing the spatial features of surrounding traffic agents. Whether using GCN or GAT, traffic agents can learn local spatial information. The development of attention mechanisms provides new assistance for trajectory prediction, and researchers use attention mechanisms to adaptively learn global spatial features between traffic agents. Using attention mechanisms, both local and global spatial features can be considered, allowing traffic agents to adaptively learn the spatial features of all surrounding traffic agents at each time step. However, there is a lack of mining spatial features over the entire historical time series, and different dimensions of spatial features are not fused. In the time dimension, early researchers use recurrent neural networks (RNN) for trajectory prediction. RNN can capture short-term temporal dependencies of trajectories, but it is somewhat insufficient for capturing long-term temporal dependencies of trajectory information. Later, with the advent of Temporal Convolutional Networks (TCN), it can capture long-term temporal dependencies in time series by using scalable receptive fields. This helps better understand long-term temporal dependencies in sequences. Later, Transformer can easily capture long-term temporal dependencies in sequences through self-attention mechanisms without the need to gradually expand the receptive field through convolutional layers. In capturing temporal dependencies, such methods can capture the temporal dependencies of traffic trajectories from different time dimensions, but do not fuse different dimensions of temporal features.In addition, most of the existing trajectory prediction methods study homogeneous traffic agents in a single scene, which has great limitations in dealing with dense urban environments where heterogeneous traffic agents coexist and interact. SUMMARY

[0004] Considering that traffic agents contain rich spatial information in the road network, for example, the future state of a traffic agent is greatly affected by the traffic agents within a short range and is also affected by the traffic agents within a long range. Although existing models can capture local spatial features or global spatial features, they cannot capture local, global and overall spatial features under the entire historical traffic trajectory sequence. Moreover, the historical trajectory of a traffic agent contains rich temporal information, the future trajectory of a traffic agent has great similarity with the trajectory within a short period, but the future trajectory is also related to the long-term trajectory in the past. Existing models cannot capture long-term and short-term temporal dependencies at the same time.

[0005] Therefore, in view of the defects and deficiencies of the prior art, the purpose of the present application is to provide a trajectory prediction method for multi-dimensional spatio-temporal feature fusion for automatic driving, which can effectively support the real-time prediction of the future state of surrounding traffic agents by the vehicle in automatic driving. In the spatial dimension, the local spatial features of the traffic agent are captured using a graph convolution network, the global spatial features over the time sequence are captured using a spatial attention mechanism, and the overall spatial features over the entire historical time sequence are captured in combination with an LSTM, and the captured spatial features are fused through a gated fusion mechanism. In the time dimension, the long-term temporal dependency of the traffic trajectory sequence is mined based on the encoder of the Transformer, and the short-term temporal dependency obtained by the LSTM is fused. Finally, two time convolution networks (TCN) with different kernel sizes are used to predict the future trajectory of the traffic agent. By accurately predicting the trajectories of other traffic participants, the automatic driving system can better plan and execute safe driving decisions, reduce the risk of traffic accidents, and improve the overall safety of road traffic.

[0006] The specific technical scheme adopted is as follows:

[0007] A trajectory prediction method for multi-dimensional spatio-temporal feature fusion for automatic driving: through the trajectory information of historical traffic agents, local, global and overall spatial modeling is performed by combining a graph convolution network with an attention mechanism; the encoder of the Transformer and the long short-term memory network LSTM are combined for time modeling to predict the future trajectory of the traffic agent.

[0008] Further, in the spatial dimension, the local spatial features of the traffic subjects are captured using a graph convolution network, the global spatial features on the time sequence are captured using a spatial attention mechanism, and the overall spatial features on the entire historical time sequence are captured in combination with an LSTM, and then the captured spatial features are fused through a gated fusion mechanism; in the time dimension, the long-term time dependency of the traffic trajectory sequence is mined based on the encoder of the Transformer and the short-term time dependency obtained through the LSTM is fused; finally, two time convolution networks TCN with different kernel sizes are used to predict the future trajectory of the traffic subject.

[0009] Further, comprising the following steps:

[0010] Step S1: embedding representation of traffic trajectory data and constructing multiple traffic graphs from the trajectory data, comprising:

[0011] Step S1.1: embedding representation of traffic trajectory data through a fully connected layer;

[0012] Step S1.2: constructing a distance adjacency matrix and a heading angle adjacency matrix from historical trajectory data;

[0013] Step S2: capturing local, global and overall spatial features in the trajectory data through a spatial module, comprising:

[0014] Step S2.1: capturing local spatial features using a distance adjacency matrix and a heading angle adjacency matrix through a graph convolution network;

[0015] Step S2.2: capturing global spatial features using a spatial attention mechanism;

[0016] Step S2.3: capturing overall spatial features on the entire historical trajectory sequence using a combination of LSTM and spatial attention;

[0017] Step S3: spatial feature fusion through a spatial gated fusion mechanism, comprising:

[0018] Step S3.1: fusing local and global spatial features using a spatial gated fusion mechanism;

[0019] Step S3.2: fusing the fusion output obtained in step S3.1 with the overall spatial features using a spatial gated fusion mechanism to obtain a multi-dimensional spatial feature output;

[0020] Step S4: capturing short-term and long-term time dependencies contained in the trajectory data through a time module, comprising:

[0021] Step S4.1: capturing long-term time dependency using the encoder of the Transformer;

[0022] Step S4.2: using LSTM in the space module to obtain short-term time dependency;

[0023] Step S5: performing time feature fusion, adding the captured long-term time dependency and the short-term time dependency to obtain space-time features;

[0024] Step S6: obtaining a final future trajectory through a trajectory prediction module, including:

[0025] Step S6.1: passing the obtained space-time features through a time convolution network (TCN) with two different convolution kernel sizes to obtain two different hidden states;

[0026] Step S6.2: concatenating the obtained two hidden states and then passing them through two fully connected layers to obtain a final model output.

[0027] Compared with the prior art, the present application and its preferred schemes have at least the following outstanding beneficial effects:

[0028] The space-time dependency in different space-time dimensions can be effectively captured, and the future trajectory of the traffic subject can be predicted.

[0029] In the space dimension, the local, global and overall space features of the traffic trajectory information in the entire historical time sequence are effectively fused.

[0030] In the time dimension, the long-term and short-term time dependency of the trajectory information is fully mined and fused. In particular, two TCNs with different convolution kernel sizes are used in the trajectory prediction module to generate the future trajectory. BRIEF DESCRIPTION OF DRAWINGS

[0031] The present application will be further described in detail below in combination with the drawings and specific embodiments:

[0032] Figure 1 A comparison chart of WSADE and WSFDE values under different hyperparameter settings of the embodiments of the present application;

[0033] Figure 2 A comparison chart of single trajectory prediction results of the embodiments of the present application and S2TNet;

[0034] Figure 3 A comparison chart of prediction results in extreme scenarios of the embodiments of the present application and S2TNet;

[0035] Figure 4 A comparison chart of prediction results in mixed traffic flow scenarios of the embodiments of the present application and S2TNet;

[0036] Figure 5 A comparison chart of displacement errors under different prediction steps of the embodiments of the present application and S2TNet;

[0037] Figure 6 For the embodiment of the application MDSTF overall framework diagram;

[0038] Figure 7 For the embodiment of the application traffic main body driving at the intersection schematic diagram;

[0039] Figure 8 For the embodiment of the application cavity causal convolution residual network framework diagram. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0041] In order to facilitate the detailed description of the embodiments of the present application, first, the related basic concepts are uniformly explained.

[0042] Definition 1 (traffic main body historical trajectory X) The present application represents the historical trajectory of all traffic main bodies in the traffic scene as X i , which represents the historical trajectory information of traffic main body i: , wherein , respectively represent the x-axis and y-axis coordinates of traffic main body i at time t, , represent other information of traffic main body i, such as: traffic main body type, traffic main body length and traffic main body width, etc. obs , represents the observation length.

[0043] Definition 2 (traffic main body predicted trajectory ) The present application represents the predicted future trajectory of traffic main body as , wherein the predicted trajectory of traffic main body i at time t is: , wherein T pred , represents the predicted future time length. In addition, the present application represents the real future trajectory of traffic main body i as Y i .

[0044] Definition 3 (traffic graph G t ) For each traffic main body at time step t, the present application regards each traffic main body as a node of the graph, the relationship between each traffic main body represents the edge of the graph, and then a traffic graph G t = (V t , E t ) is constructed. Wherein, the node set V t represents all traffic main bodies at time t, E t represents the relationship between traffic main bodies at time t. The present application uses the adjacent matrix A t ∈ RN×N N represents the number of nodes of each graph. t Each element in A represents whether the nodes of the graph are adjacent: if and E t , then otherwise wherein and represent the traffic agents i and j at time t.

[0045] Definition 4 (Distance adjacency matrix ) For each time stamp t, the present application constructs a distance-based adjacency matrix

[0046]

[0047] wherein, represents the element of the adjacency matrix . represents the Euclidean distance between node i and node j at time t, D threshold represents the Euclidean distance threshold value. If the distance between two nodes is less than the threshold value, it is considered that they have an interaction relationship.

[0048] Definition 5 (Heading angle adjacency matrix ) For each time stamp t, the present application constructs a heading angle-based adjacency matrix

[0049]

[0050] wherein, θ i , θ j represent the heading angles of traffic agents i and j at time t, respectively, and θ threshold represents the heading angle threshold value.

[0051] Thus, the trajectory prediction model (MDSTF) for automatic driving provided by the present application is as shown in Figure 6The model is shown in the figure. It mainly consists of three modules: spatial module, temporal module and trajectory prediction module. First, the historical trajectories of each traffic agent and the traffic graph are extracted from the historical traffic scene as the input of the spatial module. Second, in the spatial module, the traffic graph is used to model the local spatial features through the graph convolution network (GCN), and the global spatial attention mechanism is introduced to model the global spatial features (GSA). In the overall spatial network (LSAN), the LSTM combined with the spatial attention is used to model the spatial features in the whole historical time sequence, and then the spatial feature fusion is performed using the spatial gating fusion mechanism. In the temporal module, the encoder of the Transformer is used to capture the long-term temporal dependence and fuse the short-term temporal dependence output by the LSTM in the overall spatial network (LSAN). Finally, in the trajectory prediction module, two time convolution networks (TCN) with different convolution kernel sizes are used to generate the future trajectories of the traffic agents.

[0052] The following further introduces each module:

[0053] (1) Data input (module)

[0054] In order to improve the universality of the model, the historical speed data is used instead of the coordinate data as the input data of the model. The advantages of predicting speed are two. On the one hand, speed prediction can eliminate the influence of geographical location difference, which is conducive to training the proposed model using data from different locations. On the other hand, using speed as input, the model can achieve better convergence with reduced numerical space.

[0055] In addition, in order to represent more complex spatio-temporal dependence, the present application first uses a fully connected layer to convert the obtained historical speed data into an embedding representation. And a distance-based adjacency matrix And a heading angle-based adjacency matrix is used as the input of the spatial module, and its working process can be described as the following steps:

[0056] Step 1: Convert the coordinate data of the historical trajectory into speed data

[0057] Step 2: Convert the speed data input by the model into an embedding representation:

[0058]

[0059] wherein is the embedding feature vector, D is the embedding dimension, and FC represents the fully connected layer.

[0060] Step 3: Construct a distance adjacency matrix And a heading angle adjacency matrix

[0061] Therefore, the input features of the spatio-temporal layer of the MDSTF are where N represents the number of traffic agents.

[0062] Up to now, the basic work of the data input layer of the present application is completed.

[0063] (2) Spatial module

[0064] This module is used for multi-dimensional spatial feature fusion. In the traffic trajectory prediction problem, the motion of a traffic agent is greatly influenced by the motion of surrounding objects, so it is extremely important to model the spatial features of the traffic agent. Most related studies limit the spatial dependence relationship to local and global dependence relationships, while ignoring the overall dynamic spatial dependence relationship of the traffic agent over the entire historical time sequence. In order to fully exploit the spatial dependence relationship between traffic agents, the model of the present application is based on multi-dimensional spatial feature modeling.

[0065] Specifically, a graph convolution network is used to capture local spatial features, a global spatial attention mechanism is used to capture global spatial features for distant traffic agents, an LSTM combined with a spatial attention mechanism is used to model overall spatial features, and finally a gating mechanism is used to fuse all spatial features to fully exploit the spatial dependence relationship between traffic agents. The following embodiment describes the detailed process of local, global, and overall spatial modeling and the gating mechanism for fusing spatial features.

[0066] Local spatial modeling: the state of a traffic agent is greatly influenced by traffic agents within a close range. Here, the present application uses a graph convolution network to capture the local spatial relationship between traffic agents, where the graph convolution network describes the state transition between nodes as a spatial diffusion process of the nodes, and simulates this process by performing random walks on the graph. This Markov random process converges to a smooth distribution after K time steps. In a traffic scenario, the graph convolution network can be represented as: In the model of the present application, the diffusion convolution at time t can be represented as:

[0067]

[0068] where H t ∈R N×D represents the input graph features at time step t, L t ∈R N×D represents the output graph features after processing at time step t, P k represents the power series of the transition matrix, W k1 ,W k2 are learnable parameters, and K is the diffusion step length. The present application uses the distance adjacency matrix defined by equation (4) and the heading angle adjacency matrix defined by equation (5) and As input, rowsum() represents the sum operation on the matrix row value. The graph convolution in formula 4 is actually used to aggregate the transformed feature information from different neighborhood orders. The output result L t of the diffusion convolution is just the spatial information of the target traffic agent aggregated from the surrounding traffic agents.

[0069] So far, the local spatial information of the traffic agent can be captured by the graph convolution network.

[0070] Global spatial modeling: The state of a traffic agent is not only affected by the state of the surrounding nearby traffic agents, but also by the state of the distant traffic agents. Although the graph convolution network can well aggregate the information of the nearby nodes, it lacks the aggregation of the global spatial information. To solve this problem, the multi-head self-attention mechanism is used to capture the global information. Its working process can be represented by the following steps:

[0071] Step 1: Given the input traffic feature H t ∈R N×D at time step t, the subspace is generated by linear transformation

[0072]

[0073] where is the learnable parameter.

[0074] Step 2: Then the attention score of traffic agent i and traffic agent j is obtained by multi-head scaled dot product, and finally the features of the adjacent traffic agents are aggregated to the target traffic agent according to the obtained attention weight, which can be represented as:

[0075] G t =Concat(head1,head2,…,head n )W G (6)

[0076]

[0077] where n represents the number of attention heads, m represents the mth attention head, softmax represents the activation function, and W G represents the weight matrix. G t represents the feature of each traffic agent after aggregating the global traffic agents.

[0078] So far, the global spatial information of the traffic agent can be captured by the spatial attention mechanism.

[0079] Overall spatial modeling: Although the existing methods have studied the spatial relationship of traffic agents in detail, they are limited to local and global relationships, ignoring the overall spatial relationship of traffic agents in the entire historical time sequence. Taking the scene shown in the left as an example, the orange vehicle and the yellow vehicle keep the same heading angle and driving speed in the entire historical time sequence. Therefore, the yellow vehicle has greater influence and similarity to the orange vehicle, but in the scene shown in the right, the yellow vehicle enters a right-turn lane in the middle process. Therefore, compared with the yellow vehicle in the middle, the yellow vehicle on the right has relatively less influence and similarity to the orange vehicle. In order to capture this spatial relationship, the present application proposes a method of using LSTM combined with multi-head attention mechanism to model the overall spatial features of traffic agents, which can be represented as the following steps: Figure 7 Figure 7 Figure 7 Figure 7

[0080] Step 1: input the traffic features H into the LSTM, and use the last hidden state H' of the LSTM as the time feature of the traffic agent after fusing the historical time sequence of each traffic agent, that is: N ×D

[0081] s i

[0082] i S

[0083] Step 2: In order to capture the spatial features of traffic agents in the entire historical time sequence, a multi-head attention mechanism is used to capture this complex spatial feature. Given the input feature H' of the traffic agent, the subspace is generated by linear transformation: N×D Their form is as follows:

[0084]

[0085] Where is a learnable parameter.

[0086] Step 3: Then, the attention score of traffic agent i and traffic agent j is obtained by multi-head scaled dot product. Finally, according to the obtained attention weight, the features of adjacent traffic agents are aggregated to the target traffic agent, and the process can be represented as:

[0087] O=Concat(head1,head2,…,head n )W​​​​​​​​​​​​​O (10)

[0088]

[0089] where W O is the learnable parameter. O represents the overall spatial feature captured by LSTM and multi-head attention mechanism.

[0090] So far, the embodiment scheme can capture the overall spatial information of the traffic subject by combining LSTM with the spatial attention mechanism.

[0091] Spatial gate fusion: in order to fuse the local spatial feature L obtained by the graph convolution network, the global spatial feature G obtained by the attention mechanism and the overall spatial feature O, the application designs a gate fusion mechanism with a sigmoid activation function. This mechanism uses a sigmoid activation function to map the importance of input features to a value between 0 and 1. Then, these weights are multiplied by the corresponding features to achieve the weighted fusion of different input features. This mechanism is represented by the following steps:

[0092] Step 1: First, the application uses this spatial gate fusion mechanism to fuse local and global features as follows:

[0093] z1 = σ(L·W 11 + G·W 12 + b g1 (12)

[0094] GL = z1⊙L + (1-z1)⊙G (13)

[0095] In the above formula, σ represents the sigmoid activation function, W 11 ,W 12 is the learnable weight, b g1 is the bias value, and ⊙ represents bitwise multiplication.

[0096] Step 2: Then, the application also uses the above gate mechanism to fuse the overall spatial feature:

[0097] z2 = σ(GL·W 21 + O·W 22 + b g2 ) (14)

[0098] GLO = z2⊙GL + (1-z2)⊙O (15)

[0099] GLO contains complex spatial features in the historical trajectory of the traffic subject.

[0100] So far, the application has fused the local, global and overall spatial features in the traffic trajectory sequence through the gate fusion mechanism.

[0101] (3) Time module

[0102] In this step, the present application gives the detailed process of multi-dimensional time feature fusion. The traffic trajectory sequence contains rich time features, and the Transformer performs outstanding performance in processing long time sequence problems. The Transformer network mainly consists of self-attention mechanism and position encoding, which provides position information for each position of the input sequence by introducing position encoding. The self-attention mechanism allows the Transformer network to calculate the correlation between each input position and other positions when processing sequence data. This enables the Transformer network to directly capture the dependency between different positions in the input sequence without iteratively processing the sequence one by one like RNN. In the MDSTF model, the present application uses the encoder of the Transformer network to capture the long-term time dependence of the traffic trajectory sequence:

[0103] Step 1: Traffic trajectory data is classic time series data, and analyzing the temporal position relationship between data can improve the performance of traffic trajectory prediction. The Transformer introduces sine and cosine position embedding to record the position information of time series data, and the position embedding is defined as:

[0104]

[0105]

[0106] where pos represents the position in the sequence, k represents the length of the sequence, d is the dimension of the sequence, 2k represents the even dimension, and 2k+1 represents the odd dimension. For each element in the sequence data, the present application generates embedding by calculating different frequencies through sin and cos functions.

[0107] Step 2: Add the output GLO obtained in the spatial module to the position encoding, as shown in the following formula:

[0108] X = GLO + pe (18)

[0109] where pe represents the position encoding.

[0110] Step 3: Perform self-attention operation on the obtained X, as shown in the following formula:

[0111] TemporalAttention(X) = Concat(head1, head2, …, head n )W T (19)

[0112]

[0113] TemporalAttention represents multi-head self-attention. W T These are learnable parameters.

[0114] Step 4: Normalize TemporalAttention(X) using Dropout (DP) and Layer Normalization (LN) layers, as shown in the following formula:

[0115] Res=LN(DP(TemporalAttention(X))+X) (21)

[0116] Where Res represents the intermediate features of the encoder.

[0117] Step 5: Input the intermediate features Res from the encoder into the feedforward network, as shown in the following formula:

[0118] FeedForward(Res)=ReLU(ResW1+b1)W2+b2 (22)

[0119] FeedForward is a feedforward network, which is a two-layer fully connected network. The activation function of the first layer is ReLU, and the second layer does not use an activation function. W1, W2, b1, and b2 are the corresponding learnable parameters.

[0120] Step 6: Normalize the obtained FeedForward(Res) through the Dropout(DP) layer and LayerNormalization(LN) layer, and set a residual connection, as shown in the following formula:

[0121] ST L =LN(DP(FeedForward(Res))+Res) (23)

[0122] Among them ST L This represents the output of the Transformer encoder.

[0123] Step 7: Finally, the short-term time dependency T obtained by LSTM in the overall spatial modeling module is integrated into the present invention. s :

[0124] ST out =ST L +T s (twenty four)

[0125] ST outThis represents the multidimensional spatiotemporal features captured after the traffic history trajectory passes through the spatial and temporal modules.

[0126] Thus, by fusing spatiotemporal features from different dimensions, this invention can capture the complex spatiotemporal features of historical traffic trajectories.

[0127] (4) Trajectory Prediction Module

[0128] In existing technologies, most researchers use a Seq2Seq structure for final trajectory prediction. However, the prediction results of the Seq2Seq structure rely excessively on the context vector generated by the encoder, and the loss of context vector information will lead to inaccurate prediction results. CNN-based TCN has the characteristics of translation invariance and parallel computation, is not limited by the context window, and compared with the Seq2Seq structure, TCN has fewer parameters and higher training efficiency.

[0129] Therefore, this invention uses TCN for the final trajectory prediction. Figure 8 As shown, TCN consists of a series of stacked residual blocks that implement dilated causal convolutional layers. Each layer of dilated causal convolution uses the same-sized kernel and different dilation rates. In the scenario of this invention, for a given input time series... and filter Γ=(f1,f2,…,f U ), in x t The dilated causal convolution at a given location is defined as:

[0130]

[0131] Where d is the dilation rate, which indicates the distance between the convolution kernels.

[0132] Each residual block primarily consists of two 2D convolutional layers, WeightNorm normalization, ReLU activation, and Dropout. TCN residual blocks capture information at different time scales by stacking multiple convolutional layers, accelerate training and address the vanishing gradient problem through residual connections, and expand the receptive field using dilated convolutions. These designs enable TCN to handle long-term dependencies.

[0133] The expansion factor in TCNs is typically set to an exponential form of 2, which significantly improves computational speed but may lose local information, leading to a decrease in prediction accuracy. To improve prediction accuracy, this invention uses two different TCNs to extract temporal dependencies and sets different convolutional kernel sizes (k1 and k2, respectively) and expansion rates for the two TCN networks. Finally, the results from the two different TCNs are concatenated and then passed through two fully connected layers to obtain the final output. Given ST... out The trajectory prediction module of the present invention is shown below:

[0134]

[0135] where Θ1, Θ2 are two independent TCN networks, FC represents the fully connected layer. The time T obs +1 to T obs +T pred Final future trajectory

[0136] The present application uses L2-loss as the loss function of the MDSTF model:

[0137]

[0138] where is the predicted value of the MDSTF model, Y t is the true value.

[0139] The following further introduces and demonstrates the present application scheme through verification experiments:

[0140] The experiments of the present embodiment are carried out on the Apolloscape traffic trajectory dataset, and the average displacement error (ADE) and the final displacement error (FDE) are used as two indicators to evaluate the model effect of the present application. Among them, the average displacement error (ADE) refers to the average Euclidean distance between all predicted positions and true positions, and the final displacement error (FDE) refers to the Euclidean distance between the position at the last prediction time and the true position. According to the description of the official dataset, since the heterogeneous traffic subjects are different in scale, the weighted average displacement error (WSADE) and the weighted final displacement error (WSFDE) are used as the measurement indicators:

[0141] WSADE = D v ·ADE v + D p ·ADE p + D b ·ADE b (28)

[0142] WSFDE = D v ·FDE v + D p ·FDE p + D b ·FDE b (29)

[0143] where D v , D p , and D bThe values of the three variables are set to 0.20, 0.58 and 0.22 respectively in relation to the inverse of the average speed of vehicles, pedestrians and cyclists in the data set.

[0144] Experimental data set

[0145] In order to verify the actual effectiveness of the MDSTF, the model of the present application is evaluated on the ApolloScape trajectory dataset. The ApolloScape trajectory dataset includes images, point clouds and manually annotated trajectories, containing 53 minutes of training trajectory sequences and 50 minutes of test trajectory sequences captured at 2 frames per second. It mainly consists of complex traffic flow, in which vehicles (small and large vehicles), pedestrians, cyclists (motorcyclists and cyclists) are mixed in the city. Therefore, this dataset has certain significance for the study of mixed traffic scenes.

[0146] Parameter setting experiment

[0147] A comparative experiment is conducted on the parameter settings involved in the model of the present application, including the model parameter distance threshold D threshold , the heading angle threshold θ threshold , the TCN convolution kernel size k1 and k2, etc. The experimental results are shown in Table 1. Among them, Parameter represents the parameter name studied in the experiment; Object Type represents the type of traffic subject; Parameter Value represents the reasonable value of the parameter experimental setting. In addition, the embedding dimension D, the number of spatiotemporal layer stacks L, and the number of multi-head attention heads n in the text are also explored, and the experimental results are shown in Table 2. Figure 1

[0148] In constructing the distance adjacency matrix, the present embodiment sets different values for the distance threshold of different traffic subject categories. Considering that in the real world, the range that affects the motion state of pedestrians, cyclists and vehicles will be different, the present application sets the distance threshold of pedestrians to be smaller than that of cyclists, and the distance threshold of cyclists to be smaller than that of vehicles. According to Table I, when the distance thresholds of pedestrians, cyclists and vehicles are set to 10, 15 and 20 respectively, the experimental effect is the best. When the respective distance thresholds are increased by 5, WSADE and WSFDE become larger and the experimental effect becomes worse, which shows that when the distance threshold increases, too many traffic subjects will be treated as invalid neighbor nodes in the traffic graph, thereby interfering with the accuracy of the model. When the respective distance thresholds are reduced by 5, WSADE and WSFDE also become larger and the experimental effect becomes worse, which also shows that too small distance threshold will cause the traffic graph to lack effective neighbor nodes, reducing the accuracy of the model.

[0149] ​In order to verify the rationality of setting different distance thresholds for different traffic agents, the application also sets a comparative test. As shown in Table 1, the same distance threshold is set for different traffic agents. When the distance threshold is set to 20, the experimental results are best, and when the distance threshold is reduced or increased, WSADE and WSFDE both show an increasing trend, but compared with setting different distance thresholds for different traffic agents, there is still a gap, which also verifies the rationality of setting different distance thresholds for different traffic agents.

[0150] For the heading angle threshold θ threshold The parameters are set, and the embodiment does not distinguish the difference between different types of traffic agents. When θ threshold When increased from π / 12 to π / 4, WSADE and WSFDE first decrease and then increase, and the experimental effect becomes worse. Due to the possible deviation or noise of the recorded heading angle in the data set, a smaller heading angle threshold may cause some traffic agents with similar states to be ignored, and a larger heading angle threshold may include traffic agents with different traffic states into the neighbor nodes, thereby affecting the accuracy of the model. When θ threshold = π / 6, the experimental effect is best.

[0151] For the convolution kernel size k1 and k2 in the two TCNs, because the historical trajectory step length of the Apollo scape data set is 6, the embodiment selects three convolution kernel sizes of 2, 3 and 5, and combines them in pairs. As can be seen from Table I, the effects of different combinations are not much different, and when k1 = 2 and k2 = 3, the experimental effect is best.

[0152] Figure 1 The WSADE and WSFDE prediction results of MDSTF with different hyperparameter setting values are shown. When one parameter is adjusted, the other parameters are set to the best value by default. As can be seen from the figure, appropriately increasing the number of space-time layers and attention heads can improve the performance of the model. When the embedding dimension is 64 and the number of attention heads is 8, the effect is best. For the space-time layer, the more the number of layers, the better the experimental effect, and when L = 5, the performance of the model is best, but the more the number of layers, the more likely the model is too complex and reduces the calculation efficiency.

[0153] In summary, the distance threshold D threshold is set to 20, 15 and 10 respectively, the heading angle threshold θ threshold is set to π / 6, the embedding feature dimension is set to 64, and the number of heads of the multi-head attention mechanism is set to 8. The convolution kernel size k1 of one TCN is set to 2, and the convolution kernel size k2 of the other TCN is set to 3. The Dropout is uniformly set to 0.2.

[0154] This embodiment trains MDSTF on an NVIDIA GeForce 3080Ti GPU using Pytorch. The optimizer uses Adam, with an initial learning rate set to 0.001 and a batch size set to 32 during training.

[0155] Table 1 Experimental study on parameter values of the method of the present application

[0156]

[0157] Ablation experiment

[0158] In order to verify the effectiveness of each component in the model, this embodiment sets several groups of ablation experiments to observe the changes in the performance of the model, and the experimental results are shown in Table 2. The names of each module in the table are as follows: LSAN represents the overall spatial network module; GSA represents the global spatial attention module; DA represents the distance adjacency matrix; HA represents the heading angle adjacency matrix; Single-TCN represents a single TCN; Double-TCN represents two TCNs; PE represents the position encoding of the Transformer; S-LSTM represents whether the short-term time dependence of the LSTM output is fused in the Transformer encoder, and this embodiment uses “√” to represent whether the MDSTF contains the above-mentioned modules.

[0159] From Table 2, this embodiment can draw the following conclusions:

[0160] • Comparing A and B, the model with overall spatial modeling has a significant improvement in accuracy compared to the model without overall spatial modeling, with WSADE and WSFDE decreasing by 3.34% and 3.68%, respectively. This also verifies that the local and global spatial modeling is not able to fully capture the spatial relationship between traffic agents, and the overall spatial modeling makes the modeling of spatial dependence more comprehensive.

[0161] • Comparing A and C, the global spatial modeling also improves the accuracy of the model, with WSADE and WSFDE decreasing. This indicates that the state of traffic agents is not only affected by nearby traffic agents, but also by distant traffic agents, thereby verifying the rationality of considering global spatial features.

[0162] • Comparing A, D, and E, the distance adjacency matrix DA and the heading angle adjacency matrix HA can improve the prediction accuracy and enhance the features. Among them, the distance adjacency matrix DA has a greater effect on the model, while the heading angle adjacency matrix HA has a smaller effect on the model, which indicates that the future trajectory of traffic agents is more affected by nearby traffic agents.

[0163] • Comparing A and F, we can see that two TCNs with different kernel sizes improve the prediction accuracy of the model, which also shows that TCN may lose local information, and also verifies that the capture of time relationship of two TCNs with different kernel sizes can complement each other.

[0164] • Comparing A and G, we can see that the position encoding in the time Transformer can improve the prediction accuracy, because the historical trajectory is a time series, which contains the time relationship, which also proves that the position encoding can make the model learn the time relationship of the trajectory sequence. At the same time, comparing A and H, we can see that the Transformer cannot capture the short-term time dependence of the traffic trajectory well, which also shows that the traffic trajectory is more affected by the state of the traffic subject in the short term.

[0165] Table 2 Ablation experiment of the model of the application

[0166]

[0167]

[0168] Baseline comparison experiment

[0169] In order to evaluate the performance of the MDSTF model, the embodiments compare the officially released representative latest models of ApolloScape, and the baseline models involved are as follows:

[0170] • TrafficPredict: A real-time traffic prediction algorithm based on long short-term memory network (LSTM). It is the baseline model of the ApolloScape trajectory dataset.

[0171] • Social LSTM (S-LSTM): A social pool mechanism is designed to capture neighbor information and use LSTM to extract pedestrian trajectory features.

[0172] • Social GAN (S-GAN): Use GAN to predict socially reasonable future trajectories.

[0173] • StarNet: Predict the future walking path by constructing a pedestrian trajectory network, and use a star topology to depict the interaction between pedestrians.

[0174] • Transformer: Use ordinary Transformer to model pedestrian trajectories without any complex interaction terms.

[0175] • TPNet: First, generate a candidate trajectory set of future trajectories. Then, through classification and refinement of the candidate trajectory set, the final prediction result is obtained.

[0176] • GRIP++: uses graphs to represent the interaction between traffic agents and employs multiple graph convolution blocks to aggregate features. Then, an LSTM model with an encoder-decoder structure is used to predict the future traffic behavior.

[0177] • MVHGN: constructs a multi-view logical network to realize multi-view logical feature extraction, and then combines an adaptive spatial topology network and a macro-level regional clustering network to capture micro-level logical physical features and global logical physical features, respectively. A Seq2Seq model with GRU is used for prediction.

[0178] • S2TNet: captures spatio-temporal dependencies through spatial self-attention mechanism and TCN, and predicts future trajectories through a Transformer network.

[0179] The comparison results of MDSTF with the above baseline models are shown in Table 3. First, from the experimental results, the performance of MDSTF is better than all the baselines, and compared with the most advanced scheme (S2TNet), MDSTF reduces by 4.37%, 6.23% in the two most important indicators WSADE and WSFDE, respectively, and on the three traffic agents (vehicles, pedestrians, and cyclists) of the Apolloscape dataset, MDSTF reduces by 5.98%, 1.37%, 5.82% in ADE, respectively, and by 6.84%, 2.52%, 9.58% in FDE, respectively. Second, MDSTF has a larger improvement in FDE index relative to the S2TNet model, which indicates that the MDSTF model has better accuracy and higher robustness for long-time trajectory prediction. Finally, compared with S2TNet (which considers global spatial features and long-term temporal dependencies), MDSTF has better accuracy; compared with MVHGN (which constructs multiple adjacency matrix GCNs), MDSTF is much better than MVHGN in WSADE and WSFDE indicators.

[0180] Table 3 Comparison results of the Apolloscape dataset

[0181] Method WSADE ADEv ADEp ADEb WSFDE FDEv FDEp FDEb TrafficPredict 8.5881 7.9467 7.1811 12.8805 24.2262 12.7757 11.1210 22.7912 S-LSTM 1.8922 2.9456 1.2856 2.5337 3.4024 5.2802 2.3240 4.5384 S-GAN 1.5829 3.0430 0.9836 1.8354 2.7796 5.0913 1.7264 3.4547 StarNet 1.3425 2.3860 0.7854 1.8628 2.4984 4.2857 1.5156 3.4645 Transformer 1.2803 2.2322 0.7398 1.8398 2.4024 4.0317 1.4309 3.4826 TPNet 1.2800 2.2100 0.7400 1.8500 2.3400 3.8600 1.4100 3.4000 GRIP++ 1.2588 2.2400 0.7142 1.8024 2.3631 4.0762 1.3732 3.4155 MVHGN 1.2069 2.1349 0.7192 1.6490 2.2539 3.8925 1.3876 3.0483 S2TNet 1.1679 1.9874 0.6834 1.7000 2.1798 3.5783 1.3048 3.2151 MDSTF 1.1168 1.8684 0.6740 1.6009 2.0438 3.3333 1.2718 2.9069

[0182] Analysis and visualization

[0183] This embodiment mainly discusses the future trajectory prediction problem of traffic agents in mixed traffic scenes, and visualizes several prediction results of the ApolloScape Trajectory dataset, as shown in Figures 2 to 5

[0184] 1) Single trajectory prediction results of different categories of traffic agents compared with S2TNet

[0185] In​Figure 2 The trajectories of individual traffic agents of the four traffic agent categories compared with S2TNet are shown in FIG. 6. MDSTF can accurately predict the trajectories of different traffic agent categories in the future 3 seconds, which is consistent with the WSADE in the experimental results of the embodiment. As the prediction length increases, the position of the last prediction time of MDSTF is more consistent compared with S2TNet, which is consistent with the WSADE in the experimental results of the embodiment, and the cumulative error of MDSTF is better than that of S2TNet.

[0186] 2) Single trajectory prediction results of different categories of traffic agents in extreme scenarios compared with S2TNet

[0187] In Figure 3 , the embodiment shows the trajectories of traffic agents in some extreme or sharp turning situations, and compares them with the visualization results of S2TNet. In such traffic scenarios, both the observed trajectory and the future trajectory are irregular, and the future trajectory is not a simple continuation of the observed trajectory. Therefore, it is difficult for S2TNet to accurately predict the future trajectory of the traffic agent. Figure 3 It can be seen that the trajectory prediction of the application in such extreme scenarios can also achieve good prediction results compared with S2TNet. For example, Figure 3 (a) it can be seen that the car is almost in a static state in the past time, but the future trajectory suddenly starts to move, and MDSTF can roughly predict the future trajectory, but S2TNet predicts that it is still in place. MDSTF can fit the real future trajectory of the traffic agent as much as possible.

[0188] 3) Trajectory prediction results in mixed traffic flow compared with S2TNet

[0189] Figure 4 In Figure 4 , the embodiment shows the trajectory prediction visualization of a few traffic agents and multiple traffic agents in mixed scenarios compared with S2TNet. In Figure 4 (a), there are few traffic agents in the scene, and the interaction between them may be small, but in Figure 4In (b), the number of traffic agents is large, and from the visualization figure, it can be seen that compared with S2TNet, MDSTF also has good prediction effect on the future trajectory prediction of multiple traffic agents in the scene. Especially for traffic agents D and E, their historical trajectories only have the last time, but their future trajectories have a large amplitude of movement. S2TNet predicts that they are still in place, but MDSTF can well predict their future trajectories. From the above, it can be seen that MDSTF has good performance for multi-traffic agent future trajectory prediction under mixed traffic flow conditions.

[0190] 4) DE index on prediction steps

[0191] In addition to comparing WSADE and WSFDE indexes on the ApolloScape dataset, this embodiment also compares the DE index on all prediction steps, where DE refers to the displacement error between the predicted position at time t and the true position. The experimental results are shown in Figure 5 In addition to the above traffic agent categories, another type of unknown traffic agent is also shown here. Whether it is a vehicle, a pedestrian, a cyclist, or other types of traffic agents, the DE index of the first few steps is not much different from the S2TNet result, but at the last few time steps, the DE index of the vehicle and the cyclist is much lower than that of S2TNet, and the DE index of the pedestrian and other types of traffic agents is slightly lower, which shows that MDSTF has a greater advantage in long-term prediction compared with S2TNet.

[0192] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied in the medium.

[0193] The application is described with reference to the flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the flowcharts and / or block diagrams. Figure 1 one flow or multiple flows and / or blocks Figure 1apparatuses that carry out functions specified in one or more of the blocks.

[0194] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 one or more flows and / or blocks Figure 1 functions specified in one or more of the blocks.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 steps of functions specified in one or more of the blocks.

[0196] The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application to other forms. Any person skilled in the art can make modifications or improvements to the above-mentioned disclosed technical contents, or make equivalent changes and equivalent embodiments. However, any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments, without departing from the technical solution of the present application, shall still fall within the protection scope of the present application.

[0197] The present application is not limited to the above-mentioned preferred embodiments, and anyone can derive other various forms of a multi-dimensional space-time feature fusion trajectory prediction method for automatic driving under the inspiration of the present application. Any equivalent changes and modifications made within the scope of the present application shall fall within the scope of the present application.

Claims

1. A trajectory prediction method for automatic driving oriented multi-dimensional spatio-temporal feature fusion, characterized in that: The local, global and overall space modeling is performed by combining a graph convolution network and an attention mechanism through historical traffic subject trajectory information; the time modeling is performed by combining an encoder of a Transformer and a long short-term memory network (LSTM) to predict the future trajectory of the traffic subject; In the spatial dimension, the local spatial features of the traffic subject are captured by using a graph convolution network, the global spatial features in the time sequence are captured by using a spatial attention mechanism, and the overall spatial features in the entire historical time sequence are captured by combining an LSTM, and then the captured spatial features are fused by a gating fusion mechanism; in the time dimension, the long-term time dependency of the traffic trajectory sequence is mined based on an encoder of a Transformer, and the short-term time dependency obtained by the LSTM is fused; finally, two time convolution networks (TCNs) with different convolution kernel sizes are used to predict the future trajectory of the traffic subject; The method comprises the following steps: Step S1: embedding representation of traffic trajectory data and construction of multiple traffic graphs from the trajectory data, comprising: Step S1.1: embedding representation of the traffic trajectory data through a fully connected layer; Step S1.2: construction of a distance adjacency matrix and a heading angle adjacency matrix from the historical trajectory data; Step S2: capturing local, global and overall spatial features in the trajectory data by a spatial module, comprising: Step S2.1: capturing local spatial features by a graph convolution network using the distance adjacency matrix and the heading angle adjacency matrix; Step S2.2: capturing global spatial features by using a spatial attention mechanism; Step S2.3: capturing overall spatial features in the entire historical trajectory sequence by combining an LSTM and a spatial attention mechanism; Step S3: spatial feature fusion by a spatial gating fusion mechanism, comprising: Step S3.1: fusion of the local and global spatial features by the spatial gating fusion mechanism; Step S3.2: fusion of the fusion output obtained in step S3.1 with the overall spatial features by the spatial gating fusion mechanism to obtain a multi-dimensional spatial feature output; Step S4: capturing short-term and long-term time dependencies contained in the trajectory data by a time module, comprising: Step S4.1: capturing long-term time dependency by an encoder of a Transformer; Step S4.2: obtaining short-term time dependency by an LSTM in the spatial module; Step S5: time feature fusion, adding the captured long-term time dependency and the short-term time dependency to obtain a spatiotemporal feature; Step S6: obtaining a final future trajectory by a trajectory prediction module, comprising: Step S6.1: obtaining two different hidden states by passing the obtained spatiotemporal feature through two time convolution networks (TCNs) with different convolution kernel sizes; Step S6.2: obtaining a final model output by concatenating the obtained two hidden states and then passing them through two fully connected layers; The time module uses an encoder of a Transformer network to capture the long-term time dependency of the traffic trajectory sequence: The Transformer uses sine and cosine position embedding to record the position information of the time series data, and the position embedding is defined as: where pos represents the position in the sequence, k represents the length of the sequence, d is the dimension of the sequence, 2k represents the even dimension, and 2k+1 represents the odd dimension; for each element in the sequence data, the embedding is generated by calculating different frequencies through sin and cos functions; Then add the output GLO obtained in the space module to the position encoding, as shown in the following formula: X=GLO+pe (18) Where pe represents the position encoding; Then perform self-attention operation on the obtained X, as shown in the following formula: TemporalAttention(X) = Concat(head1, head2,..., head n )W T (19) where TemporalAttention denotes multi-head self-attention, W T are learnable parameters; Then normalize TemporalAttention(X) through the Dropout layer and the Layer Normalization layer, as shown in the following formula: Res=LN(DP(TemporalAttention(X))+X) (21) Where Res represents the intermediate feature of the encoder; Input the intermediate feature Res of the encoder into the feedforward network FeedForward, as shown in the following formula: FeedForward(Res)=ReLU(ResW1+b1)W2+b2 (22) Where FeedForward is a two-layer fully connected layer, the activation function of the first layer is ReLU, and the second layer does not use the activation function, W1, W2, b1, and b2 are the corresponding learnable parameters; The obtained intermediate feature Res is further subjected to the Dropout layer and the Layer Normalization layer normalization operation, and a residual connection is set, as shown in the following formula: ST L = LN(DP(FeedForward(Res)) + Res) (23) where ST L denotes the output of the Transformer encoder; Finally, the short-term temporal dependencies T obtained from the LSTM are fused in the holistic space modeling module s : ST out = ST L + T s (24) ST out represents the multi-dimensional spatio-temporal features captured after the traffic historical trajectory passes through the spatial module and the temporal module.

2. The trajectory prediction method for multi-dimensional spatio-temporal feature fusion for automatic driving according to claim 1, characterized in that: Let X denote the historical trajectories of all traffic agents in the traffic scene, X i denotes the historical trajectory information of traffic agent i: wherein denote the x-axis and y-axis coordinates of traffic agent i at time t, respectively, denotes other information of traffic agent i, specifically the traffic agent type, the traffic agent length and the traffic agent width, T obs denotes the observation length; a predicted future trajectory of a traffic agent wherein the predicted trajectory of the traffic agent i at time t is: wherein T pred represents a predicted future time length; and the real future trajectory of the traffic agent i is denoted as Y i ; For each traffic agent at time step t, each traffic agent is regarded as a node of a graph, and the relationship between each traffic agent represents an edge of the graph, and then a traffic graph G is constructed t = (V t , E t ); wherein, the node set V t represents all traffic agents at time t, E t represents the relationship between traffic agents at time t; the adjacency matrix A t ∈R N×N is used to represent each graph, N is the number of nodes; the value of each element in A represents whether the nodes of the graph are adjacent: if and then otherwise wherein and represent traffic agents i and j at time t; For each time stamp t, construct a distance-based adjacency matrix wherein, denotes an element of the adjacency matrix ; denotes the Euclidean distance between node i and node j at time t, D threshold represents the Euclidean distance threshold value; if the distance between two nodes is less than the threshold value, it is considered that they have an interaction relationship. For each time stamp t, construct a heading-based adjacency matrix where θ i ,θ j represent the heading angles of the traffic subjects i and j at time t, respectively, and θ threshold represents a heading angle threshold value.

3. The trajectory prediction method for automatic driving oriented multi-dimensional spatio-temporal feature fusion according to claim 2, characterized in that: The trajectory prediction model for multi-dimensional spatio-temporal feature fusion for automatic driving comprises a space module, a time module and a trajectory prediction module; first, the historical trajectories of each traffic subject and the traffic graph are extracted from the historical traffic scene as the input of the space module; second, in the space module, the traffic graph is used to model the local spatial features through the graph convolution network, and the global spatial attention mechanism is introduced to model the global spatial features; in the whole spatial network, the LSTM combined with the spatial attention is used to model the spatial features in the whole historical time sequence, and then the spatial feature fusion is performed using the spatial gating fusion mechanism; in the time module, the encoder of the Transformer is used to capture the long-term time dependence and fuse the short-term time dependence output by the LSTM in the whole spatial network; finally, in the trajectory prediction module, two time convolution networks TCN with different convolution kernel sizes are used to generate the future traffic subject trajectory.

4. The trajectory prediction method for multi-dimensional spatio-temporal feature fusion for automatic driving according to claim 3, characterized in that: In the trajectory prediction model for autonomous driving, historical speed data is used as input data instead of coordinate data. First, the historical speed data is converted into embedded representation using a fully connected layer. and a distance-based adjacency matrix is constructed from the historical traffic trajectory sequence and a heading-based adjacency matrix for input to the spatial module: Converting coordinate data of historical trajectories into velocity data The speed data input into the model is converted into embedded representation. wherein is an embedding feature vector, D is an embedding dimension, and FC denotes a fully connected layer; constructing a distance adjacency matrix from historical trajectory data and a heading angle adjacency matrix The input features of the spatiotemporal layer are obtained as where N represents the number of traffic agents.

5. The trajectory prediction method for autonomous driving according to claim 3, characterized in that: In the space module, local spatial features are captured using a graph convolution network, global spatial features are captured using a global spatial attention mechanism for distant traffic agents, and the overall spatial features are modeled using a combination of LSTM and spatial attention mechanism, and finally, a gating mechanism is used to fuse all spatial features to fully exploit the spatial dependency between traffic agents. Specifically, it includes: Local space modeling: a graph convolution network is used to capture the local spatial relationship between traffic agents. In the traffic scene, the diffusion convolution at time t is represented as: where H t ∈ R N×D represents the graph features input at time step t, L t ∈ R N×D represents the graph features output after processing at time step t, P k represents the power series of the transition matrix, W k1 ,W k2 are learnable parameters, K is the diffusion step length; the distance adjacency matrix and the heading angle adjacency matrix are used as inputs, rowsum() represents the sum operation on the row values of the matrix; the graph convolution is used to aggregate the transformed feature information from different neighborhood orders; the output result L t of the diffusion convolution is the spatial information aggregated by the target traffic agent from the surrounding traffic agents; Global space modeling: a multi-head self-attention mechanism is used to capture global information: traffic features H given input at time step t t ∈R N×D subspace generated by linear transformation wherein are learnable parameters; Then, the attention score of traffic agent i and traffic agent j is obtained through multi-head scaled dot product. Finally, according to the obtained attention weight, the features of adjacent traffic agents are aggregated to the target traffic agent, which is represented as: G t = Concat(head1, head2, …, head n )W G (6) wherein n represents the number of attention heads, m represents the mth attention head, softmax represents an activation function, W G represents a weight matrix; G t represents the feature of each traffic agent after aggregation of global traffic agents Overall space modeling: a combination of LSTM and multi-head attention mechanism is used to model the overall spatial features of traffic agents: Traffic features are input into the LSTM, using the last hidden state H' ∈ R N×D as the time feature after fusing the respective historical time series of the traffic agents; i.e.: T s ,H' i =LSTM(H) (8) Feature H' i represents the historical features of the traffic agent, T S denotes all hidden states of the LSTM output; To capture the spatial features of traffic agents over the entire history time series, a multi-head attention mechanism is used; given the input features H' e R N×D , the subspace is generated by a linear transformation wherein are learnable parameters; Then, the attention score of traffic agent i and traffic agent j is obtained through multi-head scaled dot product. Finally, according to the obtained attention weight, the features of adjacent traffic agents are aggregated to the target traffic agent, which is represented as: O = Concat(headl, head2,..., head n )W O (10) wherein W O is a learnable parameter; O denotes the overall spatial features captured by the LSTM and the multi-head attention mechanism; Spatial gating fusion: in order to fuse the local spatial features L obtained by the graph convolution network, the global spatial features G obtained by the attention mechanism and the overall spatial features O, the sigmoid activation function is used to map the importance of the input features to a value between 0 and 1. Then, these weights are multiplied with the corresponding features to achieve the weighted fusion of different input features. First, the local and global features are fused using the spatial gating fusion mechanism as follows: z1 = σ(L · W 11 + G · W 12 + b g1 )(12) GL=z1⊙L+(1-z1)⊙G (13) In the above equation, σ denotes a sigmoid activation function, W 11 ,W 12 are learnable weights, and b g1 is a bias value, where denotes a bitwise multiplication. Then, the overall spatial features are fused using the gating mechanism: z2= σ (GL· W 21 + O· W 22 + b g2 ) (14) GLO=z2⊙GL+(1-z2)⊙O (15) GLO contains complex spatial features in the historical trajectory of the traffic agent.

6. The trajectory prediction method for autonomous driving according to claim 3, characterized in that: The trajectory prediction module for a given input time series and filter Γ = (f1, f2,..., f U ), the dilated causal convolution at x t is defined as: where d is the dilation rate, which indicates the distance between the convolution kernels; Each residual block is composed of two layers of two-dimensional convolution, WeightNorm normalization, ReLU activation function and Dropout. Two different TCNs are used to extract time dependency, and different sizes of convolution kernels are set for the two TCN networks, which are k1 and k2 respectively, and the dilation rate is set to d1 and d2 respectively. Finally, the results of the two different TCNs are spliced, and then two fully connected layers are used to obtain the final output. For a given ST out The trajectory prediction module is shown below: where Θ1, Θ2 are two independent TCN networks, FC denotes a fully connected layer; time T is obtained by TCN obs +1 to T obs +T pred Final future trajectory 7. The trajectory prediction method for automatic driving oriented multi-dimensional spatio-temporal feature fusion according to claim 3, characterized in that: L2-loss is used as the loss function of the model.

Images