A trajectory prediction method under scene fusion based on a spatiotemporal structure causal model

Abstract

The application discloses a scene fusion under track prediction method based on a space-time structure causal model, firstly, through the front door criterion, mixed paths in the feature extraction process are eliminated, and scene context is merged into a space-time state; then, a dynamic space-time causal relationship is embedded into a space-time causal graph constructed, and a space-time dependent relationship is extracted through causal convolution; finally, a counterfactual representation reasoning method is used, and space-time features of the track are deduced into a future traffic scene to perform track prediction. The application proposes a track prediction structure causal model based on causal reasoning, explains the real reason why scene context brings performance improvement in track prediction, analyzes mixed path interference limiting model performance, realizes dynamic interaction between track prediction and scenes, and has good robustness.

Description

Technical Field

[0001] This invention relates to the field of vehicle trajectory prediction in traffic management and intelligent navigation, and particularly to a trajectory prediction method based on scene fusion using a spatiotemporal structure causal model. Background Technology

[0002] In recent years, trajectory prediction has attracted widespread attention in the fields of mobility analytics and applications. With the rapid development of mobile devices and sensor technologies, trajectory prediction aims to predict future mobility patterns based on historical trajectory data. This technology has wide applications in various fields such as traffic management, intelligent navigation, autonomous driving, and logistics planning.

[0003] However, trajectory prediction faces many challenges due to the complexity and uncertainty of trajectory data. Fortunately, much work has been done on vehicle trajectory prediction, and coupled with the rich contextual scenarios provided by HD maps, trajectory prediction has made great progress.

[0004] Trajectory prediction methods can be broadly categorized into traditional trajectory prediction methods and data-driven trajectory prediction methods. The former utilizes probabilistic and statistical models to predict the probability of a target moving under different motion patterns, or directly uses kinematic models for prediction. The latter employs deep learning techniques to learn trajectory features and their interactive behaviors from a large amount of existing trajectory data, and then uses the learned model for prediction.

[0005] Deo et al. (Convolutional social pooling for vehicle trajectory prediction, 2018, 1468-1476) proposed using convolutional social pooling as an upgraded version of the social pooling layer on the LSTM encoder-decoder model to robustly learn the interdependencies of vehicle motion. Pecher et al. (Data-driven vehicle trajectory prediction, 2016, 13-22) proposed a method to increase the order of Markov and neural network models to enhance the accuracy of trajectory prediction. Schmid et al. (Encoding hd maps and agent dynamics from vectorized representation, 2020, 11525-11533) proposed VectorNet, introducing vectorized maps into trajectory prediction, and a large amount of work began to focus on vector maps. M. Wang et al. (Goal Area Network for MotionForecasting, 2023) enhanced future interaction information by designing information fusion in a small area near the endpoints. Zikang Zhou et al. (Query-Centric Trajectory Prediction, 2023, 17863-17873) proposed a query-centric global spatiotemporal representation. Gilles et al. (Graph-oriented heatmap output for future motion estimation, 2022, 9107-9114) proposed a method to encode and decode historical and local contexts to output a heatmap. GoRela et al. (Go Relative for Viewpoint-Invariant Motion Forecasting, 2022) proposed a pairwise relative position encoding method, which, being viewpoint invariant, makes learning more efficient. Sakaridis et al. (Real-Time Motion Prediction via Heterogeneous PolylineTransformer with RelativePose Encoding, 2023) proposed a method that leverages the powerful representations of Transformers to enhance the performance of pairwise position representations. Park et al. (Leveraging FutureRelationship Reasoning for Vehicle Trajectory Prediction, 2022) and others focus more on future vehicle interactions.

[0006] While these models improve predictive power by integrating historical scene context into trajectory data, they lack formal explanations, making it impossible to identify the root causes limiting model performance. Therefore, designing trajectory prediction methods that can explain the true reasons for the performance improvements brought about by scene context in trajectory prediction and analyze the interference of cluttered paths that limit model performance has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide a trajectory prediction method based on a spatiotemporal structural causal model for scene fusion. Causal reasoning is introduced into trajectory prediction, and a front-door criterion based on causal intervention is applied to eliminate confounding biases in the feature extraction process, ensuring that the model can more fairly incorporate the context of each scene into the spatiotemporal features of the trajectory data.

[0008] The technical solution adopted in this invention is:

[0009] A trajectory prediction method based on a spatiotemporal structural causal model for scene fusion, comprising the following steps:

[0010] S1. Define a structural causal model to eliminate cluttered paths in the feature extraction process using the front-door criterion;

[0011] S2. Construct a spatiotemporal causal graph embedding dynamic spatiotemporal causal relationships, and extract spatiotemporal causal dependencies through a spatiotemporal causal graph convolution;

[0012] S3. Using counterfactual reasoning representation methods, the spatiotemporal characteristics of the trajectory are extrapolated to future traffic scenarios to predict the trajectory and obtain the prediction results.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] S1-1. Define the structural causal model, and obtain the four paths of the structural causal graph as follows:

[0015] X t →S t →H t →Y t+1

[0016] X t →T t →H t →Y t+1

[0017] X t ←C t →H t →Y t+1

[0018] Ht ←C t+1 →Y t+1

[0019] Where X represents trajectory data from the dataset, C represents external factors such as scene context, S represents trajectory spatial features, T represents trajectory temporal features, H represents highly coupled trajectory spatiotemporal features, Y represents the model's prediction result, and the subscript t represents the current time.

[0020] S1-2. Define the trajectory prediction task. Predict a segment of the trajectory of a future vehicle as follows:

[0021] P(Y|X t )=∑ H P(Y|H)P(H|X) (1);

[0022] Wherein, P(H|X) represents the process of the encoder extracting spatiotemporal causal features, and P(Y|H) represents the trajectory decoding process of the decoder;

[0023] S1-3. Introduce the front door criterion to cut off the false correlation between the trajectory input data X and the spatiotemporal features H, and perform causal intervention operations on variable X to eliminate the confounding paths in the feature extraction process.

[0024] Furthermore, the process of extracting spatiotemporal features from the model after performing causal intervention on variable X in steps S1-3 is represented as follows:

[0025]

[0026] Wherein, P(X′) t P(S) represents intervention in X. t ,T t |X t P(H) represents the process of extracting spatiotemporal features from noisy input data. t |S t ,T t ,X′ t ) represents the potential spatiotemporal state representation that changes over time from the features.

[0027] Furthermore, step S2 specifically includes the following steps:

[0028] S2-1. Introduce a feature fusion network to integrate scene context features into trajectory features, and use an LSTM network to embed the trajectory feature sequence into the corresponding feature embedding H. J ∈R N×D The specific expression is as follows:

[0029] H J =LSTM(Traj his)∈R N×D (3);

[0030] Among them, Traj his =J T J T-1 J1 represents the historical trajectory sequence of the autonomous vehicle, J T (P, H, V, ts) is a quadruple describing the position and state of an autonomous vehicle; P = {x, y} represents the vehicle's position coordinates at the current moment; H represents the vehicle's orientation at the current moment; V = {V X V Y} represents the instantaneous velocity V of the vehicle along the X direction at the current moment. X Instantaneous velocity V in the Y direction Y Size; ts is the timestamp describing the current moment; R N×D This represents an N×D real number field matrix, i.e., a trajectory sequence matrix; N represents the number of rows in the matrix, corresponding to the length of the trajectory sequence; D represents the dimension, i.e., each trajectory sequence is embedded in a D-dimensional vector space;

[0031] S2-2. The lane line node information is convolved and aggregated using the lane map convolution method. The left and right nodes are convolved only once, while the preceding and succeeding nodes are convolved k times. Therefore, the semantic information H(C) extracted from the map is represented as follows:

[0032]

[0033] Where C represents lane node characteristics, M represents the number of lane nodes, and A represents lane node features. pre It is an adjacency matrix representing the pre-type relationship; represents the k-th power of a matrix of type i; W is the trainable parameter; K represents the order of the dilated convolution; pre represents the predecessor, suc represents the successor, left represents the left neighbor, and right represents the right neighbor, and the four edge connection relationships represent the topological structure of the lane line nodes; R M×D A matrix representing the M×D real number field;

[0034] S2-3. Trajectory feature representation of vehicle i at each time step For road node j, attention fusion is used to integrate scene context features. The fused trajectory feature X′ is obtained by integrating it into the vehicle's trajectory features. i The specific expression is as follows:

[0035]

[0036] Where W0, W1, W2 are trainable parameters, Δ i，jσ is the Euclidean distance difference between nodes I and j, and σ is the operation of the nonlinear activation function ReLU.

[0037] S2-4. Extract the feature X′ at each time step. t After input, compare with the hidden features H from the previous time step. t-1 Concatenate them together, and then transform them to I through a fully connected layer. t :

[0038] I t =FC(X′) t ||H t-1 )∈R N×D (6);

[0039] Where, X′ t The feature representing the input at each time t is an N*D matrix; I t This represents the new feature representation obtained at each time t, which includes temporal causality; FC stands for fully connected layer, used to perform linear transformation on the input;

[0040] S2-5. Perform global average pooling on all features of each node to obtain the importance index sequence of the corresponding group of nodes. Then, pass it through a fully connected layer and a sigmoid function to obtain a gating factor α with each value in (0, 1). The calculation process is as follows:

[0041]

[0042] in, It is for I t Each node feature in the algorithm is averaged; d is the number of feature dimensions D.

[0043] I t [:,c] represents I t The c-th column in the table represents the value of each node in the c-th feature dimension at time t.

[0044] S2-6, Gating factor α and feature I t Performing the Hadamard element-wise product yields the dynamic spatiotemporal causal representation (DST) of the node at time t. t :

[0045] DST t =I t ⊙α∈R N×D (8);

[0046] S2-7. Embed dynamic causal relationships into dynamic spatiotemporal causal graphs through self-attention mechanisms;

[0047]

[0048] Where φ is the tanh function;

[0049] S2-8. The interaction between vehicles and the environment is represented by the spatiotemporal interaction diagram G. int The graph is represented as (V,E,A), where V is the set of vehicles and pedestrians, and each agent is considered a node; edge E indicates whether there is interaction between different agents; a bidirectional fully connected graph is constructed based on the interaction relationships between vehicles, as shown in the following expression:

[0050]

[0051] S2-9, Trajectory Spatiotemporal Causal Feature Map A dst Interactive graph A of trajectory spatiotemporal interaction int Construct a spatiotemporal causal interactive convolutional network to fuse causal and interactive features;

[0052] Furthermore, the specific execution steps of the spatiotemporal causal interactive convolutional network in S2-9 are as follows:

[0053] S2-9-1. At time t, input features Represented as:

[0054]

[0055] Where X′ is the output of the feature enhancement network, H t-1 It is a hidden feature passed from the previous moment to the current moment;

[0056] S2-9-2, Spatiotemporal causal interaction graph convolution is defined as:

[0057]

[0058] Where l represents the layer number; θ0, θ1, and θ2 are the three weight coefficients that can be learned.

[0059] Specifically, the computation of Formula 12 through the Spatiotemporal Causal Graph Convolutional Network (STCGCN) (integration and updating of information on trajectory and scene context) is similar in operation to the corresponding part of Formula (2) (product of various probability distributions of trajectory and scene context) in the model, mainly for constructing the inference and training process of the entire model.

[0060] S2-10. Train a linear network that extracts features from the output of the Spatiotemporal Causal Interaction Graph Convolutional Network (STCGCN) to restore the features to their original feature dimensions:

[0061]

[0062] Furthermore, in steps S2-8, the Euclidean distance between the agents is calculated based on their current coordinate positions. When the distance is less than a threshold, it is considered that there is an interaction relationship between the two vehicles or agents.

[0063] Furthermore, the threshold is set to 100.

[0064] Furthermore, the encoder is abstracted into a nested GRU network structure, namely a causal spatiotemporal autoencoder, where each encoder cell describes the complete causal intervention effect P(H). t |do(X t The feature enhancement network describes P(X′), and the spatiotemporal causal graph convolution describes... process.

[0065] Furthermore, the calculation process of the structural causal unit of the causal spatiotemporal autoencoder in step S2-10 is as follows:

[0066] r t =σ(θ) r＊G (X′ t ||H t-1 )+b r )

[0067] z t =σ(θ) z＊G (X′ t ||H t-1 )+b z )

[0068] h t =σ(θ) r＊G (X′ t ||r t ⊙H t-1 )+b h )

[0069] H t =z t ⊙H t-1 +(1-z t )⊙h t

[0070] Where ⊙ is the Hadama product, i.e., element-wise multiplication; θ r ,θ z ,θ h ,b r ,b z ,b h All are learnable parameters, *G is the GIGCN convolution process defined in the formula above; H t It is the spatiotemporal state of the structural causal unit at time t; r tIt is a gating variable used to control the information to be updated, called the reset gate; z t It is another gating variable used to control the extent to which new candidate information is updated to the current hidden state; this is called the update gate. t It is a candidate hidden state, calculated based on the current input and the reset gate.

[0071] Furthermore, the specific steps of step S3 are as follows:

[0072] S3-1, Utilizing trajectory data X in a real-world scenario t Calculate the counterfactual scenario C t+1 The following future trajectory is obtained:

[0073]

[0074] Among them, P(H′) t |X t C t+1 () indicates the counterfactual reasoning process. This refers to a prediction process based on counterfactual representations.

[0075] S3-2, Set the global scene context C corresponding to the current trajectory. g The similarity of the scene context at each road node is calculated using a self-attention mechanism; during the counterfactual reasoning process, the output of the encoder is transformed into the input of the decoder using an attention matrix; the formula for the counterfactual representation reasoning module is:

[0076] Q = C g W Q ∈R M×D ;

[0077] K = C g W K ∈R M×D ;

[0078] V = H his W V ;

[0079] S3-3, Calculate the input H of the decoder. pred :

[0080]

[0081] Where D is the feature dimension, W Q W K W V There are three sets of trainable parameters. It is a way of focusing attention.

[0082] S3-4. The decoder outputs the prediction result based on the representation transformation of trajectory features on the scene context nodes.

[0083] Furthermore, representation transformation refers to the transformation operation on trajectory features, converting the original trajectory features into a new representation on the scene context nodes in a certain way. Here, the encoder output is weighted by the attention matrix Cg to obtain new representations Q, K, and V. The purpose of this transformation is to combine trajectory features with the scene context for better prediction or generation in the decoder.

[0084] The present invention, employing the above technical solution, has the following beneficial effects: it proposes a causal model for trajectory prediction based on causal reasoning, explains the true reason why scene context brings performance improvement in trajectory prediction, and analyzes the interference of mixed paths that limits model performance. Experiments on two real-world datasets verify the effectiveness and robustness of this end-to-end method in real traffic scenarios, especially in scenarios involving interactions between multiple agents. Attached Figure Description

[0085] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0086] Figure 1 This is an overall flowchart of a trajectory prediction method based on a spatiotemporal structural causal model according to an embodiment of the present invention;

[0087] Figure 2 This is a flowchart illustrating the construction of a dynamic spatiotemporal causal graph in a trajectory prediction method based on a spatiotemporal structural causal model according to an embodiment of the present invention.

[0088] Figure 3 This is a schematic diagram of the framework of a trajectory prediction method based on a spatiotemporal structural causal model in scene fusion according to an embodiment of the present invention.

[0089] Implementation

[0090] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0091] like Figures 1 to 3 As shown in the figure, this invention discloses a trajectory prediction method based on a spatiotemporal structural causal model under scene fusion, which includes the following steps:

[0092] S1. Define a structural causal model to eliminate cluttered paths in the feature extraction process using the front-door criterion;

[0093] S2. Construct a spatiotemporal causal graph embedding dynamic spatiotemporal causal relationships, and extract spatiotemporal causal dependencies through a spatiotemporal causal graph convolution;

[0094] S3. Using counterfactual reasoning representation methods, the spatiotemporal characteristics of the trajectory are extrapolated to future traffic scenarios to predict the trajectory and obtain the prediction results.

[0095] Furthermore, this invention is based on the assumption that the temporal and spatial characteristics of a trajectory can be decoupled from spatiotemporal data features, and that they are fused to form a complete spatiotemporal feature representation of the trajectory. Step S1 specifically includes the following steps:

[0096] S1-1. Define the structural causal model, and obtain the four paths of the structural causal graph as follows:

[0097] X t →S t →H t →Y t+1

[0098] X t →T t →H t →Y t+1

[0099] X t ←C t →H t →Y t+1

[0100] H t ←C t+1 →Y t+1

[0101] Where X represents trajectory data from the dataset, C represents external factors such as scene context, S represents trajectory spatial features, T represents trajectory temporal features, H represents highly coupled trajectory spatiotemporal features, Y represents the model's prediction result, and the subscript t represents the current time.

[0102] S1-2. Define the trajectory prediction task. Predict a segment of the trajectory of a future vehicle as follows:

[0103] P(Y|X t )=∑ H P(Y|H)P(H|X) (1);

[0104] Wherein, P(H|X) represents the process of the encoder extracting spatiotemporal causal features, and P(Y|H) represents the trajectory decoding process of the decoder;

[0105] S1-3. Introduce the front door criterion to cut off the false correlation between the trajectory input data X and the spatiotemporal features H, and perform causal intervention operations on variable X to eliminate the confounding paths in the feature extraction process.

[0106] Furthermore, the process of extracting spatiotemporal features from the model after performing causal intervention on variable X in steps S1-3 is represented as follows:

[0107]

[0108] Wherein, P(X′) t P(S) represents intervention in X. t ,T t |X t P(H) represents the process of extracting spatiotemporal features from noisy input data. t |S t ,T t ,X′ t ) represents the potential spatiotemporal state representation that changes over time from the features.

[0109] Furthermore, step S2 specifically includes the following steps:

[0110] S2-1. Introduce a feature fusion network to integrate scene context features into trajectory features, and use an LSTM network to embed the trajectory feature sequence into the corresponding feature embedding H. J ∈R N×D The specific expression is as follows:

[0111] H J =LSTM(Traj his )∈R N×D (3);

[0112] Among them, Traj his =J T J T-1 J1 represents the historical trajectory sequence of the autonomous vehicle, J T (P, H, V, ts) is a quadruple describing the position and state of an autonomous vehicle; P = {x, y} represents the vehicle's position coordinates at the current moment; H represents the vehicle's orientation at the current moment; V = {V X V Y} represents the instantaneous velocity V of the vehicle along the X direction at the current moment. X Instantaneous velocity V in the Y direction Y Size; ts is the timestamp describing the current moment; R N×DThis represents an N×D real number field matrix, i.e., a trajectory sequence matrix; N represents the number of rows in the matrix, corresponding to the length of the trajectory sequence; D represents the dimension, i.e., each trajectory sequence is embedded in a D-dimensional vector space;

[0113] S2-2. The lane line node information is convolved and aggregated using the lane map convolution method. The left and right nodes are convolved only once, while the preceding and succeeding nodes are convolved k times. Therefore, the semantic information H(C) extracted from the map is represented as follows:

[0114]

[0115] Where C represents lane node characteristics, M represents the number of lane nodes, and A represents lane node features. pre It is an adjacency matrix representing the pre-type relationship; represents the k-th power of a matrix of type i; W is the trainable parameter; K represents the order of the dilated convolution; pre represents the predecessor, suc represents the successor, left represents the left neighbor, and right represents the right neighbor, and the four edge connection relationships represent the topological structure of the lane line nodes; R M×D A matrix representing the M×D real number field;

[0116] S2-3. Trajectory feature representation of vehicle i at each time step For road node j, attention fusion is used to integrate scene context features. The fused trajectory feature X′ is obtained by integrating it into the vehicle's trajectory features. i The specific expression is as follows:

[0117]

[0118] Where W0, W1, W2 are trainable parameters, Δ i，j σ is the Euclidean distance difference between nodes I and j, and σ is the operation of the nonlinear activation function ReLU.

[0119] S2-4. Extract the feature X′ at each time step. t After input, compare with the hidden features H from the previous time step. t-1 Concatenate them together, and then transform them to I through a fully connected layer. t :

[0120]

[0121] Where, X′ t The feature representing the input at each time t is an N*D matrix; I t This represents the new feature representation obtained at each time t, which includes temporal causality; FC stands for fully connected layer, used to perform linear transformation on the input;

[0122] S2-5. Perform global average pooling on all features of each node to obtain the importance index sequence of the corresponding group of nodes. Then, pass it through a fully connected layer and a sigmoid function to obtain a gating factor α with each value in (0, 1). The calculation process is as follows:

[0123]

[0124] in, It is for I t Each node feature in the algorithm is averaged; d is the number of feature dimensions D.

[0125] I t [:,c] represents I t The c-th column in the table represents the value of each node in the c-th feature dimension at time t.

[0126] S2-6, Gating factor α and feature I t Performing the Hadamard element-wise product yields the dynamic spatiotemporal causal representation (DST) of the node at time t. t :

[0127] DST t =I t ⊙α∈R N×D (8);

[0128] S2-7. Embed dynamic causal relationships into dynamic spatiotemporal causal graphs through self-attention mechanisms;

[0129]

[0130] Where φ is the tanh function;

[0131] S2-8. The interaction between vehicles and the environment is represented by the spatiotemporal interaction diagram G. int The graph is represented as (V,E,A), where V is the set of vehicles and pedestrians, and each agent is considered a node; edge E indicates whether there is interaction between different agents; a bidirectional fully connected graph is constructed based on the interaction relationships between vehicles, as shown in the following expression:

[0132]

[0133] S2-9, Trajectory Spatiotemporal Causal Feature Map A dst Interactive graph A of trajectory spatiotemporal interaction int Construct a spatiotemporal causal interactive convolutional network to fuse causal and interactive features;

[0134] Furthermore, the specific execution steps of the spatiotemporal causal interactive convolutional network in S2-9 are as follows:

[0135] S2-9-1. At time t, input features Represented as:

[0136]

[0137] Where X′ is the output of the feature enhancement network, H t-1 It is a hidden feature passed from the previous moment to the current moment;

[0138] S2-9-2, Spatiotemporal causal interaction graph convolution is defined as:

[0139]

[0140] Where l represents the layer number; θ0, θ1, and θ2 are the three weight coefficients that can be learned.

[0141] Specifically, the computation of Formula 12 through the Spatiotemporal Causal Graph Convolutional Network (STCGCN) (integration and updating of information on trajectory and scene context) is similar in operation to the corresponding part of Formula (2) (product of various probability distributions of trajectory and scene context) in the model, mainly for constructing the inference and training process of the entire model.

[0142] S2-10. Train a linear network that extracts features from the output of the Spatiotemporal Causal Interaction Graph Convolutional Network (STCGCN) to restore the features to their original feature dimensions:

[0143]

[0144] Furthermore, in steps S2-8, the Euclidean distance between the agents is calculated based on their current coordinate positions. When the distance is less than a threshold, it is considered that there is an interaction relationship between the two vehicles or agents.

[0145] Furthermore, the threshold is set to 100.

[0146] Furthermore, the encoder is abstracted into a nested GRU network structure, namely a causal spatiotemporal autoencoder, where each encoder cell describes the complete causal intervention effect P(H). t |do(X t The feature enhancement network describes P(X′), and the spatiotemporal causal graph convolution describes... process.

[0147] Furthermore, the calculation process of the structural causal unit of the causal spatiotemporal autoencoder in step S2-10 is as follows:

[0148] r t =σ(θ) r＊G (X′ t ||Ht-1 )+b r )

[0149] z t =σ(θ) z＊G (X′ t ||H t-1 )+b z )

[0150] h t =σ(θ) r＊G (X′ t ||r t ⊙H t-1 )+b h )

[0151] H t =z t ⊙H t-1 +(1-z t )⊙h t

[0152] Where ⊙ is the Hadama product, i.e., element-wise multiplication; θ r ,θ z ,θ h ,b r ,b z ,b h All are learnable parameters, *G is the GIGCN convolution process defined in the formula above; H t It is the spatiotemporal state of the structural causal unit at time t; r t It is a gating variable used to control the information to be updated, called the reset gate; z t It is another gating variable used to control the extent to which new candidate information is updated to the current hidden state; this is called the update gate. t It is a candidate hidden state, calculated based on the current input and the reset gate.

[0153] Furthermore, the specific steps of step S3 are as follows:

[0154] S3-1, Utilizing trajectory data X in a real-world scenario t Calculate the counterfactual scenario C t+1 The following future trajectory is obtained:

[0155]

[0156] Among them, P(H′) t |X t C t+1 () indicates the counterfactual reasoning process. This refers to a prediction process based on counterfactual representations.

[0157] S3-2, Set the global scene context C corresponding to the current trajectory. g The similarity of the scene context at each road node is calculated using a self-attention mechanism; during the counterfactual reasoning process, the output of the encoder is transformed into the input of the decoder using an attention matrix; the formula for the counterfactual representation reasoning module is:

[0158] Q = C g W Q ∈R M×D ;

[0159] K = C g W K ∈R M×D ;

[0160] V = H his W V ;

[0161] S3-3, Calculate the input H of the decoder. pred :

[0162]

[0163] Where D is the feature dimension, W Q W K W V There are three sets of trainable parameters. It is a way of focusing attention.

[0164] S3-4. The decoder outputs the prediction result based on the representation transformation of trajectory features on the scene context nodes.

[0165] Furthermore, representation transformation refers to the transformation operation on trajectory features, converting the original trajectory features into a new representation on the scene context nodes in a certain way. Here, the encoder output is weighted by the attention matrix Cg to obtain new representations Q, K, and V. The purpose of this transformation is to combine trajectory features with the scene context for better prediction or generation in the decoder.

[0166] As described above, the beneficial effects of this invention are as follows: it proposes a causal model for trajectory prediction based on causal reasoning, explains the real reason why scene context brings performance improvement in trajectory prediction, and analyzes the interference of mixed paths that limits model performance. Experiments on two real-world datasets verify the effectiveness and robustness of this end-to-end method in real traffic scenarios, especially in scenarios involving interactions between multiple agents.

[0167] The working principle of this invention will be explained in detail below:

[0168] Please refer to Figure 1 Embodiment 1 of the present invention is as follows:

[0169] A trajectory prediction method based on a spatiotemporal structural causal model for scene fusion is presented in this embodiment. This method utilizes a causal reasoning-based structural causal model for trajectory prediction, employing the front-door criterion to decompose the spatiotemporal feature extraction process into several sub-problems, and proposing specific modules to solve these sub-problems respectively. Furthermore, the dynamic spatiotemporal causal effects in the trajectory are embedded into the causal graph, and finally, counterfactual representation reasoning is used to improve the model's performance on future prediction tasks.

[0170] like Figure 1 As shown in this embodiment, a trajectory prediction method based on a spatiotemporal structural causal model under scene fusion includes the following steps:

[0171] S1. Define a structural causal model to eliminate confounding paths in the feature extraction process through the front-door criterion.

[0172] S2. Construct a spatiotemporal causal graph embedding dynamic spatiotemporal causal relationships, and extract spatiotemporal causal dependencies through a spatiotemporal causal graph convolution.

[0173] S3. Using counterfactual reasoning representation methods, the spatiotemporal characteristics of the trajectory are extrapolated to future traffic scenarios for trajectory prediction.

[0174] In this embodiment, a front-door criterion based on causal intervention is applied to eliminate confounding biases in the feature extraction process, ensuring that the model can more fairly incorporate the context of each scenario into the spatiotemporal features of the trajectory data. A novel spatiotemporal structural causal model is used to extract spatiotemporal causal features from the trajectory data. By employing a counterfactual reasoning representation method, the features of historical trajectories in factual scenarios are extrapolated to future scenario contexts to enhance the model's understanding of future scenario contexts and further improve the model's predictive performance.

[0175] Embodiment 2 of the present invention is as follows:

[0176] Based on the above embodiment one, in the scene fusion trajectory prediction method based on the spatiotemporal structure causal model of this embodiment, the present invention is based on the following assumption: the temporal and spatial features of the trajectory can be decoupled from the spatiotemporal data features, and they are fused to form a complete spatiotemporal feature representing the trajectory.

[0177] In step S1 of the above embodiment, a structural causal model is defined, and the front-door criterion is used to eliminate confounding paths in the feature extraction process. Specifically, this includes the following steps:

[0178] S1-1. Define the structural causal model. Here, X represents trajectory data from the dataset, C represents external factors such as scene context, S represents trajectory spatial features, T represents trajectory temporal features, H represents highly coupled trajectory spatiotemporal features, Y represents the model's prediction result, and the subscript t represents the current time.

[0179] The defined structural cause-effect graph has the following four paths:

[0180] X t →S t →H t →Y t+1

[0181] X t →T t →H t →Y t+1

[0182] X t ←C t →H t →Y t+1

[0183] H t ←C t+1 →Y t+1

[0184] S1-2. Define the trajectory prediction task: predict a segment of the trajectory of a future vehicle, which can be represented as:

[0185] P(Y|X t )=∑ H P(Y|H)P(H|X) (1);

[0186] Wherein, P(H|X) represents the process of the encoder extracting spatiotemporal causal features, and P(Y|H) represents the trajectory decoding process of the decoder.

[0187] S1-3. Performing causal intervention on variable X, the process of the model extracting spatiotemporal features is represented as follows:

[0188]

[0189] Wherein, P(X′) t P(S) represents intervention in X. t ,T t |X t P(H) represents the process of extracting spatiotemporal features from noisy input data. t |S t ,T t ,X′ t ) represents the potential spatiotemporal state representation that changes over time from the features.

[0190] In this embodiment, the front-door criterion of causal intervention is applied to eliminate confounding biases in the feature extraction process, ensuring that the model can more fairly incorporate the context of each scene into the spatiotemporal features of the trajectory data.

[0191] Please refer to Figures 2 to 3 Embodiment 3 of the present invention is as follows:

[0192] Building upon Embodiment 2 above, this embodiment constructs a spatiotemporal causal graph that embeds dynamic spatiotemporal causal relationships, and extracts spatiotemporal causal dependencies through a spatiotemporal causal graph convolution. Furthermore, it utilizes a counterfactual reasoning representation method to extrapolate the features of the historical trajectory of a factual scenario into the context of a future scenario. Specifically, it includes the following steps:

[0193] S2-1. Introduce a feature fusion network to integrate scene context features into trajectory features, and then use an LSTM network to embed the trajectory feature sequence into its features, obtaining its feature embedding H. J ∈R N×D :

[0194] H J =LSTM(Traj his )∈R N×D (3);

[0195] Among them, Traj his =J T J T-1 J1 represents the historical trajectory sequence of the autonomous vehicle, J T (P, H, V, ts) is a quadruple describing the position and state of an autonomous vehicle; P = {x, y} represents the vehicle's position coordinates at the current moment; H represents the vehicle's orientation at the current moment; V = {V X V Y} represents the instantaneous speed of the vehicle along the X and Y directions at the current moment; ts is the timestamp describing the current moment.

[0196] S2-2. The Lane Graph Convolutional Network (LaneGCN) method is used to convolve and aggregate the lane line node information. The left and right nodes are convolved only once, while the preceding and succeeding nodes are convolved k times. Therefore, the semantic information H(C) extracted from the map is represented as follows:

[0197]

[0198] Where C represents lane node characteristics, M represents the number of lane nodes, and A represents lane node features. pre It is an adjacency matrix representing the pre-type relationship; represents the k-th power of a matrix of type i; W is the trainable parameter; K represents the order of the dilated convolution; pre, suc, left, and right neighbors.

[0199] Four edge connection relationships represent the topology of lane line nodes.

[0200] S2-3. Trajectory feature representation of vehicle i at each time step Consider road nodes j within a certain range, and use attention fusion to integrate scene context features. It is incorporated into the vehicle's trajectory characteristics as X′ i :

[0201]

[0202] Where W0, W1, W2 are trainable parameters, Δ i,j σ is the Euclidean distance difference between nodes i and j, and σ is the operation of the nonlinear activation function ReLU.

[0203] S2-4. Extract the feature X′ at each time step. t After input, compare with the hidden features H from the previous time step. t-1 Concatenate them together, and then transform them to I through a fully connected layer. t :

[0204] I t =FC(X′) t ||H t-1 )∈R N×D (6);

[0205] S2-5. Perform global average pooling on all features of each node to obtain the importance index sequence of this set of nodes. Then, pass it through a fully connected layer and a sigmoid function to obtain a gating factor α with each value in (0,1). The calculation process is as follows:

[0206]

[0207] S2-6, Gating factor α and feature I t Performing the Hadamard element-wise product yields the dynamic spatiotemporal causal representation (DST) of the node at time t. t :

[0208] DST t =I t ⊙α∈R N×D (8);

[0209] S2-7. Embed dynamic causal relationships into dynamic spatiotemporal causal graphs through self-attention mechanisms;

[0210]

[0211] Where φ is the tanh function. Figure 2 The above process was demonstrated;

[0212] S2-8. Calculate the Euclidean distance between agents based on their current coordinates. If the distance is less than a threshold, an interaction is considered to exist between the two vehicles or agents. In practice, this threshold is set to 100. For each subgraph, it is assumed that all vehicles within it have an interaction relationship; therefore, a bidirectional fully connected graph is constructed.

[0213]

[0214] S2-9. In this embodiment, the trajectory spatiotemporal causal feature map A is obtained. dst Interactive graph A of trajectory spatiotemporal interaction int Therefore, by integrating causal and interactive features, a novel spatiotemporal causal interactive convolutional network is proposed. The specific steps are as follows:

[0215] S2-9-1. At time t, input features It can be represented as:

[0216]

[0217] Where X′ is the output of the feature enhancement network, H t-1 It is a hidden feature passed from the previous moment to the current moment;

[0218] S2-9-2, Spatiotemporal causal interaction graph convolution is defined as:

[0219]

[0220] Where l represents the layer number; θ0, θ1, and θ2 are the three weight coefficients that can be learned.

[0221] S2-10. Train a linear network, further extract the features from the CIGCN output, and restore them to their original feature dimensions:

[0222]

[0223] In this embodiment, the linear network of S2-10 employs multiple stacked Structural Causal Units (SCUs) to form an encoder-decoder structure. Map information representing the scene context and historical trajectory sequences are input into the encoder, and the spatiotemporal causal features extracted by the encoder are sent to the counterfactual reasoning module. The calculation process of the Structural Causal Unit (SUC) of the causal spatiotemporal autoencoder is as follows:

[0224] rt =σ(θ) r＊G (X′ t ||H t-1 )+b r )

[0225] z t =σ(θ) z＊G (X′ t ||H t-1 )+b z )

[0226] h t =σ(θ) r＊G (X′ t ||r t ⊙)+b h )

[0227] H t =z t ⊙H t-1 +(1-z t )⊙h t

[0228] Here, ⊙ represents the Hadamard product, which is element-wise multiplication. θ r ,θ z ,θ h ,b r ,b z ,b h These are all learnable parameters, and *G is the STCGCN convolution process defined in the formula above. H t It is the spatiotemporal state of the structural causal unit (SCU) at time t.

[0229] In this embodiment, step S3 further includes:

[0230] We utilize counterfactual reasoning to focus more on trajectory features where the current scenario more closely resembles the future scenario. The specific steps are as follows:

[0231] S3-1, Utilizing trajectory data X in a real-world scenario t Calculate the counterfactual scenario C t+1 The following future trajectory can be obtained:

[0232]

[0233] Among them, P(H′) t |X t C t+1 () indicates the counterfactual reasoning process. This refers to a prediction process based on counterfactual representations.

[0234] S3-2, Set the global scene context C corresponding to the current trajectory.g The similarity of the scene context at each road node is calculated using a self-attention mechanism. During counterfactual reasoning, this attention matrix is ​​used to transform the encoder's output into the decoder's input. The formula for the counterfactual representation reasoning module is:

[0235] Q = C g W Q ∈R M×D ;

[0236] K = C g W K ∈R M×D ;

[0237] V = H his W V ;

[0238] S3-3, The input H of the decoder is obtained from the above steps. pred :

[0239]

[0240] Where D is the feature dimension, W Q W K W V There are three sets of trainable parameters. It is a way of focusing attention.

[0241] S3-4 At this point, the representation transformation of the trajectory features on the scene context nodes is obtained, and the decoder outputs the prediction result.

[0242] In this embodiment, a novel spatiotemporal structural causal model is proposed to extract spatiotemporal causal features from trajectory data. The dynamic spatiotemporal causal effects in the trajectory are embedded into the causal graph to improve the representation capability of the causal autoencoder. Then, counterfactual representation reasoning is used to improve the model's performance on future prediction tasks.

[0243] In summary, the present invention has the following beneficial effects:

[0244] (1) In terms of trajectory prediction: The vehicle trajectory predicted by this invention can take into account the interaction of other vehicles, such as the deceleration to avoid collisions, resulting in a shorter trajectory. In complex scenarios, it achieves a good understanding of the scene context and the dynamic interaction between vehicles, and provides multimodal prediction results, which plays a very good role in safe driving.

[0245] (2) Performance analysis of the method: The end-to-end method proposed in this invention has good robustness and effectiveness in real traffic scenarios, especially in the scenario of interaction between multiple agents.

[0246] Obviously, the described embodiments are only a portion, not all, of the embodiments of this application. Without conflict, the embodiments and features described and illustrated herein can be combined with each other. The components of the embodiments of this application generally described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. A trajectory prediction method based on a spatiotemporal structural causal model under scene fusion, characterized in that: It includes the following steps: S1. Define a structural causal model to eliminate cluttered paths in the feature extraction process using the front-door criterion; S2. Construct a spatiotemporal causal graph embedding dynamic spatiotemporal causal relationships, and extract spatiotemporal causal dependencies through a spatiotemporal causal graph convolution; specifically including the following steps: S2-1. Introduce a feature fusion network to integrate scene context features into trajectory features, and use an LSTM network to embed the trajectory feature sequence into the corresponding feature embedding. ∈ The specific expression is as follows: (3)； in, This is represented as a historical trajectory sequence of autonomous vehicles. It is a quadruple that describes the position and state of an autonomous vehicle; This indicates the vehicle's current position coordinates; Indicates the vehicle's orientation at the current moment; This represents the instantaneous velocity of the vehicle along the X direction at the current moment. Instantaneous velocity in the Y direction size; It is a timestamp describing the current moment; express A matrix in the real number field, i.e., a trajectory sequence matrix; N represents the number of rows in the matrix, corresponding to the length of the trajectory sequence; D represents the dimension, i.e., each trajectory sequence is embedded in a D-dimensional vector space; S2-2. The lane line node information is convolved and aggregated using the lane map convolution method. The left and right nodes are convolved only once, while the preceding and succeeding nodes are convolved k times. Therefore, the semantic information H(C) extracted from the map is represented as follows: (4)； in, It is a lane node feature, M is the number of lane nodes, A pre It is an adjacency matrix representing the pre-type relationship; is the k-th power of a matrix of type i; W is the trainable parameter; K represents the order of the dilated convolution; pre represents the predecessor, suc represents the successor, left represents the left neighbor and right represents the right neighbor, and the four edge connection relationships represent the topology of the lane line nodes; express A matrix in the real number field; S2-3. Trajectory feature representation of vehicle i at each time step For road node j, attention fusion is used to integrate scene context features. The fused trajectory features are obtained by integrating them into the vehicle's trajectory features. The specific expression is as follows: in, These are trainable parameters. σ is the Euclidean distance difference between nodes I and j, and σ is the operation of the nonlinear activation function ReLU. S2-4. Features of each time step After input, the hidden features compared to the previous time step. Concatenate them together, and then transform them through a fully connected layer. : (6)； in, The feature representing the input at each time t is an N*D matrix; This represents the new feature representation obtained at each time t, which includes temporal causality; FC stands for fully connected layer, used to perform linear transformation on the input; S2-5. Perform global average pooling on all features of each node to obtain the importance index sequence of the corresponding group of nodes. Then, pass it through a fully connected layer and a sigmoid function to obtain a gating factor with each value in (0, 1). The calculation process is as follows: ))∈ (7)； in, Yes Each node feature in the dataset is averaged and pooled. It is the number of feature dimensions D; express The c-th column in the table represents the value of each node in the c-th feature dimension at time t. S2-6, Gating Factor and characteristics Performing the Hadamard element-wise product yields the dynamic spatiotemporal causal representation of the node at time t. : (8)； S2-7. Embed dynamic causal relationships into dynamic spatiotemporal causal graphs through self-attention mechanisms; (9)； in, It is the tanh function; S2-8. The interaction between vehicles and the environment is represented by a spatiotemporal interaction diagram. To indicate, among which The set consists of vehicles and pedestrians, with each agent considered a node; edges This indicates whether there are interactive behaviors between different intelligent agents; a bidirectional fully connected graph is constructed based on the interaction relationships between vehicles, as shown in the following expression: (10)； S2-9, Based on trajectory spatiotemporal causal feature map Interactive graph of trajectory spatiotemporal Construct a spatiotemporal causal interactive convolutional network to fuse causal and interactive features; Furthermore, the specific execution steps of the spatiotemporal causal interactive convolutional network in S2-9 are as follows: S2-9-1. At time t, input features Represented as: (11)； in, It is the output of the feature enhancement network. It is a hidden feature passed from the previous moment to the current moment; S2-9-2, Spatiotemporal causal interaction graph convolution is defined as: (12)； in, Indicates the number of layers; , , The three weighting coefficients are learnable parameters; S2-10. Train a linear network that extracts features from the output of the Spatiotemporal Causal Interaction Graph Convolutional Network (STCGCN) to restore the features to their original feature dimensions: =ReLU( )∈ = || ) (13); S3. Using counterfactual reasoning representation methods, the spatiotemporal characteristics of the trajectory are extrapolated to future traffic scenarios to predict the trajectory and obtain the prediction results.

2. The trajectory prediction method based on a spatiotemporal structural causal model under scene fusion according to claim 1, characterized in that: Step S1 specifically includes the following steps: S1-1. Define the structural causal model, and obtain the four paths of the structural causal graph as follows: ； ； ； ； Where X represents trajectory data from the dataset, C represents external factors such as scene context, S represents trajectory spatial features, T represents trajectory temporal features, H represents highly coupled trajectory spatiotemporal features, Y represents the model's prediction result, and the subscript t represents the current time. S1-2. Define the trajectory prediction task. Predict a segment of the trajectory of a future vehicle as follows: (1)； in, This represents the process by which the encoder extracts spatiotemporal causal features. This represents the trajectory decoding process of the decoder; S1-3. Introduce the front door criterion to cut off the false correlation between the trajectory input data X and the spatiotemporal features H, and perform causal intervention operations on variable X to eliminate the confounding paths in the feature extraction process.

3. The trajectory prediction method based on a spatiotemporal structural causal model under scene fusion according to claim 2, characterized in that: The process of extracting spatiotemporal features from the model after performing causal intervention on variable X in steps S1-3 is represented as follows: (2); in, Indicates intervention in X This refers to the process of extracting spatiotemporal features from noisy input data. This represents the potential spatiotemporal state representation that changes over time from the features.

4. The trajectory prediction method based on a spatiotemporal structural causal model under scene fusion according to claim 1, characterized in that: In steps S2-8, the Euclidean distance between the agents is calculated based on their current coordinate positions. When the distance is less than a threshold, it is considered that there is an interaction relationship between the two vehicles or agents.

5. The trajectory prediction method based on a spatiotemporal structural causal model under scene fusion according to claim 4, characterized in that: The threshold is set to 100.

6. The trajectory prediction method based on a spatiotemporal structural causal model under scene fusion according to claim 1, characterized in that: The encoder is abstracted as a nested GRU network structure, namely a causal spatiotemporal autoencoder, where each encoder cell describes the complete causal intervention effect P. The feature enhancement network describes Spatiotemporal causal graph convolution description process.

7. A trajectory prediction method based on a spatiotemporal structural causal model under scene fusion as described in claim 1 or 6, characterized in that: The calculation process of the structural causal unit of the causal spatiotemporal autoencoder in step S2-10 is as follows: =σ( || )+ ); =σ( || )+ ); =σ( || )+ ); = +(1- ) ； in, It is the Hadama product, which is element-wise multiplication; All of these are learnable parameters. It is the STCGCN convolution process defined by the formula above; It is the spatiotemporal state of the structural causal unit at time t; It is a gated variable used to control the information to be updated, and is called the reset gate; It is another gating variable used to control the extent to which new candidate information is updated to the current hidden state; it is called the update gate. It is a candidate hidden state, calculated based on the current input and the reset gate.

8. The trajectory prediction method based on a spatiotemporal structural causal model under scene fusion according to claim 1, characterized in that: The specific steps of step S3 are as follows: S3-1, Utilizing trajectory data in real-world scenarios Calculate counterfactual scenarios The following future trajectory is obtained: ) (14)； in, This indicates the counterfactual reasoning process. This indicates a prediction process based on counterfactual representations; S3-2, Set the global scene context corresponding to the current trajectory. The similarity of the scene context at each road node is calculated using a self-attention mechanism; during the counterfactual reasoning process, the output of the encoder is transformed into the input of the decoder using an attention matrix; the formula for the counterfactual representation reasoning module is: ； ； ； S3-3, Calculate the input to the decoder. : Where D is the feature dimension. There are three sets of trainable parameters. It is a way of focusing attention; S3-4. The decoder outputs the prediction result based on the representation transformation of trajectory features on the scene context nodes.

Images