A trajectory prediction method based on spatio-temporal interaction feature fusion

Abstract

The application provides a trajectory prediction method based on space-time interaction feature fusion, comprising the following steps: collecting traffic participant information on a road in real time; constructing a traffic scene expression model based on the traffic participant information; constructing a multi-scale space-time graph convolution model, so that the multi-scale space-time graph convolution model extracts space-time interaction features of the traffic participant based on the traffic scene expression model; constructing a trajectory prediction model, so that the trajectory prediction model performs trajectory prediction based on the space-time interaction features to obtain a predicted trajectory. The trajectory prediction method based on space-time interaction feature fusion provided by the application can capture the spatial correlation of the behavior of the traffic participant at different time scales by constructing a traffic scene expression and extracting space-time interaction features of the traffic participant, and can effectively improve the accuracy of trajectory prediction by extracting multi-scale space-time fusion interaction features.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a trajectory prediction method based on spatiotemporal interactive feature fusion. Background Art

[0002] Accurately predicting the future trajectories of surrounding traffic participants is crucial for intelligent driving systems. It's a prerequisite for intelligent vehicles to engage in rational and efficient behavioral interactions with other participants and perform collision detection. In complex traffic scenarios, the future behavior of surrounding traffic participants is highly uncertain, making it difficult for intelligent driving systems to represent this highly uncertain future behavior with a single, deterministic predicted trajectory. Therefore, each decision cycle of the intelligent driving system requires not only collision risk detection based on the most likely behavior of surrounding traffic participants, but also a comprehensive consideration of the probability distribution of all possible future behaviors.

[0003] In order to fully and realistically describe the highly uncertain future behavior of surrounding traffic participants, some recent technical studies have achieved multimodal trajectory prediction of surrounding traffic participants by considering the road topology structure. However, due to the small number of predicted trajectory modes, it still cannot fully describe the highly uncertain future behavior of surrounding traffic participants. At the same time, many methods lack consideration of vehicle kinematics, and the generated trajectories sometimes do not conform to the vehicle kinematic constraints, making it difficult to provide accurate surrounding obstacle motion information for subsequent decision-making and planning links.

[0004] Many prediction methods have begun to improve trajectory prediction accuracy by modeling the behavioral interactions between traffic participants. However, existing trajectory prediction methods fail to adequately model the spatial and temporal correlations of these interactions, resulting in low trajectory prediction accuracy. Most of these trajectory prediction methods only extract behavioral interactions from spatial correlations, paying little attention to the temporal correlations of these interactions. While other approaches model both spatial and temporal correlations, they do so separately, neglecting to capture more spatial correlations at different time scales. Summary of the Invention

[0005] The present invention aims to provide a trajectory prediction method based on the fusion of spatiotemporal interaction features to solve the above technical problems. By constructing a traffic scene representation and extracting the spatiotemporal interaction features of traffic participants, the spatial correlation of traffic participants' behaviors at different time scales can be captured, effectively improving the accuracy of trajectory prediction.

[0006] In order to solve the above technical problems, the present invention provides a trajectory prediction method based on spatiotemporal interactive feature fusion, comprising the following steps:

[0007] Collect information about traffic participants on the road in real time;

[0008] Construct a traffic scene expression model based on traffic participant information;

[0009] Construct a multi-scale spatiotemporal graph convolution model to extract the spatiotemporal interaction features of traffic participants based on the traffic scene expression model;

[0010] Construct a trajectory prediction model to enable the trajectory prediction model to perform trajectory prediction based on spatiotemporal interaction features and obtain a predicted trajectory.

[0011] The above scheme constructs a traffic scene representation and extracts the spatiotemporal interaction characteristics of traffic participants, so that the final vehicle prediction trajectory can capture the spatial correlation of traffic participants' behaviors at different time scales, extract multi-scale spatiotemporal fusion interaction features, and effectively improve the accuracy of trajectory prediction.

[0012] Furthermore, the traffic scene expression model includes a Euclidean distance adjacency matrix, a potential collision adjacency matrix, and an adaptive adjacency matrix; and constructing the traffic scene expression model based on traffic participant information includes:

[0013] Based on the traffic participant information, the Euclidean distance adjacency matrix is ​​constructed, which is specifically expressed as:

[0014]

[0015] Where, is the Euclidean distance adjacency matrix An element in , which represents the adjacency relationship between traffic participants i and j at time t; represents the distance between traffic participants i and j at time t; D close Represents the distance threshold between neighbor participants; 1 indicates that traffic participant i and traffic participant j are neighbors at time t; 0 indicates that traffic participant i and traffic participant j are not neighbors at time t.

[0016] It should be noted that once a traffic participant and a vehicle are identified as neighbors, the behavior of the traffic participant will be considered to have an impact on the vehicle trajectory in the subsequent prediction stage. In real traffic, the distance threshold D close The behavior of traffic participants outside the vehicle still has an impact on the vehicle. Therefore, in order to solve the problem that the fixed distance threshold of the Euclidean distance adjacency matrix cannot be dynamically adjusted, this solution obtains the current speed and direction of the traffic participant based on the traffic participant information, obtains several directed line segments, and constructs the future trajectory model of the traffic participant based on these directed line segments. The specific expression is:

[0017]

[0018] Where, Lpred Indicates the length of the directed line segment; u cur is the current speed of the traffic participant; t pred is the prediction time domain; θ represents the direction of the future trajectory; θ cur The current direction of the traffic participant;

[0019] The potential collision adjacency matrix is ​​constructed based on the future trajectory model of traffic participants, which is specifically expressed as:

[0020]

[0021] Where, is the potential collision adjacency matrix An element in represents the potential collision relationship between traffic participants i and j at time t. In this calculation, traffic participant i is the vehicle itself; AB is the future trajectory of the vehicle calculated based on the traffic participant future trajectory model; CD is the future trajectory of the current traffic participant calculated based on the traffic participant future trajectory model; if there is an intersection (x, y), it means that the future trajectory of the vehicle and the future trajectory of the current traffic participant overlap, and there is a potential collision relationship. In this case, Assign a value of 1.

[0022] It should be noted that the potential collision adjacency matrix A simplified collision risk assessment algorithm is used to identify traffic participants that may affect each other. Because the future trajectory is still unknown during the trajectory prediction stage, this solution uses a directed line segment generated based on the current speed and direction of the participant as the future trajectory for collision detection.

[0023] Based on the above-mentioned construction of the Euclidean distance adjacency matrix and the potential collision adjacency matrix, in order to continue to explore the hidden behavioral interaction relationships, an adaptive adjacency matrix can be constructed based on the traffic participant information. The specific expression is:

[0024]

[0025] Where, is the potential adaptive adjacency matrix An element in , which represents the adaptive relationship between traffic participants i and j at time t; E i Indicates that traffic participant i is taken as the source node, E j Indicates that traffic participant j is taken as the target node; ReLU is the activation function, which is used to eliminate weak connections and increase the nonlinearity of the network layer; SoftMax is the normalization function used to normalize the adaptive adjacency matrix; the adaptive adjacency matrix It is used to capture the dependency between the source node and the target node embedding, that is, to represent the interaction between traffic participants i and j.

[0026] It should be noted that node embedding is to map a node to a multidimensional vector and express it with this multidimensional vector, or more commonly, to replace this node input into the subsequent neural network layer.

[0027] Furthermore, the multi-scale spatiotemporal graph convolution model is constructed to extract the spatiotemporal interaction features of traffic participants based on the traffic scene expression model, including:

[0028] A multi-scale spatiotemporal graph convolutional model is constructed by stacking multiple dilated TCN-GCN spatiotemporal layers. The GCN in each spatiotemporal layer is used to capture the interactions between traffic participants at different time scales, and the dilated TCN in each spatiotemporal layer is used to capture the movement trends of traffic participants over time.

[0029] The multi-scale spatiotemporal graph convolution model is used to extract the spatiotemporal interaction features of traffic participants based on the traffic scene representation model;

[0030] Among them: Dilated TCN stands for Dilated Temporal Convolutional Network; GCN stands for Graph Convolutional Network.

[0031] The expanded TCN constructed by the above scheme achieves an optimal balance between predictive performance and computational efficiency, providing feasibility for the algorithm's deployment on low-computing domain control chips. It also constructs a method of representing traffic scene expression models using three adjacency matrices: the Euclidean distance adjacency matrix, the potential collision adjacency matrix, and the adaptive adjacency matrix. This is conducive to fully modeling prior knowledge and thus optimizing the performance of the GCN graph convolutional layer.

[0032] Furthermore, in the multi-scale spatiotemporal graph convolution model, for any dilated TCN, the dilated convolution operation F(t) at time t is specifically expressed as:

[0033]

[0034] Where Q represents the one-dimensional input sequence; f represents the filter; a is the element index in the filter; k1 is the size of the filter; d represents the dilation factor, which is used to control the distance between the convolution kernel elements; in the multi-scale spatiotemporal graph convolution model, the value of d for the odd-numbered dilated TCN-GCN spatiotemporal layer is 1, and the value of d for the even-numbered dilated TCN-GCN spatiotemporal layer is 2.

[0035] Furthermore, in the multi-scale spatiotemporal graph convolution model, for any GCN, its output spatiotemporal interaction feature Z is specifically expressed as:

[0036]

[0037] Where k2 represents the diffusion step; A power series representing the Euclidean distance adjacency matrix; A power series representing the potential collision adjacency matrix; represents the power series of the adaptive adjacency matrix; X is the output of the expanded TCN; is the learnable parameter matrix in the convolution operation; Indicates the last observation timestamp t obs The Euclidean distance adjacency matrix of ; Indicates the last observation timestamp t obs The potential collision adjacency matrix of ; Indicates the last observation timestamp t obs The Euclidean distance adjacency matrix, the potential collision adjacency matrix and the adaptive adjacency matrix constitute a traffic scene expression model.

[0038] Furthermore, the constructing of the trajectory prediction model so as to make the trajectory prediction model perform trajectory prediction based on the spatiotemporal interaction features and obtain the predicted trajectory includes:

[0039] Constructing a trajectory prediction model with an encoder-decoder structure; wherein both the encoder and decoder have LSTM neural networks inside;

[0040] The spatiotemporal interaction features are input into the encoder and are learned and processed by the LSTM neural network to obtain contextual feature information;

[0041] The context feature information is input into the decoder, processed and learned by the LSTM neural network, and combined with the current position of the vehicle to obtain several sequence points;

[0042] Perform trajectory prediction based on several sequence points to obtain the predicted trajectory.

[0043] Furthermore, the constructing of the trajectory prediction model so as to make the trajectory prediction model perform trajectory prediction based on the spatiotemporal interaction features and obtain the predicted trajectory includes:

[0044] Construct a trajectory prediction model, which includes a target lane prediction sub-model, a trajectory cluster prediction sub-model, and a trajectory evaluation output sub-model; wherein:

[0045] The target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired vehicle historical trajectory data and lane data;

[0046] The trajectory cluster prediction sub-model is used to obtain the vehicle initial trajectory cluster based on pre-acquired vehicle historical trajectory data and the vehicle target lane;

[0047] The trajectory evaluation output sub-model is used to construct vehicle trajectory clusters based on the initial vehicle trajectory clusters and evaluate the vehicle trajectory clusters based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory clusters, and obtain predicted trajectories.

[0048] Furthermore, the target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired vehicle historical trajectory data and lane data, including:

[0049] Based on the pre-acquired vehicle historical trajectory data and lane data, the relative position of the vehicle and the candidate lane is obtained;

[0050] Based on the cross-attention mechanism and the relative position of the vehicle and the candidate lane, the relevance of the candidate lane centerline to the vehicle's historical trajectory is calculated;

[0051] The target lane of the vehicle is obtained based on the correlation of the center line of the candidate lane with the historical trajectory of the vehicle.

[0052] In the above scheme, if all lanes surrounding the vehicle were directly sampled and corresponding vehicle trajectory clusters were generated without consideration, the number of vehicle trajectory clusters would be too large, increasing the complexity of the prediction model. Therefore, to reduce the sampling space for predicted trajectory clusters, this scheme uses a cross-attention mechanism to first select the target lane where the vehicle is most likely to travel, and then generate vehicle trajectory clusters based on this lane.

[0053] Furthermore, determining the target lane primarily relies on the relative positional relationship between the vehicle's historical trajectory and the centerlines of the surrounding lanes, that is, the relative position of the vehicle and the candidate lanes. To characterize this relative positional relationship, the Frenet coordinates s and d of the vehicle's historical trajectory at the centerline of each candidate lane are first calculated. Here, s represents the longitudinal distance traveled along the centerline, and d represents the lateral distance perpendicular to the centerline. For curved lanes, the Frenet coordinate system provides a clearer and more direct description of the relative positional relationship between the vehicle and the lane than Cartesian coordinates. After using Frenet coordinates to describe the relative positional relationship between the vehicle and the candidate lanes, this solution utilizes a cross-attention mechanism to process the relationship between the sequence of vehicle historical trajectories and the sequence of pathpoints along the candidate lane centerlines. Both the cross-attention mechanism and the self-attention mechanism belong to attention networks, which use (Q, K, V) triples to model and capture the dependencies of the input sequence. According to the different input sources of the three vectors Q, K, and V, in this scheme, the candidate lane centerline can be used as the query vector Q of the cross-attention mechanism, and the vehicle historical trajectory can be used as the key vector K and value vector V, thereby constructing the self-attention mechanism and the cross-attention mechanism.

[0054] Furthermore, in the self-attention mechanism, Q, K, and V are obtained by multiplying the same feature input vector X by three different weight matrices W. Q ,W K ,W V The three vectors generated are of the same dimension, i.e. Q = XW Q ,K=XW K ,V=XW V , the weight matrix can be obtained through neural network training. The essence of the self-attention mechanism is to calculate the mutual influence between the internal elements of the input vector X, which can be implemented by using the Transformer network. The query vector Q in the cross attention mechanism is composed of the lane center line X A K and V are converted from the vehicle's historical trajectory X. B It is mainly used to model the relationship between two different sequences, that is, Q = X A W Q ,K=X B W K ,V=X B W V .

[0055] Therefore, in the target lane prediction sub-model, the lane centerline X A As the query vector Q of the cross attention mechanism, the vehicle history trajectory X B As the key vector K and the value vector V, the two are used together to perform a cross-attention operation to find the relevance of the lane centerline to the historical trajectory. The specific steps are as follows:

[0056] 1) Feature Mapping: Q = X A W Q ,K=X B W K ,V=X B W V , X A is the lane centerline, X B is the vehicle history trajectory;

[0057] 2) Similarity calculation: Through the dot product operation of Q and K, the similarity score between the query vector Q and the key vector K is calculated. These similarity scores represent the similarity between X and K. A Each element in X B The relationship between all elements in can be expressed as:

[0058]

[0059] Among them, S is the similarity score matrix, is a scaling factor used to prevent the dot product value from being too large;

[0060] 3) Attention weight calculation: The similarity score is normalized by the Softmax function to obtain the attention weight, which reflects the X A Each element in X B The importance of all elements in can be expressed as:

[0061] W a A→B =Softmax(S A→B )

[0062] Among them, W a is the attention weight;

[0063] 4) Weighted summation: The attention weights are weighted summed with the value vector V to obtain the output representation after cross-attention adjustment, which can be expressed as:

[0064] O A→B =W a A→B V B

[0065] Where, O A→B The dimension of O is consistent with the dimension of the query vector Q. Through the above attention mechanism, O A→B Each element in the query vector Q contains the association information of the key vector K and the value vector V, which models the lane centerline sequence X. A and the vehicle history trajectory sequence X B The mutual relationship between them.

[0066] Furthermore, after uniformly encoding lane centerline information and vehicle trajectory history to establish their interconnectedness, all candidate lanes can be constructed as an [n, c]-dimensional vector: Each candidate lane is represented by a c-dimensional vector, and this is sequentially added to n lanes to form an [n, c]-dimensional lane array. Here, n is the number of candidate lanes, and c is the lane feature dimension. Next, a Transformer self-attention network is used to learn the connections and distinctions between candidate lanes. A multi-layer perceptron (MLP) and softmax are then used to output a probability distribution of candidate lanes. Finally, considering possible prediction errors, this solution strikes a balance between sampling efficiency and prediction accuracy, selecting the top three lanes with the highest probability as the vehicle's target lane prediction output.

[0067] Furthermore, the trajectory cluster prediction sub-model is used to obtain the vehicle initial trajectory cluster based on the pre-acquired vehicle historical trajectory data and the vehicle target lane, including:

[0068] Construct the Frenet coordinate system of the vehicle's target lane's centerline;

[0069] In the Frenet coordinate system of the lane centerline, spatial sampling of the vehicle target lane is performed based on the local path planning method to obtain the first path polynomial curve;

[0070] Performing Cartesian coordinate transformation on the first path polynomial curve to obtain a second path polynomial curve;

[0071] The curvature of the second path polynomial curve is detected, and the path polynomial curve that meets the preset requirements is screened to obtain the initial vehicle trajectory cluster.

[0072] It should be noted that in order to generate a predicted trajectory cluster that conforms to the road topology and vehicle kinematics, the idea of ​​local path planning is applied to spatially sample the target lane and express the initial vehicle trajectory cluster using a polynomial curve. This process can include the following steps:

[0073] 1) Sampling is performed in the Frenet coordinate system of the lane centerline, where s represents the longitudinal distance along the centerline and d represents the lateral distance perpendicular to the centerline. The target lane centerline is fitted as a function of the parameter s through a cubic spline curve:

[0074]

[0075] Where, are the spline coefficients, and x,y are the Cartesian coordinate values ​​of the centerline point. Parameterizing the centerline facilitates subsequent restoration of the trajectory cluster to a Cartesian coordinate system through s-interpolation.

[0076] 2) The longitudinal distance s and lateral distance d of the vehicle initial trajectory cluster in Frenet coordinates are constructed as fourth-order and fifth-order polynomials of the prediction time t respectively:

[0077]

[0078] In the formula, for the quartic polynomial, the initial state can be combined and terminal state Five equations are established to solve. The above states correspond to the initial longitudinal distance, initial longitudinal velocity, initial longitudinal acceleration, terminal longitudinal velocity, and terminal longitudinal acceleration. Similar to the lateral distance d, the fifth-order polynomial coefficients are obtained. The initial states are then and terminal state Create five equations to solve.

[0079] 3) Sampling different terminal states S t and terminal state D tBy combining the parameters, a series of first path polynomial curves in Frenet coordinates can be generated. Then, the center line parameterized in step 1 is interpolated and restored to the Cartesian coordinate system to obtain the second path polynomial curve. In this scheme, the terminal velocity sampling range is set to:

[0080]

[0081] Where, is the current speed, which is obtained by the difference between historical trajectory points; t pred is the prediction time domain; a max,brake and a max,acc The maximum deceleration and maximum acceleration are respectively. The terminal velocity is then sampled at a resolution of 1 km / h, and the terminal lateral distance is sampled at a resolution of 0.5 m. Through a combination of longitudinal and lateral methods, the vehicle's driving area along the target lane is sampled and a second path polynomial curve is generated.

[0082] 4) Perform curvature detection on the second path polynomial curve to eliminate unreasonable candidate trajectories with excessive curvature. The trajectory cluster curvature can be calculated using the triangle circumscribed circle curvature method. Finally, based on the calculation results, select path polynomial curves that meet the preset requirements to obtain the vehicle's initial trajectory cluster.

[0083] Furthermore, the trajectory evaluation output sub-model is used to construct vehicle trajectory clusters based on the initial vehicle trajectory clusters and evaluate the vehicle trajectory clusters based on the spatiotemporal interaction characteristics, calculate the probability distribution of the vehicle trajectory clusters, and obtain the predicted trajectory, including:

[0084] The vehicle trajectory cluster and the vehicle historical trajectory are uniformly represented for heterogeneous information to construct the vehicle trajectory cluster;

[0085] The vehicle trajectory clusters and their temporal and spatial interaction features are integrated and encoded through a cross-attention mechanism. The preset Transformer self-attention model is used to capture the temporal motion characteristics of the vehicle trajectory clusters. Finally, the probability distribution of the vehicle trajectory clusters is output through an MLP multi-layer perceptron and a Softmax layer.

[0086] The predicted trajectory is obtained based on the probability distribution of vehicle trajectory clusters.

[0087] It's important to note that on structured roads, human drivers first determine which lane surrounding vehicles are traveling in, and then, based on this, determine their relative positions to the vehicle. Therefore, lane guidance information is crucial for trajectory prediction. However, encoding multi-source heterogeneous information, such as lane markings and the historical trajectories of surrounding vehicles, and using a neural network to learn their interrelationships presents a significant challenge. Lane markings are a series of waypoints, and unlike the historical trajectories of surrounding vehicles, this sequence of waypoints does not implicitly convey velocity or acceleration characteristics. Furthermore, the target vehicle typically has more than one lane surrounding it, significantly hindering the network model's ability to learn the relationship between the vehicle's historical trajectory and the correct lane during training.

[0088] To address the aforementioned technical issues, this solution combines historical vehicle trajectories with each trajectory in the predicted initial vehicle trajectory cluster to construct a complete vehicle trajectory cluster from the starting point of the historical trajectory to the end point of the predicted trajectory. The predicted vehicle trajectory cluster itself is a motion trajectory that includes velocity and acceleration information. Furthermore, the generation of the predicted vehicle trajectory cluster also takes into account the road topology, reflecting the changing trends of lane direction and position. Therefore, by combining historical trajectories with the initial vehicle trajectory cluster, a unified representation of the heterogeneous information of static map information and dynamic vehicle motion trajectories is achieved.

[0089] The Transformer's self-attention mechanism can be further used to learn continuous motion features from historical vehicle trajectories to predict vehicle trajectory clusters. The complete vehicle trajectory cluster constructed above is the x, y coordinates in the Cartesian coordinate system. In a scenario guided by the lane centerline, converting the x, y coordinates to Frenet coordinates s and d based on the lane centerline can obtain a more direct and essential relative position relationship, thereby accelerating the training of the neural network. However, due to special scenarios such as curves, the distance s of the xy trajectory projected along the lane centerline is not linearly related to the vehicle speed. Therefore, the trajectory feature vector designed in this scheme uses the more essential vehicle speed v and the distance d in the direction perpendicular to the lane centerline to characterize the vehicle's forward and backward movement trend.

[0090] Furthermore, by unifying the road topology and vehicle trajectory information, a temporally complete predicted vehicle trajectory cluster is created, encompassing both the vehicle's historical trajectory and the predicted candidate trajectories. Therefore, for each temporally complete trajectory input, the self-attention mechanism encodes the motion characteristics of the preceding and following points in the trajectory sequence, identifying the optimal candidate trajectory that best matches the preceding and following motion trends. This allows the construction of a trajectory evaluation output sub-model to evaluate the initial vehicle trajectory cluster, ultimately obtaining the probability distribution and predicted trajectory of the final vehicle trajectory cluster.

[0091] This approach can predict the future trajectories of the vehicle itself and surrounding vehicles, taking into account road topology and vehicle kinematic constraints. Ultimately, it generates a multimodal predicted vehicle trajectory cluster and its probability distribution, fully characterizing the uncertain behavior of the vehicle. This provides more accurate information about the motion of surrounding obstacles for subsequent decision-making and planning, resulting in the final predicted trajectory. The generated vehicle trajectory cluster revolves around the vehicle's most probable predicted trajectory, covering almost all possible vehicle motion intentions. A corresponding probability is generated for each trajectory in the predicted trajectory cluster, offering the advantage of precise, long-term multimodal trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 A schematic flow chart of a trajectory prediction method based on spatiotemporal interactive feature fusion provided by one embodiment of the present invention;

[0093] Figure 2 An example of a space-time diagram provided by an embodiment of the present invention;

[0094] Figure 3 A multimodal trajectory prediction flow chart considering road topology and vehicle kinematic constraints is provided in one embodiment of the present invention;

[0095] Figure 4 A schematic diagram of an encoder-decoder structure of an LSTM neural network provided by one embodiment of the present invention;

[0096] Figure 5 A schematic diagram of a target lane prediction sub-model provided by one embodiment of the present invention;

[0097] Figure 6 A schematic diagram of a vehicle initial trajectory cluster provided by one embodiment of the present invention;

[0098] Figure 7 A schematic diagram of a triangle circumscribed circle curvature method provided by one embodiment of the present invention;

[0099] Figure 8 A schematic diagram of a method for splicing a historical trajectory with an initial vehicle trajectory provided by one embodiment of the present invention;

[0100] Figure 9 A schematic diagram of a trajectory assessment output sub-model architecture provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0101] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0102] See Figure 1 This embodiment provides a trajectory prediction method based on spatiotemporal interactive feature fusion, comprising the following steps:

[0103] S1: Real-time collection of traffic participant information on the road;

[0104] In this embodiment, traffic participant information includes category information such as vehicles, bicycles, and pedestrians, as well as other basic information. Category information is a dimension of the state variable. The subsequent adjacency matrix solution does not explicitly and specifically generate different adjacency matrices based on different categories. Instead, the impact of potential collision risks and adaptive adjacency matrices is universally and uniformly considered or learned indirectly.

[0105] S2: Construct a traffic scene expression model based on traffic participant information;

[0106] S3: Construct a multi-scale spatiotemporal graph convolutional model to extract the spatiotemporal interaction features of traffic participants based on the traffic scene representation model;

[0107] S4: Construct a trajectory prediction model to enable the trajectory prediction model to perform trajectory prediction based on the spatiotemporal interaction features to obtain a predicted trajectory.

[0108] This embodiment constructs a traffic scene representation and extracts the spatiotemporal interaction features of traffic participants, so that the final predicted vehicle trajectory can capture the spatial correlation of traffic participant behaviors at different time scales, extract multi-scale spatiotemporal fusion interaction features, and effectively improve the accuracy of trajectory prediction.

[0109] It should be noted that the traffic scene expression model represents the traffic scene as a space-time graph in graph theory, where the nodes in the graph represent traffic participants and the edges in the graph represent the interaction relationship between two traffic participants. Then, this space-time graph can be mathematically represented as an adjacency matrix. The following embodiments consider the three aspects of Euclidean distance, potential collision and adaptability, and construct three adjacency matrices: Euclidean distance adjacency matrix, potential collision adjacency matrix and adaptive adjacency matrix. The combination of the three adjacency matrices can jointly express the true relationship between the participants in the traffic scene.

[0110] In one embodiment, the traffic scene expression model includes a Euclidean distance adjacency matrix, a potential collision adjacency matrix, and an adaptive adjacency matrix. Figure 2 , in order to model the dynamic interaction relationship between traffic participants, Figure 2 The traffic scene in a is represented as Figure 2 b and Figure 2 c shows the space-time graph G t ={V t ,E t}. Among them, G t The node set V represents the traffic behavior impact relationship at time t with traffic participants as nodes. The nodes in the space-time graph represent traffic participants, and the edges in the graph represent the interaction relationship between two traffic participants. t Each node in E corresponds to a traffic participant. t is a set of edges that represent the relationship between two participants at time t. The adjacency matrix can then be used to mathematically represent the spatiotemporal graph structure, specifically:

[0111] The traffic scene expression model based on traffic participant information is constructed, including: constructing a Euclidean distance adjacency matrix based on traffic participant information, for details, see Figure 2 b, Figure 2 (b) depicts the Euclidean distance adjacency matrix scenario representation method, which uses distance to represent and abstract traffic scenarios. Each solid circle represents a traffic participant node. The Euclidean distance adjacency matrix can be expressed as:

[0112]

[0113] Where, is the Euclidean distance adjacency matrix An element in , which represents the adjacency relationship between traffic participants i and j at time t; represents the distance between traffic participants i and j at time t; D close Represents the distance threshold between neighbor participants; 1 indicates that traffic participant i and traffic participant j are neighbors at time t; 0 indicates that traffic participant i and traffic participant j are not neighbors at time t.

[0114] It should be noted that once a traffic participant and a vehicle are identified as neighboring participants, the behavior of the traffic participant will be considered to have an impact on the vehicle trajectory in the subsequent prediction stage. Figure 2 As shown in b, from the perspective of the red target vehicle, the radius is D closeAll traffic participants within the blue circle are its neighbors. However, in real traffic, the behavior of traffic participants outside the blue circle can still affect the red target vehicle. For example, in the figure, two pedestrians are crossing the road in front of the red target vehicle. For a human driver, the red target vehicle would typically begin to assess the risk of collision with the two pedestrians and ultimately slow down to avoid the closer pedestrian. However, because both pedestrians are outside the blue circle, this significant pedestrian is not taken into account by the Euclidean distance adjacency matrix.

[0115] Therefore, in order to solve the problem that the fixed distance threshold of the Euclidean distance adjacency matrix cannot be dynamically adjusted, this embodiment proposes an adjacency matrix that takes into account the potential collision risk. Used to consider the possibility of future collisions. A simplified collision risk assessment algorithm is used to find traffic participants that have an impact on each other. Figure 2 c continues to use the above-mentioned pedestrian crossing the road as an example to illustrate the core idea of ​​this collision assessment algorithm. Because the future trajectory is still unknown in the trajectory prediction stage, this embodiment uses a directed line segment generated according to the participant's current speed and direction as the future trajectory for collision detection. Therefore, directed line segments AB, CD, and EF represent Figure 2 a and Figure 2 The future trajectories of the vehicle and two pedestrians in b are calculated and generated using the constant velocity kinematic model, specifically expressed as:

[0116]

[0117] Where, L pred Indicates the length of the directed line segment; u cur is the current speed of the traffic participant; t pred is the prediction time domain; θ represents the direction of the future trajectory; θ cur is the current direction of the traffic participant; it is simplified to always be consistent with the current direction. This model provides a simple and fast method to assess collision risk. That is, if line segments AB and CD intersect at a point, it is considered that there is a risk of collision between the vehicle and the pedestrian. Obviously, when a collision may occur between two participants, it is necessary to model their behavioral interactions, because the possibility of a collision will force them to change their original movement patterns. Therefore, a potential collision adjacency matrix is ​​constructed based on the future trajectory model of the traffic participants, which is specifically expressed as:

[0118]

[0119] Where, is the potential collision adjacency matrix An element in represents the potential collision relationship between traffic participants i and j at time t. In this calculation, traffic participant i is the vehicle itself; AB is the future trajectory of the vehicle calculated based on the traffic participant future trajectory model; CD is the future trajectory of the current traffic participant calculated based on the traffic participant future trajectory model; if there is an intersection (x, y), it means that the future trajectory of the vehicle and the future trajectory of the current traffic participant overlap, and there is a potential collision relationship. In this case, Assign a value of 1.

[0120] Based on the above construction of the Euclidean distance adjacency matrix and the potential collision adjacency matrix, in order to continue to explore the hidden behavioral interaction relationship, this embodiment constructs an adaptive adjacency matrix To model the spatial dependencies and interactions between traffic participants that are not easily perceived. Adaptive adjacency matrix The main idea is to embed the source node into E i and target node embedding E j Multiply to construct a dynamic and learnable adjacency matrix. Specifically, an adaptive adjacency matrix can be constructed based on traffic participant information, which is specifically expressed as:

[0121]

[0122] Where, is the potential adaptive adjacency matrix An element in , which represents the adaptive relationship between traffic participants i and j at time t; E i Indicates that traffic participant i is taken as the source node, E j Indicates that traffic participant j is taken as the target node; ReLU is the activation function, which is used to eliminate weak connections and increase the nonlinearity of the network layer; SoftMax is the normalization function used to normalize the adaptive adjacency matrix; the adaptive adjacency matrix It is used to capture the dependency between the source node and the target node embedding, that is, to represent the interaction between traffic participants i and j.

[0123] It should be noted that node embedding is to map a node to a multidimensional vector and express it with this multidimensional vector, or more generally, to replace the node input into the subsequent neural network layer. Both node embeddings are randomly initialized to a set of learnable parameters.

[0124] It should be noted that, based on the traffic scene expression model, the spatial correlation of traffic participants' behaviors at different time scales can be captured by stacking multiple expanded TCN-GCN spatiotemporal layers, thereby realizing multi-scale spatiotemporal fusion interaction feature extraction. Thanks to the receptive fields of different scales, the GCN of each spatiotemporal layer is able to capture the interactive relationships between participants at different time scales. More specifically, at the bottom layer, GCN encodes short-term behavioral interaction information, while at the top layer, GCN understands the interactive relationships from a longer-term time level. However, as the number of stacked layers increases, the number of model parameters and the computational burden increase significantly. To solve this problem, when stacking multiple spatiotemporal layers, this embodiment adopts an expanded TCN temporal convolutional network, improves it, and achieves a balance between prediction accuracy and computational efficiency. Specifically:

[0125] The multi-scale spatiotemporal graph convolution model is constructed to extract the spatiotemporal interaction features of traffic participants based on the traffic scene expression model, including:

[0126] A multi-scale spatiotemporal graph convolutional model is constructed by stacking multiple dilated TCN-GCN spatiotemporal layers. The GCN in each spatiotemporal layer is used to capture the interactions between traffic participants at different time scales, and the dilated TCN in each spatiotemporal layer is used to capture the movement trends of traffic participants over time.

[0127] The multi-scale spatiotemporal graph convolution model is used to extract the spatiotemporal interaction features of traffic participants based on the traffic scene representation model;

[0128] Among them: Dilated TCN stands for Dilated Temporal Convolutional Network; GCN stands for Graph Convolutional Network.

[0129] The expanded TCN constructed in this embodiment achieves an optimal balance between prediction performance and computational efficiency, providing feasibility for the algorithm to be deployed on low-computing-power domain control chips. It also constructs a method of representing traffic scene expression models using three adjacency matrices: the Euclidean distance adjacency matrix, the potential collision adjacency matrix, and the adaptive adjacency matrix. This is conducive to fully modeling prior knowledge and thus optimizing the performance of the GCN graph convolution layer.

[0130] In one embodiment, in the multi-scale spatiotemporal graph convolution model, for any dilated TCN, the dilated convolution operation F(t) at time t is specifically expressed as:

[0131]

[0132] Where Q represents the one-dimensional input sequence; f represents the filter; a is the element index in the filter; k1 is the size of the filter; d represents the dilation factor, which is used to control the distance between the convolution kernel elements; in the multi-scale spatiotemporal graph convolution model, the value of d for the odd-numbered dilated TCN-GCN spatiotemporal layer is 1, and the value of d for the even-numbered dilated TCN-GCN spatiotemporal layer is 2.

[0133] In this embodiment, to extract the temporal correlation between behavioral interactions between traffic participants, the present invention uses a temporal convolutional neural network (TCN) to process time series information. Compared to recurrent neural networks (RNNs), TCNs are based on convolutional neural networks (CNNs), offering the advantages of parallel computing and gradient stability, and can extract feature information over longer timescales. The present invention captures the temporal motion trends of traffic participants by stacking dilated temporal convolutional layers with different receptive fields as a TCN model.

[0134] Furthermore, if Figure 3 As shown, this embodiment shows three different ways to implement dilated convolution. Figure 3 The stacked dilation TCN method shown in a is to design a dilation factor sequence of 1, 2; 1, 2; 1, 2; 1, 2, and then apply it to the dilation parameter of each convolution function Conv1d (called G-WaveNet TCN in this embodiment). If this method is directly applied to trajectory prediction, the time consumption is still very high. On the contrary, Figure 3 The exponential expansion time convolution shown in b greatly speeds up the calculation speed, but it excessively increases the loss of local information, resulting in a decrease in prediction accuracy. In order to solve the above dilemma, this embodiment Figure 3 c explores a new implementation method of stacked dilated TCN layers, that is, assigning 1 or 2 to the stride parameter of the Conv1d function according to the odd and even layers to implement the dilated TCN layer.

[0135] In one embodiment, in order to extract the mutual influence of the behaviors of spatially adjacent traffic participants, this embodiment further uses a graph convolution network to extract the dependency relationship of each node on the adjacency matrix based on the traffic scene expression of three adjacency matrices: Euclidean distance, potential collision, and adaptation. Graph convolution has outstanding advantages in extracting information about surrounding traffic participants due to its ability to aggregate and transform the adjacency information of nodes. Through the design of the adjacency matrix, prior knowledge can be integrated into the prediction to improve the prediction accuracy. After the Chebyshev spectral filtering can be simplified by first-order approximation, standard convolution can be used to process graph structure data, and the spatiotemporal interaction feature Z output by any GCN in the multi-scale spatiotemporal graph convolution model can be specifically expressed as:

[0136]

[0137] Where k2 represents the diffusion step; A power series representing the Euclidean distance adjacency matrix; A power series representing the potential collision adjacency matrix; represents the power series of the adaptive adjacency matrix; X is the output of the expanded TCN; is the learnable parameter matrix in the convolution operation; Indicates the last observation timestamp t obs The Euclidean distance adjacency matrix of ; Indicates the last observation timestamp t obs The potential collision adjacency matrix of ; Indicates the last observation timestamp t obs The Euclidean distance adjacency matrix, the potential collision adjacency matrix and the adaptive adjacency matrix constitute a traffic scene expression model.

[0138] It should be noted that compared with the standard convolution operation, the essence of graph convolution is still the convolution operation implemented by the two-dimensional convolution layer Conv2D. The key difference lies in the processing of the input signal, which is achieved by multiplying The item absorbs the structural information of the graph very well, so that the expert prior knowledge can be integrated into the prediction model through the design of the adjacency matrix. So far, a GCN layer and an expanded TCN layer together constitute a time-space layer. Then, by stacking multiple TCN-GCN spatiotemporal layers, the spatial correlation of traffic participants' behaviors at different time scales is captured, and multi-scale spatiotemporal fusion interaction feature extraction is achieved. Based on this, this embodiment can capture long-term and short-term behavioral interaction patterns at the same time. For example, in a scene where there is a pedestrian crossing the lane ahead, the target vehicle will generally make a decision on a more macroscopic time scale and will not give way to the pedestrian because the pedestrian temporarily stops when crossing the road. GCN can be combined with the top TCN layer to capture this long-term interaction. In the following car scenario, the rear car may maintain a constant relative speed and distance with the front car for a short period of time. The present invention can also learn this short-term behavioral interaction through the cooperation of the bottom TCN layer and the corresponding GCN layer.

[0139] In one embodiment, constructing a trajectory prediction model to enable the trajectory prediction model to perform trajectory prediction based on spatiotemporal interaction features to obtain a predicted trajectory includes:

[0140] Constructing a trajectory prediction model with an encoder-decoder structure; wherein both the encoder and decoder have LSTM neural networks inside;

[0141] The spatiotemporal interaction features are input into the encoder and are learned and processed by the LSTM neural network to obtain contextual feature information;

[0142] The context feature information is input into the decoder, processed and learned by the LSTM neural network, and combined with the current position of the vehicle to obtain several sequence points;

[0143] Perform trajectory prediction based on several sequence points to obtain the predicted trajectory.

[0144] In this embodiment, the encoder-decoder structure based on the LSTM neural network can output a trajectory point sequence, thereby obtaining a predicted trajectory. Figure 4 As shown in Figure 1, the encoder-decoder structure consists of two main parts: encoder and decoder. The core network layers of both encoder and decoder are two-layer LSTM neural networks. First, the spatiotemporal interaction features are passed into the encoder, which can be converted into context information after processing and saved as h t ,have:

[0145] h t =LSTM(Z t )

[0146] Where Z t Represents the spatiotemporal interaction characteristics; the encoding process inside the encoder can be described by the following formula:

[0147] i t =sigmoid(W i Z t +U i h t-1 +b i )

[0148] f t =sigmoid(W f Z t +U f h t-1 +b f )

[0149] o t =sigmoid(W o Z t +U o h t-1 +b o )

[0150] a t =tanh(W a Z t +U a h t-1 +b a )

[0151] c t =f t c t-1 +i t a t

[0152] h t =o t tanh(c t )

[0153] Among them, i t ,f t ,o t They are the input gate, forget gate, and output gate of the LSTM network, respectively. i ,W f ,W o is the linear transformation weight of the corresponding network layer, sigmoid is the nonlinear activation function, c t The current state information or memory extracted by the LSTM network.

[0154] Next, the context information is passed into the decoder. After processing and learning by the LSTM network in the decoder, combined with the current position [x0, y0] of the target vehicle i, a set of point sequences of the predicted trajectory can be iteratively generated according to the following formula:

[0155]

[0156] Finally, the predicted trajectory is obtained based on several sequence points.

[0157] In one embodiment, constructing a trajectory prediction model to enable the trajectory prediction model to perform trajectory prediction based on spatiotemporal interaction features to obtain a predicted trajectory includes:

[0158] Construct a trajectory prediction model, which includes a target lane prediction sub-model, a trajectory cluster prediction sub-model, and a trajectory evaluation output sub-model; wherein:

[0159] The target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired vehicle historical trajectory data and lane data;

[0160] The trajectory cluster prediction sub-model is used to obtain the vehicle initial trajectory cluster based on pre-acquired vehicle historical trajectory data and the vehicle target lane;

[0161] The trajectory evaluation output sub-model is used to construct vehicle trajectory clusters based on the initial vehicle trajectory clusters and evaluate the vehicle trajectory clusters based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory clusters, and obtain predicted trajectories.

[0162] The trajectory prediction model constructed in this embodiment can generate initial vehicle trajectory clusters that satisfy road topology and vehicle kinematic constraints. First, the relative relationship between the vehicle's historical trajectory and the vehicle's target lane is utilized. Then, spatial sampling of the target lane is performed to generate initial vehicle trajectory clusters that satisfy the road topology. Finally, the trajectory evaluation output sub-model evaluates the probability that each trajectory in the initial vehicle trajectory cluster is a predicted trajectory. This probability distribution reveals the most likely future trajectory. This probability distribution also captures the uncertainty of adjacent vehicle behavior, a prerequisite for decision-making and planning to address the uncertainties of adjacent vehicle behavior.

[0163] In one embodiment, the target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired vehicle historical trajectory data and lane data, including:

[0164] Based on the pre-acquired vehicle historical trajectory data and lane data, the relative position of the vehicle and the candidate lane is obtained;

[0165] Based on the cross-attention mechanism and the relative position of the vehicle and the candidate lane, the relevance of the candidate lane centerline to the vehicle's historical trajectory is calculated;

[0166] The target lane of the vehicle is obtained based on the correlation of the center line of the candidate lane with the historical trajectory of the vehicle.

[0167] In this embodiment, if all lanes around the vehicle are directly sampled and the corresponding vehicle trajectory clusters are generated without consideration, the number of vehicle trajectory clusters will be too large, increasing the complexity of the prediction model. Therefore, in order to reduce the sampling space of the prediction trajectory cluster, this embodiment uses the cross attention mechanism to build a Figure 5 The target lane prediction sub-model shown first selects the target lane in which the vehicle is most likely to travel, and then generates a vehicle trajectory cluster based on this lane.

[0168] In one embodiment, obtaining the target lane of the vehicle mainly relies on the relative position relationship between the vehicle's historical trajectory and the center lines of the surrounding lanes, that is, the relative position of the vehicle and the candidate lanes. In order to characterize the above relative position relationship, the Frenet coordinates s and d of the vehicle's historical trajectory on the center line of each candidate lane in the adjacent lane set are first calculated. Among them, s represents the longitudinal distance traveled along the center line, and d represents the lateral distance in the direction perpendicular to the center line. For curved lanes, the Frenet coordinate system can more clearly and directly describe the relative position relationship between the vehicle and the lane than the Cartesian coordinate system. After using Frenet coordinates to describe the relative position relationship between the vehicle and the candidate lane, this embodiment can use the cross-attention mechanism (Cross-Attention) to process the relationship between the sequence of vehicle historical trajectories and the sequence of candidate lane centerline path points. Among them, the cross-attention mechanism and the self-attention mechanism (Self-Attention) both belong to the attention network, which uses (Q, K, V) triples to model and capture the dependency relationship of the input sequence. According to the different input sources of the three vectors Q, K, and V, in this embodiment, the candidate lane centerline can be used as the query vector Q of the cross-attention mechanism, and the vehicle historical trajectory can be used as the key vector K and value vector V, thereby constructing the self-attention mechanism and the cross-attention mechanism.

[0169] In one embodiment, Q, K, and V in the self-attention mechanism are respectively multiplied by three different weight matrices W for the same feature input vector X. Q ,W K ,W V The three vectors generated are of the same dimension, i.e. Q = XW Q ,K=XW K ,V=XW V , the weight matrix can be obtained through neural network training. The essence of the self-attention mechanism is to calculate the mutual influence between the internal elements of the input vector X, which can be implemented by using the Transformer network. The query vector Q in the cross attention mechanism is composed of the lane center line X A K and V are converted from the vehicle's historical trajectory X. B It is mainly used to model the relationship between two different sequences, that is, Q = X A W Q ,K=X B W K ,V=X B W V .

[0170] Therefore, in the target lane prediction sub-model, the lane centerline X A As the query vector Q of the cross attention mechanism, the vehicle history trajectory X BAs the key vector K and the value vector V, the two are used together to perform a cross-attention operation to find the relevance of the lane centerline to the historical trajectory. The specific steps are as follows:

[0171] 1) Feature Mapping: Q = X A W Q ,K=X B W K ,V=X B W V , X A is the lane centerline, X B is the vehicle history trajectory;

[0172] 2) Similarity calculation: Through the dot product operation of Q and K, the similarity score between the query vector Q and the key vector K is calculated. These similarity scores represent the similarity between X and K. A Each element in X B The relationship between all elements in can be expressed as:

[0173]

[0174] Among them, S is the similarity score matrix, is a scaling factor used to prevent the dot product value from being too large;

[0175] 3) Attention weight calculation: The similarity score is normalized by the Softmax function to obtain the attention weight, which reflects the X A Each element in X B The importance of all elements in can be expressed as:

[0176] W a A→B =Softmax(S A→B )

[0177] Among them, W a is the attention weight;

[0178] 4) Weighted summation: The attention weights are weighted summed with the value vector V to obtain the output representation after cross-attention adjustment, which can be expressed as:

[0179] O A→B =W a A→B V B

[0180] Where, O A→B The dimension of O is consistent with the dimension of the query vector Q. Through the above attention mechanism, O A→B Each element in the query vector Q contains the association information of the key vector K and the value vector V, which models the lane centerline sequence X. Aand the vehicle history trajectory sequence X B The mutual relationship between them.

[0181] In one embodiment, after uniformly encoding lane centerline information and the vehicle's historical trajectory to establish their interconnectedness, all candidate lanes can be constructed as an [n, c]-dimensional vector: Each candidate lane is represented by a c-dimensional vector, and this is sequentially added to n lanes to form an [n, c]-dimensional lane array. Here, n is the number of candidate lanes, and c is the lane's characteristic dimension. Next, a Transformer self-attention network is used to learn the connections and distinctions between candidate lanes, and a multi-layer perceptron (MLP) and softmax are used to output a probability distribution of the candidate lanes. Finally, considering possible prediction errors, this embodiment strikes a balance between sampling efficiency and prediction accuracy, selecting the top three lanes with the highest probability as the predicted target lane for the vehicle.

[0182] In one embodiment, the trajectory cluster prediction sub-model is used to obtain the vehicle initial trajectory cluster based on pre-acquired vehicle historical trajectory data and the vehicle target lane, including:

[0183] Construct the Frenet coordinate system of the vehicle's target lane's centerline;

[0184] In the Frenet coordinate system of the lane centerline, spatial sampling of the vehicle target lane is performed based on the local path planning method to obtain the first path polynomial curve;

[0185] Performing Cartesian coordinate transformation on the first path polynomial curve to obtain a second path polynomial curve;

[0186] The curvature of the second path polynomial curve is detected, and the path polynomial curve that meets the preset requirements is screened to obtain the initial vehicle trajectory cluster.

[0187] It should be noted that in order to generate a predicted trajectory cluster that conforms to the road topology and vehicle kinematics, the idea of ​​local path planning is applied to spatially sample the target lane and express the initial vehicle trajectory cluster using a polynomial curve. This process can include the following steps:

[0188] 1) Sampling is performed in the Frenet coordinate system of the lane centerline, where s represents the longitudinal distance along the centerline and d represents the lateral distance perpendicular to the centerline. The target lane centerline is fitted as a function of the parameter s through a cubic spline curve:

[0189]

[0190] Where, are the spline coefficients, and x,y are the Cartesian coordinate values ​​of the centerline point. Parameterizing the centerline facilitates subsequent restoration of the trajectory cluster to a Cartesian coordinate system through s-interpolation.

[0191] 2) The longitudinal distance s and lateral distance d of the vehicle initial trajectory cluster in Frenet coordinates are constructed as fourth-order and fifth-order polynomials of the prediction time t respectively:

[0192]

[0193] In the formula, for the quartic polynomial, the initial state can be combined and terminal state Five equations are established to solve. The above states correspond to the initial longitudinal distance, initial longitudinal velocity, initial longitudinal acceleration, terminal longitudinal velocity, and terminal longitudinal acceleration. Similar to the lateral distance d, the fifth-order polynomial coefficients are obtained. The initial states are then and terminal state Create five equations to solve.

[0194] 3) Sampling different terminal states S t and terminal state D t By combining the parameters, a series of first path polynomial curves in Frenet coordinates can be generated. Then, the center line parameterized in step 1) is interpolated and restored to the Cartesian coordinate system to obtain the second path polynomial curve. In this embodiment, the terminal velocity sampling range is set to:

[0195]

[0196] Where, is the current speed, which is obtained by the difference between historical trajectory points; t pred is the prediction time domain; a max,brake and a max,acc The maximum deceleration and maximum acceleration are respectively. The terminal velocity is then sampled at a resolution of 1 km / h, and the terminal lateral distance is sampled at a resolution of 0.5 m. Through a combination of longitudinal and lateral methods, the vehicle's driving area along the target lane is sampled and a second path polynomial curve is generated.

[0197] 4) Perform curvature detection on the second path polynomial curve to eliminate unreasonable candidate trajectories with excessive curvature. The trajectory cluster curvature can be calculated using the triangle circumscribed circle curvature method. Finally, based on the calculation results, select path polynomial curves that meet the preset requirements to obtain the vehicle's initial trajectory cluster.

[0198] For details, please refer to Figure 6 As shown, the light green trajectory cluster is a feasible trajectory that meets the curvature constraint, and the blue trajectory cluster is an infeasible trajectory. The curvature calculation of the triangle circumscribed circle curvature method is performed as follows:

[0199] See Figure 7 First, assume that A, B, and C are three consecutive discrete points on the predicted trajectory, and a, b, and c are their opposite sides. In ΔABC, the cosine theorem yields:

[0200]

[0201] Then, the curvature k can be obtained from the law of sines:

[0202]

[0203] In one embodiment, the trajectory evaluation output sub-model is used to construct vehicle trajectory clusters based on initial vehicle trajectory clusters, evaluate the vehicle trajectory clusters based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory clusters, and obtain predicted trajectories, including:

[0204] The vehicle initial trajectory cluster and the vehicle historical trajectory are uniformly represented for heterogeneous information to construct the vehicle trajectory cluster;

[0205] The vehicle trajectory clusters and their temporal and spatial interaction features are integrated and encoded through a cross-attention mechanism. The preset Transformer self-attention model is used to capture the temporal motion characteristics of the vehicle trajectory clusters. Finally, the probability distribution of the vehicle trajectory clusters is output through an MLP multi-layer perceptron and a Softmax layer.

[0206] The predicted trajectory is obtained based on the probability distribution of vehicle trajectory clusters.

[0207] It's important to note that on structured roads, human drivers first determine which lane surrounding vehicles are traveling in, and then, based on this, determine their relative positions to the vehicle. Therefore, lane guidance information is crucial for trajectory prediction. However, encoding multi-source heterogeneous information, such as lane markings and the historical trajectories of surrounding vehicles, and using a neural network to learn their interrelationships presents a significant challenge. Lane markings are a series of waypoints, and unlike the historical trajectories of surrounding vehicles, this sequence of waypoints does not implicitly convey velocity or acceleration characteristics. Furthermore, the target vehicle typically has more than one lane surrounding it, significantly hindering the network model's ability to learn the relationship between the vehicle's historical trajectory and the correct lane during training.

[0208] In order to solve the above technical problems, this embodiment combines the historical trajectory of the vehicle with each trajectory in the predicted vehicle initial trajectory cluster to construct a complete vehicle trajectory cluster from the starting point of the historical trajectory to the end point of the predicted trajectory. The predicted vehicle trajectory cluster itself is a motion trajectory containing speed and acceleration information. At the same time, when generating the predicted vehicle trajectory cluster, the road topology is also considered, reflecting the changing trend of lane direction and position. Therefore, the historical trajectory and the vehicle initial trajectory are combined according to Figure 8By splicing and reorganizing, a unified representation of static map information and dynamic vehicle motion trajectory heterogeneous information is achieved.

[0209] The Transformer's self-attention mechanism can be further used to learn continuous motion features from historical vehicle trajectories to predicted vehicle trajectory clusters. The complete vehicle trajectory cluster constructed above is the x, y coordinates in the Cartesian coordinate system. In a scenario guided by the lane centerline, converting the x, y coordinates to Frenet coordinates s and d based on the lane centerline can obtain a more direct and essential relative position relationship, thereby accelerating the training of the neural network. However, due to special scenarios such as curves, the distance s of the xy trajectory projected along the lane centerline is not linearly related to the vehicle speed. Therefore, the trajectory feature vector designed in this embodiment uses the more essential vehicle speed v and the distance d in the direction perpendicular to the lane centerline to characterize the vehicle's forward and backward movement trend. According to Figure 8 Each candidate predicted trajectory is concatenated directly onto the historical trajectory. The historical trajectory dimensions are [3, 5], where 3 represents the feature vector [x, y, v]. The predicted trajectory dimensions are [3, 15]. The concatenated complete trajectory dimensions are [3, 20]. After concatenation, the feature vector [x, y, v] is further converted to a feature vector [v, d]. The concatenated trajectory dimensions are [2, 20].

[0210] Furthermore, by unifying the road topology and vehicle trajectory information, a temporally complete predicted vehicle trajectory cluster is created, encompassing both the vehicle's historical trajectory and the predicted candidate trajectories. Therefore, for each temporally complete trajectory input, the self-attention mechanism encodes the motion characteristics of the preceding and following points in the trajectory sequence, identifying the optimal candidate trajectory that best matches the preceding and following motion trends. This allows the construction of a trajectory evaluation output sub-model to evaluate the initial vehicle trajectory cluster, ultimately obtaining the probability distribution and predicted trajectory of the final vehicle trajectory cluster.

[0211] Furthermore, according to the above idea, the trajectory evaluation output sub-model can be constructed as follows Figure 9The evaluation model shown in the figure specifically consists of three parts: behavioral interaction feature fusion encoding, predicted trajectory cluster self-attention feature extraction, and probability distribution decoding. First, after extracting the multi-scale spatiotemporal interaction features of traffic participants, the predicted vehicle trajectory cluster is integrated with the spatiotemporal interaction features through cross-attention encoding, allowing the vehicle trajectory cluster to attend to the behavioral interaction information of traffic participants. Here, the predicted trajectory cluster serves as the query Q vector of the Cross-Attention module, and the spatiotemporal interaction features serve as the key vector K and value vector V of the Cross-Attention module. The output of the Cross-Attention module maintains the same dimensionality as the predicted trajectory cluster. Next, a standard Transformer self-attention network is used to capture the temporal motion features of the trajectory cluster, including the feature [v, d] representing the vehicle speed v and the distance d perpendicular to the lane centerline. Acceleration and the rate of change of distance d can also be indirectly learned. Finally, the probability distribution evaluation of the predicted trajectory cluster is modeled as a classification problem. The probability distribution of the predicted trajectory cluster is output through an MLP multi-layer perceptron and a softmax layer. Ultimately, a multimodal predicted trajectory cluster and its probability distribution are obtained, which fully represent the uncertain behavior of the predicted vehicle. Among them: Softmax normalized exponential function is a commonly used classifier in deep neural networks, and its output is the probability corresponding to different trajectories.

[0212] This embodiment can predict the future trajectory of the vehicle itself and surrounding vehicles, taking into account road topology and vehicle kinematic constraints. Ultimately, it generates a multimodal predicted vehicle trajectory cluster and its probability distribution that fully characterizes the uncertain behavior of the vehicle. This provides more accurate information about the motion of surrounding obstacles for subsequent decision-making and planning, resulting in the final predicted trajectory. The generated vehicle trajectory cluster revolves around the vehicle's most probable predicted trajectory, covering almost all possible vehicle motion intentions. A corresponding probability is generated for each trajectory in the predicted trajectory cluster, offering the advantage of precise, long-term multimodal trajectory prediction.

[0213] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A trajectory prediction method based on spatiotemporal interactive feature fusion, characterized in that: The following steps are involved: Collect information about traffic participants on the road in real time; Construct a traffic scene expression model based on traffic participant information; Construct a multi-scale spatiotemporal graph convolution model to extract the spatiotemporal interaction features of traffic participants based on the traffic scene expression model; Constructing a trajectory prediction model to enable the trajectory prediction model to perform trajectory prediction based on spatiotemporal interaction features and obtain a predicted trajectory; The step of constructing a trajectory prediction model to enable the trajectory prediction model to perform trajectory prediction based on spatiotemporal interaction features and obtain a predicted trajectory includes: Construct a trajectory prediction model, which includes a target lane prediction sub-model, a trajectory cluster prediction sub-model, and a trajectory evaluation output sub-model; wherein: The target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired vehicle historical trajectory data and lane data; The trajectory cluster prediction sub-model is used to obtain the vehicle initial trajectory cluster based on pre-acquired vehicle historical trajectory data and the vehicle target lane; The trajectory evaluation output sub-model is used to construct vehicle trajectory clusters based on the initial vehicle trajectory clusters and evaluate the vehicle trajectory clusters based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory clusters, and obtain predicted trajectories.

2. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 1, characterized in that: The traffic scene expression model includes a Euclidean distance adjacency matrix, a potential collision adjacency matrix, and an adaptive adjacency matrix; the traffic scene expression model is constructed based on traffic participant information, including: Based on the traffic participant information, the Euclidean distance adjacency matrix is ​​constructed, which is specifically expressed as: Where, is the Euclidean distance adjacency matrix An element in , which represents the adjacency relationship between traffic participants i and j at time t; represents the distance between traffic participants i and j at time t; D close represents the distance threshold between neighbor participants; 1 means that at time t, traffic participants i and j are neighbors; 0 means that at time t, traffic participants i and j are not neighbors; Based on the traffic participant information, the current speed and current direction of the traffic participant are obtained, and a number of directed line segments are obtained. The future trajectory model of the traffic participant is constructed based on the directed line segments. Specifically, it is expressed as follows: Where, L pred Indicates the length of the directed line segment; u cur is the current speed of the traffic participant; t pred is the prediction time domain; θ represents the direction of the future trajectory; θ cur The current direction of the traffic participant; The potential collision adjacency matrix is ​​constructed based on the future trajectory model of traffic participants, which is specifically expressed as: Where, is the potential collision adjacency matrix An element in represents the potential collision relationship between traffic participants i and j at time t. In this calculation, traffic participant i is the vehicle itself; AB is the future trajectory of the vehicle calculated based on the traffic participant future trajectory model; CD is the future trajectory of the current traffic participant calculated based on the traffic participant future trajectory model; if there is an intersection (x, y), it means that the future trajectory of the vehicle and the future trajectory of the current traffic participant overlap, and there is a potential collision relationship. In this case, Assign a value of 1; Based on the information of traffic participants, an adaptive adjacency matrix is ​​constructed, which is specifically expressed as: Where, is the potential adaptive adjacency matrix An element in , which represents the adaptive relationship between traffic participants i and j at time t; E i Indicates that traffic participant i is taken as the source node, E j Indicates that traffic participant j is taken as the target node; ReLU is the activation function; SoftMax is the normalization function; adaptive adjacency matrix It is used to capture the dependency between the source node and the target node embedding, that is, to represent the interaction between traffic participants i and j.

3. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 1, characterized in that: The multi-scale spatiotemporal graph convolution model is constructed to extract the spatiotemporal interaction features of traffic participants based on the traffic scene expression model, including: A multi-scale spatiotemporal graph convolutional model is constructed by stacking multiple dilated TCN-GCN spatiotemporal layers. The GCN in each spatiotemporal layer is used to capture the interactions between traffic participants at different time scales, and the dilated TCN in each spatiotemporal layer is used to capture the movement trends of traffic participants over time. The multi-scale spatiotemporal graph convolution model is used to extract the spatiotemporal interaction features of traffic participants based on the traffic scene representation model; Among them: Dilated TCN stands for Dilated Temporal Convolutional Network; GCN stands for Graph Convolutional Network.

4. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 3 is characterized in that: In the multi-scale spatiotemporal graph convolution model, for any dilated TCN, the dilated convolution operation F(t) at time t is specifically expressed as: Where Q represents the one-dimensional input sequence; f represents the filter; a is the element index in the filter; k1 is the size of the filter; d represents the dilation factor, which is used to control the distance between the convolution kernel elements; in the multi-scale spatiotemporal graph convolution model, the value of d for the odd-numbered dilated TCN-GCN spatiotemporal layer is 1, and the value of d for the even-numbered dilated TCN-GCN spatiotemporal layer is 2.

5. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 3 is characterized in that: In the multi-scale spatiotemporal graph convolution model, for any GCN, its output spatiotemporal interaction feature Z is specifically expressed as: Where k2 represents the diffusion step; A power series representing the Euclidean distance adjacency matrix; A power series representing the potential collision adjacency matrix; represents the power series of the adaptive adjacency matrix; X is the output of the expanded TCN; is the learnable parameter matrix in the convolution operation; Indicates the last observation timestamp t obs The Euclidean distance adjacency matrix of ; Indicates the last observation timestamp t obs The potential collision adjacency matrix of ; Indicates the last observation timestamp t obs The Euclidean distance adjacency matrix, the potential collision adjacency matrix and the adaptive adjacency matrix constitute a traffic scene expression model.

6. The trajectory prediction method based on spatiotemporal interactive feature fusion according to any one of claims 1 to 5, characterized in that: The step of constructing a trajectory prediction model to enable the trajectory prediction model to perform trajectory prediction based on spatiotemporal interaction features and obtain a predicted trajectory includes: Constructing a trajectory prediction model with an encoder-decoder structure; wherein both the encoder and decoder have LSTM neural networks inside; The spatiotemporal interaction features are input into the encoder and are learned and processed by the LSTM neural network to obtain contextual feature information; The context feature information is input into the decoder, processed and learned by the LSTM neural network, and combined with the current position of the vehicle to obtain several sequence points; Perform trajectory prediction based on several sequence points to obtain the predicted trajectory.

7. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 1, characterized in that: The target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired vehicle historical trajectory data and lane data, including: Based on the pre-acquired vehicle historical trajectory data and lane data, the relative position of the vehicle and the candidate lane is obtained; Based on the cross-attention mechanism and the relative position of the vehicle and the candidate lane, the relevance of the candidate lane centerline to the vehicle's historical trajectory is calculated; The target lane of the vehicle is obtained based on the correlation of the center line of the candidate lane with the historical trajectory of the vehicle.

8. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 1, characterized in that: The trajectory cluster prediction sub-model is used to obtain the vehicle's initial trajectory cluster based on pre-acquired vehicle historical trajectory data and the vehicle's target lane, including: Construct the Frenet coordinate system of the vehicle's target lane's centerline; In the Frenet coordinate system of the lane centerline, spatial sampling of the vehicle target lane is performed based on the local path planning method to obtain the first path polynomial curve; Performing Cartesian coordinate transformation on the first path polynomial curve to obtain a second path polynomial curve; The curvature of the second path polynomial curve is detected, and the path polynomial curve that meets the preset requirements is screened to obtain the initial vehicle trajectory cluster.

9. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 1, characterized in that: The trajectory evaluation output sub-model is used to construct vehicle trajectory clusters based on the initial vehicle trajectory clusters and evaluate the vehicle trajectory clusters based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory clusters, and obtain predicted trajectories, including: The vehicle initial trajectory cluster and the vehicle historical trajectory are uniformly represented for heterogeneous information to construct the vehicle trajectory cluster; The vehicle trajectory clusters and their temporal and spatial interaction features are integrated and encoded through a cross-attention mechanism. The preset Transformer self-attention model is used to capture the temporal motion characteristics of the vehicle trajectory clusters. Finally, the probability distribution of the vehicle trajectory clusters is output through an MLP multi-layer perceptron and a Softmax layer. The predicted trajectory is obtained based on the probability distribution of vehicle trajectory clusters.

Images