Trajectory prediction method and system for vulnerable road users based on motion query enhancement

Abstract

The application provides a weak road user trajectory prediction method and system based on motion query enhancement, which comprises the following steps: aligning the current position and direction of each motion sample through a normalization process, clustering the end points of the normalized motion samples, taking the obtained cluster centers as generated static queries, using extended static queries (DSQ) to extend the coverage of the static queries to provide more diverse motion patterns; combining the extraction and fusion of trajectory features and map features to obtain map features and enhanced trajectory features, which are used to obtain the update query features of each layer subsequently, using cross-connection dynamic queries (BDQ) to establish a connection between the dynamic queries output by the shallow and deep decoders, coordinating the query optimization between the decoding layers; and integrating the prediction results of multiple homogeneous models to obtain the final prediction trajectory. The application can more effectively extract features that can reflect the motion trend of weak road users, making the prediction results more accurate.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving, specifically relating to a method and system for predicting the trajectory of vulnerable road users based on motion query enhancement. Background Technology

[0002] Influenced by historical and environmental information, trajectory prediction infers the movement intentions of traffic participants to predict future trajectories, which is crucial for the safety and comfort of autonomous driving systems. However, due to the uncertainty of vulnerable road users' intentions and the complexity of the traffic environment, the movement intentions of vulnerable road users are inherently diverse, meaning that given a historical trajectory, there are multiple plausible future trajectories. This makes trajectory prediction a highly challenging task.

[0003] The development of query design in Transformers has inspired researchers to utilize queries for multimodal trajectory prediction. One approach involves representing the multimodal motion of traffic scenes using a set of motion intent queries. However, existing motion intent queries have two limitations. First, the coverage of motion patterns by static queries is limited by the effective range of motion samples. Existing static queries are within the boundary samples of motion patterns, resulting in their inability to effectively cover these patterns and thus failing to comprehensively characterize the motion intentions of vulnerable road users. Furthermore, dynamic queries are progressively optimized in cascaded decoders. Since the decoder layers are sequentially connected, each decoder achieves its optimization objective based only on the prediction of the previous decoder, leading to weak connections between distant decoders. Moreover, this sequential connection means that dynamic queries from previous decoders, regardless of their validity, are propagated to the next decoder. Therefore, deeper decoders may be affected by cascade errors, generating worse prediction results. To address these issues, this invention proposes a motion query-enhanced trajectory prediction method for vulnerable road users. Summary of the Invention

[0004] The purpose of this invention is to provide a motion query-enhanced method for predicting the trajectory of vulnerable road users, which can enhance the multimodal intent representation of queries, thereby achieving accurate prediction of the future trajectory of vulnerable road users.

[0005] To achieve the above objectives, this invention provides a method for predicting the trajectory of vulnerable road users based on motion query enhancement, comprising the following steps:

[0006] The current position and orientation of each motion sample are aligned by normalization, the endpoints of the normalized motion samples are clustered, the cluster centers obtained by clustering are used as the generated static query, and the observation sequence of the motion sample is added to the static query generation process to obtain the extended static query.

[0007] Based on the cascaded Transformer decoding layer, a cross-connected dynamic query is used to establish a connection between the dynamic queries output by the shallow decoder and the deep decoder. The cross-attention module is combined to aggregate and enhance the trajectory features O and map features C to obtain the updated query features of each layer, resulting in an optimized dynamic query.

[0008] The optimized dynamic query is used as input to the motion decoder module. In the motion decoder module, an MLP layer is used for each decoding layer to predict the probability and parameters of each Gaussian component of the Gaussian mixture model at each time step. The predicted trajectory is obtained by extracting the prediction center of the Gaussian component.

[0009] The predicted trajectories of multiple homogeneous motion decoder modules are integrated using an integration strategy based on similar motion patterns. The distance between all predicted trajectories is calculated using non-maximum suppression based on the predicted trajectories of different motion decoder modules and the score corresponding to each predicted trajectory. The trajectories with a distance less than a set threshold are selected as the prediction results.

[0010] Furthermore, the current position and orientation of each motion sample are aligned through normalization, and the endpoints of the normalized motion samples are clustered, including:

[0011] For a given motion sample The current position and orientation of each motion sample are aligned using the following process:

[0012]

[0013]

[0014] in, and These are the position and velocity of the VRU at time t; T is the length of the observation sequence; R(θ) T ) is a rotation matrix with a rotation angle of θ. T That is, the angle between the position at time T and the coordinate axis. Through the above transformation, the current position of all motion samples is aligned to the origin, and their motion direction is aligned to a relatively confined space. The distance from the target of the motion sample to the origin is called the effective range d. In the normalization process, d = T p Clustering algorithms are used to cluster the endpoints of normalized motion samples, and the resulting cluster centers are used as the generated static query. K is the number of static queries.

[0015] Furthermore, by adding the observed sequence of motion samples to the static query generation process, the extended static query includes:

[0016] Add the observed sequence of motion samples to the static query generation process, and expand the effective range d from T p Extended to T+T p ; Translate the starting position of the observation sequence of all motion samples to the origin, rotate it around the set direction of motion, and given a motion sample The normalization process is as follows:

[0017]

[0018] in, It is the starting point of the observation sequence.

[0019] Furthermore, the acquisition of enhanced trajectory features O includes the following steps: based on the vectorized trajectory data and map data, VRU trajectory features and VRU map features are extracted and stitched together and cross-attention is calculated to obtain map features C. Combining VRU trajectory features, map features and trajectory history features, interaction features of future trajectories are extracted. The interaction features of VRU trajectory features and future trajectories are then combined through feature stitching and three MLP layers to obtain enhanced trajectory features O.

[0020] Furthermore, based on the vectorized trajectory data and map data, VRU trajectory features and VRU map features are extracted and stitched together and cross-attention calculations are performed to obtain map features C, including:

[0021] A Transformer-based historical trajectory encoding method is employed. A polyline encoder encodes each polyline as an input label feature for the Transformer encoder, which is used to aggregate the features of each polyline into VRU trajectory features. A similar Transformer-based map encoding method is used to encode the road map, which is used to aggregate the features of each polyline into VRU map features. The resulting VRU map features Cp and VRU trajectory features O are then used. p After concatenation, the data is input into a Transformer encoder for cross-attention calculation to obtain map features C.

[0022] Furthermore, based on the cascaded Transformer decoding layer, a cross-connect dynamic query is used to establish a connection between the dynamic queries output by the shallow and deep decoders. Combined with a cross-attention module, the enhanced trajectory features O and map features C are aggregated to obtain the updated query features for each layer, resulting in optimized dynamic queries including:

[0023] Update the dynamic query output of each decoder according to the following formula:

[0024]

[0025] Among them, D l(q,k,v) is the l-th decoder layer in the Transformer with cross-attention mechanism, 0≤l≤L-1; F c These are the features of the VRU to be predicted; It represents the interaction features between the VRU to be predicted, neighboring VRUs, and map polylines; N is the total number of interacting objects. For the k-th dynamic query in the l-th decoder layer, the dynamic query Initialized by static query;

[0026] Dynamic queries can be achieved through aggregation or through dense connections via cross-connections.

[0027] Furthermore, the aggregation method specifically involves aggregating all dynamic queries from the intermediate layers onto the final decoder layer. For the final layer L, its expression is:

[0028]

[0029] Dense joins, which involve dynamic queries connecting each layer to all previous layers, can be represented as follows:

[0030]

[0031] D l+1 w is the (l+1)th decoder layer in the Transformer that has a cross-attention mechanism. p For balance coefficient, This is the k-th dynamic query in the p-th decoder layer.

[0032] Furthermore, the predicted trajectories of multiple homogeneous motion decoder modules are integrated using an integration strategy based on similar motion patterns. The predicted trajectories from different motion decoder modules and the corresponding scores for each predicted trajectory include:

[0033] Given a model set {M1, M2, ..., M J Each model provides K predicted trajectories and corresponding scores; the predicted trajectories... and corresponding ratings Model M j The k-th predicted trajectory, where T p Let k ∈ {1,2,…,K} and j ∈ {1,2,…,J} be the length of the future trajectory. Let the k-th predicted trajectory of each model be... Having similar movement patterns, their corresponding predicted scores are For confidence ensemble, calculate the average confidence score of the k-th trajectory given by all J models. As the confidence level of the k-th predicted trajectory after integration;

[0034] For trajectory ensemble: the endpoint of the k-th predicted trajectory given by J models is calculated. Obtain the endpoint of the average trajectory Take and Nearest destination As the final integration target point, and Corresponding predicted trajectory The k-th predicted trajectory is selected as the ensemble trajectory. For K predictions, K trajectories are obtained. and the confidence levels corresponding to the K trajectories

[0035] Furthermore, during the training of the motion decoder module:

[0036] For each decoding layer, an MLP layer is used to predict the probability p of each Gaussian component using a Gaussian mixture model at each time step. k and parameter (μ) x ,σ x ,μ y ,σ y The prediction trajectory is obtained by extracting the prediction center of the Gaussian component. The dynamic query of each decoder layer is optimized with the ground truth through a hard allocation strategy. The probability of the selected positive Gaussian component is maximized by cross-entropy loss. The final loss is the weighted sum of all Gaussian regression losses of each decoder layer, and the weights of each decoder layer are equal.

[0037] On the other hand, the present invention provides a trajectory prediction system for vulnerable road users based on motion query enhancement, including a static query acquisition module, a dynamic query acquisition module, and a prediction module;

[0038] The static query acquisition module is used to align the current position and orientation of each motion sample through normalization, cluster the endpoints of the normalized motion samples, use the cluster centers obtained from the clustering as the generated static query, and add the observation sequence of the motion sample to the static query generation process to obtain the extended static query.

[0039] The dynamic query acquisition module is based on the cascaded Transformer decoding layer. It uses cross-connect dynamic queries to establish connections between the dynamic queries output by the shallow decoder and the deep decoder. Combined with the cross-attention module, it aggregates and enhances the trajectory features O and map features C to obtain the updated query features of each layer, thus obtaining the optimized dynamic query.

[0040] The prediction module takes the optimized dynamic query as input to the motion decoder module. In the motion decoder module, an MLP layer is used for each decoding layer to predict the probability and parameters of each Gaussian component of the Gaussian mixture model at each time step. The predicted trajectory is obtained by extracting the prediction center of the Gaussian component. The predicted trajectories of multiple homogeneous motion decoder modules are integrated using an integration strategy with similar motion patterns. Based on the predicted trajectories of different motion decoder modules and the score corresponding to each predicted trajectory, the distance between all predicted trajectories is calculated using non-maximum suppression, and the trajectories with a distance less than a set threshold are selected as the prediction results.

[0041] Compared with existing technologies, this invention has at least the following advantages: It extends the effective range of motion samples from their future trajectory sequences to complete trajectory sequences. As the time range increases, richer motion behaviors are introduced into the clustered samples, enabling the generated static queries to cover more diverse motion patterns and intentions. Cross-dynamic queries introduce cross-connections across decoders instead of direct connections, which facilitates the optimization and updating of dynamic queries and improves the effectiveness of the final dynamic queries. This processing can more effectively extract features that reflect the motion trends of vulnerable road users, making the prediction results more accurate. Attached Figure Description

[0042] Figure 1 This is a diagram illustrating the scope of existing static queries and expanded static queries.

[0043] Figure 2 This forms the basic framework of the motion query-enhanced trajectory prediction method for vulnerable road users of the present invention.

[0044] Figure 3 This is a high-resolution map marker obtained after vectorizing the trajectory and map;

[0045] Figure 4 This is a schematic diagram of the Transformer decoder network structure based on motion queries;

[0046] Figure 5 Different connection methods optimized for dynamic queries;

[0047] Figure 6 This is a model integration method. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] First, let's clarify the following four concepts:

[0050] 1) Motion Samples: Used to generate static queries. Each motion sample is a trajectory sequence, including an observation sequence and a future sequence. The endpoint of the observation sequence is the motion state at the current moment, and the endpoint of the future sequence is called the intent. The motion state includes position, velocity, and angle. A motion sample is the trajectory of a specific VRU in a specific traffic scenario.

[0051] 2) Traffic Scenario: This describes the motion state of the VRU and map information. The motion state includes the predicted position and speed of the VRU, neighboring VRUs, and the vehicle itself. The map information includes the road topology and traffic signs. Typically, the future trajectory of the target VRU needs to be predicted based on the states of other VRUs and the map.

[0052] 3) Static queries: Multiple predefined intents on motion samples, i.e., independent of model training and inference. Represented by a set of vectors, such as points or sequences.

[0053] 4) Dynamic Queries: Multiple predicted intents updated in the decoding modules of multiple cascaded decoders after initialization by static queries. Each decoder layer generates a set of dynamic queries, represented by a set of vectors. (See reference) Figure 1 .

[0054] Example 1, as Figure 2 As shown, the specific implementation steps of the motion query-enhanced trajectory prediction method for vulnerable road users provided in this application are as follows:

[0055] The first step is to vectorize the trajectory and map data. For the target being predicted, its trajectory is a time-dependent directed spline curve. All these elements can be approximated as a vector sequence: for map data, a starting point and direction are chosen, and key points are uniformly sampled from the spline curve at equal spatial distances, then adjacent key points are sequentially connected into a vector; for the trajectory, key points can be sampled starting at t=0 at fixed time intervals (e.g., 0.1 seconds) and connected into a vector. With sufficiently small spatial or time intervals, the generated polyline can approximate the original map and trajectory. The data vectorization process involves a one-to-one mapping between continuous trajectories, map labels, and vector sets, allowing coordinate points to form a graphical representation on top of the vector set for encoding using a graph neural network. Vectorization divides the map into multiple polylines P. j .

[0056] The line P belongs to the broken line j Each vector v i Considered as a node in the graph, the node characteristics are given by the following formula: in, and These are the coordinates of the vector's starting and ending points; d itself can be represented as a 2D coordinate (x...). i ,y i ) or 3D coordinates (x i ,y i ,z i );a i Corresponding attribute features, such as object type, trajectory timestamp, or lane road feature type or speed limit; j is the polyline P j An integer ID, representing v i ∈P j To ensure that the input node features remain unchanged with respect to the predicted target's position, the coordinates of all vectors are normalized so that the input nodes are centered on the position of the predicted target at the last observation time.

[0057] The second step involves extracting and fusing trajectory and map features. After vectorizing the trajectory and map data, historical trajectory encoding based on Transformer is first used to represent the historical states of N traffic participants as O(n). in ∈R N×T×K Where T is the number of historical frames and K is the amount of state information (e.g., position, heading angle, and velocity), for trajectories with fewer than T frames, zeros are padded at the locations of missing frames; then, a simple piecewise linear encoder encodes each piecewise line as an input token feature for the Transformer encoder: O p =Φ(MLP(O) in In this context, MLP(·) is a multilayer perceptron network, and Φ is the max pooling operation used to aggregate the features of each piecewise linear feature into VRU trajectory features O. p ∈R N×D The feature dimension is D; then, a Transformer-based map coding method, similar to trajectory coding, is used to represent the road map as... Where M is the number of polylines on the map, n is the number of points in each polyline, and C... m This represents the number of attributes for each point (e.g., location and road type). Similarly, its encoding is: C p =Φ(MLP(C in In this context, MLP(·) is a multilayer perceptron network, and Φ is a max pooling operation used to aggregate the features of each polyline into VRU map features C. p ∈R M×D The feature dimension is D, and the resulting VRU map feature Cp and VRU trajectory feature O are... p After being concatenated together, the data is fed into a Transformer encoder for cross-attention calculation to obtain map features C∈R. M×DFinally, interaction modeling based on Transformer self-attention is adopted. Specifically, the j-th layer Transformer attention module can be represented as:

[0058]

[0059] Where MultiHeadAttn(·,·,·) is the multi-head attention layer, G0=[O p C p ]∈N (N+M)×D The concatenated feature consists of VRU trajectory features and VRU map features, while κ(·) represents the k nearest neighbors for each query polyline found using the K nearest neighbor algorithm. PE represents the sinusoidal positional encoding of the input label, which uses the latest position for encoding for each VRU.

[0060] The above VRU trajectory features O were obtained. p ∈R N×D With map features C∈R M×D Then, combining historical trajectory features, a regression head is used to densely predict the future trajectories and velocities of all VRUs, and further, the interaction features of the future trajectories are extracted. That is: S 1:Tp =MLP(O p ), where S i ∈R N×4 Includes the future position and velocity of each VRU at time step i, T p This refers to the number of future frames to be predicted. It uses O... p The same piecewise encoder for the future trajectory S of the VRU 1:T Encode the future state of the VRU into feature O. f ∈R N×D Then, enhanced trajectory features O are obtained through feature concatenation and three MLP layers, where O = MLP([O p O f The enhanced trajectory feature O and map feature C are used as the feature input F of the cascaded Transformer decoding layer.

[0061] As an example, the trajectory encoder contains a three-layer MLP with a feature dimension of 256. The input map information includes the position, direction, and type of each polyline point. Trajectory augmentation is performed in the trajectory encoder, with each map polyline containing a maximum of 20 points (approximately 10 meters in the WOMD dataset). M=768 of the nearest map polylines around the predicted target locations are selected. The map encoder contains a five-layer MLP. Because the number of map polylines is much greater than the number of targets, this invention uses a smaller feature dimension of 64. To facilitate feature fusion, both the trajectory encoder and map encoder ultimately map their feature dimensions to a unified 256 dimension through another linear layer. In the encoder's local self-attention, the number of neighbors is set to 16. For the dense future prediction module, a three-layer MLP with a feature dimension of 512 is used to predict the future positions and velocities of all targets.

[0062] By employing a cross-attention mechanism, the trajectory features of the VRU and map features are fused together and enhanced to obtain enhanced trajectory features O. This provides the decoder network with additional future contextual information, helping the model predict future trajectories that better fit the scene. Based on the vectorized data and the results of feature extraction and fusion, static and dynamic queries are generated and optimized.

[0063] refer to Figure 3 The third step involves aligning the current position and orientation of each motion sample through a normalization process, clustering the endpoints of these normalized motion samples, and using the cluster centers as the generated static query. For a given motion sample... The current position and orientation of each motion sample are aligned through a normalization process, which can be represented as follows:

[0064]

[0065]

[0066] in, and These are the position and velocity of the VRU at time t; T is the length of the observation sequence; R(θ) T ) is a rotation matrix with a rotation angle of θ. T That is, the angle between the position at time T and the coordinate axis. Through the above transformation, the current position of all motion samples is aligned to the origin, and the motion direction of the motion samples is aligned to a relatively confined space. The distance from the target of the motion sample to the origin is called the effective range d, and in this normalization process, d = T. p Subsequently, a clustering algorithm (e.g., k-Means) is used to cluster the endpoints of these normalized motion samples, and the resulting cluster centers are used as the generated static query. Where K is the number of static queries.

[0067] The traffic scenarios used for training and testing are also aligned using the normalization method described above. Given a traffic scenario S = {C, I, M}, the normalization process can be expressed as:

[0068]

[0069] in, This indicates the motion state of the VRU to be predicted; It is a set containing N neighboring VRUs; This represents the motion state of the nth neighboring VRU; Represents a map; It represents the position of the j-th polyline among J map polylines at time t; after traffic scene normalization, the static query obtained by clustering can be directly applied to this normalized traffic scene to represent the diverse future movement patterns of the static query.

[0070] Static queries can be applied to all traffic scenarios to cover the possible movement regions of the VRU to be predicted and represent diverse movement patterns. However, the limited effective range of motion samples causes cluster centers to deviate from edge movement patterns, such as boundary samples. Since the VRUs to be predicted in the training or testing traffic scenarios are normalized in the same way as the motion sample sampling, static queries also struggle to cover these movement patterns of the VRUs to be predicted. Extended static queries are used to extend the coverage of static queries to provide more diverse movement patterns. The observed sequence of motion samples is added to the static query generation process, and the effective range d of the static query is increased from T. p Extended to T+T p The starting positions of all motion sample observation sequences are translated to the origin, and then rotated around the predetermined direction of motion. Given a motion sample... The normalization process is as follows:

[0071]

[0072] in, This is the starting point of the observation sequence. In this way, the aligned motion samples are more dispersed, covering a wider variety of motion patterns.

[0073] The fourth step involves using cross-join dynamic queries to establish connections between the dynamic queries output by the shallow and deep decoders, thereby coordinating query optimization across the decoding layers. Static queries are updated sequentially in the cascaded decoder module, with each decoder learning the residuals of the previous decoder and using the output of each decoder as a dynamic query.

[0074] The motion query decoder module based on Transformer used in this invention is as follows: Figure 4 As shown, its core is an N-layer stacked Transformer decoder that continuously optimizes future trajectory prediction through iterative updates of motion queries. The decoder module includes: a static query component, a dynamic query update component, and a multimodal trajectory prediction head.

[0075] The static query component reduces the uncertainty of future trajectories by utilizing different intent queries for different motion modes. Specifically, it first generates κ representative intent points, i.e., static query points I∈R, by applying the k-means clustering algorithm to the endpoints of the actual trajectories. κ×2 Each intent point represents a motion pattern that takes into account the direction and speed of motion. Then, each static intent query is modeled as an intent feature through position encoding, as shown below:

[0076] Q I =MLP(PE(I))

[0077] Where PE(·) is the sinusoidal position code, and Q... I ∈R κ×D .

[0078] The dynamic query update component aims to supplement global intent localization by iteratively collecting fine-grained trajectory features, thereby continuously fine-tuning the trajectory. Each dynamic query is also a spatial point embedding, initialized with the corresponding static intent point, but dynamically updated based on the predicted trajectory in each decoder layer. Specifically, given the predicted future trajectory in the j-th decoder layer... The dynamic search query update for the (j+1)th decoder layer is as follows:

[0079]

[0080] In each decoder layer, static queries are used to propagate information between different motion intentions. Specifically, static queries are used as the positional embeddings of the self-attention modules, as follows:

[0081]

[0082] Among them, C j-1 ∈R κ×D These are query features from (j-1) decoder layers, C 0 Initialized to zero, and This is the updated query feature. Next, the dynamic query is used as the query location feature for cross-attention to extract local features related to trajectory prediction from the scene context. Specifically, the query feature and location feature are used for the query and key, respectively, to decouple their contribution to the attention weights. A cross-attention module is used to aggregate the VRU trajectory feature O and map feature C.

[0083]

[0084] Where [·,·] represents feature concatenation; ∪ represents aggregating trajectory features and map features as keys and values ​​for cross-attention; α(C) represents the operation of extracting local features related to trajectory prediction from the scene context using dynamic queries as query location features for cross-attention, used to collect local map features around L predicted trajectories for motion fine-tuning. Finally, the resulting C j ∈R κ×D The update query content features for each motion query pair in layer j.

[0085] The output C of each decoding layer j A multimodal trajectory prediction head is then appended to generate future trajectories. At each time step, a Gaussian Mixture Model (GMM) is used to represent the distribution of the predicted trajectory. Specifically, for each future time step i∈1,…,T, the probability p and parameter (μ) of each Gaussian component are predicted. x μ y , σ x , σ y , ρ), as shown below:

[0086]

[0087] in, Includes K Gaussian components N 1:K (μ x ,σ x μ y ,σ y ;ρ), whose probability distribution is p 1:K The predicted distribution of the target VRU position at time step i can be expressed as:

[0088]

[0089] in, The target is located in space at position o∈R 2 The probability of it appearing at that location. The predicted trajectory. It can be generated by simply extracting the prediction centers of the Gaussian components.

[0090] The above describes the specific network structure of each layer of the decoder. The dynamic query output of each decoder is updated according to the following formula:

[0091]

[0092] Among them, D l (q,k,v) is the l-th decoder layer in the Transformer with cross-attention mechanism, 0≤l≤L-1; F c These are the features of the VRU to be predicted; It is a fusion feature that enhances trajectory features O and map features C; N is the total number of interactive objects. This is the k-th dynamic query in the l-th decoder layer. Furthermore, dynamic queries... It is initialized by a static query;

[0093] Building upon this, cross-join dynamic queries are used to establish connections between shallow and deep dynamic queries, thereby coordinating query optimization across the decoding layer. (Reference) Figure 5 Cross-join dynamic queries can be implemented in two ways. The first is a convergence join, which aggregates all dynamic queries from the intermediate layers to the final decoder. For the final layer L, its expression is:

[0094]

[0095] Among them, w p It is the balance coefficient. It is the k-th dynamic query in the p-th decoder layer.

[0096] The second approach is dense joins, which connect dynamic queries from each layer to all previous layers; that is, using convergent joins within each decoder layer. The expression for dense joins can be represented as:

[0097]

[0098] The proposed Query Enhanced Motion Transformer (QE-MTR) algorithm in this invention stacks six Transformer decoder layers. It needs to predict the future motion of two types of traffic participants—pedestrians and bicycles—as well as vehicles. Since the motion patterns of each category of traffic participants differ, a set of 64 static queries is generated for each category. In the cross-dynamic query, the balance coefficient w... p The value is set to 0.1. The prediction head in each decoder layer uses a three-layer MLP with an intermediate feature dimension of 512, and the model weights are not shared between different decoder layers.

[0099] The fifth step involves model training and inference. During training, the static queries obtained from the extended static query strategy are combined with contextual features and input into the motion decoder module with cross-dynamic queries. Specifically, an MLP layer is used for each decoding layer to predict the probability p of each Gaussian component of the Gaussian Mixture Model (GMM) at each time step. k and parameter (μ) x ;σ x μ y ;σ y Finally, the predicted trajectory is obtained by using the prediction center of the extracted Gaussian components.

[0100] As an example, the dynamic queries of each decoder layer are optimized with truth values ​​using a hard-assignment strategy, as shown below:

[0101]

[0102] in, The index represents the dynamic query that is closest to the truth endpoint; GT is the future trajectory; L l It uses Gaussian regression loss to maximize the true position of the VRU at a certain time step. Likelihood; (μ x ,σ x μ y ,σ y ;ρ) is the selected positive Gaussian component of the predicted trajectory used for optimization; This is the predicted probability of the selected positive Gaussian component. In the above formula, the cross-entropy loss is used to maximize the probability of the selected positive Gaussian component; the final loss is the weighted sum of all Gaussian regression losses for each decoder layer, with equal weights.

[0103] Step 6: After training multiple homogeneous models through the above steps, the model becomes the trained motion decoder module, as shown in the reference. Figure 6 This involves integrating the prediction results of multiple homogeneous models. Given a model set {M1, M2, ..., M...} J Each model provides K predicted trajectories. And the score corresponding to each predicted trajectory Predicted trajectory and corresponding ratings R1 is model M j The k-th predicted trajectory, where T p It is the length of the future trajectory, k∈{1,2,…,K}, j∈{1,2,…,J}, assuming the k-th predicted trajectory of each model Having similar movement patterns, their corresponding predicted scores are For confidence ensemble, calculate the average confidence score of the k-th trajectory given by all J models. The confidence score of the k-th predicted trajectory after integration is calculated as follows:

[0104]

[0105] Similarly, trajectory ensemble first calculates the endpoint of the k-th predicted trajectory given by J models. Obtain the endpoint of the average trajectory

[0106]

[0107] Selection and Nearest destination As the ultimate integration target. Corresponding predicted trajectory The k-th predicted trajectory is then selected as the ensemble's predicted trajectory. For K predictions, K trajectories can be obtained. and their corresponding confidence levels Finally, to generate 6 future trajectories for evaluation, non-maximum suppression (NMS) was used to select the top 6 predictions from the predicted K=64 trajectory endpoints by calculating the distance between them, with a distance threshold set to 2.5 meters.

[0108] Example 2, based on the concept of the method described in this invention, also provides a trajectory prediction system for vulnerable road users based on motion query enhancement, including a static query acquisition module, a dynamic query acquisition module, and a prediction module;

[0109] The static query acquisition module is used to align the current position and orientation of each motion sample through normalization, cluster the endpoints of the normalized motion samples, use the cluster centers obtained from the clustering as the generated static query, and add the observation sequence of the motion sample to the static query generation process to obtain the extended static query.

[0110] The dynamic query acquisition module is based on the cascaded Transformer decoding layer. It uses cross-connect dynamic queries to establish connections between the dynamic queries output by the shallow decoder and the deep decoder. Combined with the cross-attention module, it aggregates and enhances the trajectory features O and map features C to obtain the updated query features of each layer, thus obtaining the optimized dynamic query.

[0111] The prediction module takes the optimized dynamic query as input to the motion decoder module. In the motion decoder module, an MLP layer is used for each decoding layer to predict the probability and parameters of each Gaussian component of the Gaussian mixture model at each time step. The predicted trajectory is obtained by extracting the prediction center of the Gaussian component. The predicted trajectories of multiple homogeneous motion decoder modules are integrated using an integration strategy with similar motion patterns. Based on the predicted trajectories of different motion decoder modules and the score corresponding to each predicted trajectory, the distance between all predicted trajectories is calculated using non-maximum suppression, and the trajectories with a distance less than a set threshold are selected as the prediction results.

[0112] To evaluate the motion query-enhanced trajectory prediction method for vulnerable road users proposed in this application, evaluation metrics were calculated using the official evaluation tools on the WOMD v1.2.0 dataset. These metrics included Soft mAP, mean accuracy (mAP), minimum mean error (minADE), minimum endpoint error (minFDE), error rate (MR), and overlap rate (OR). Soft mAP was the most important ranking metric, while MR was the second most important. For ease of comparison, in addition to evaluating the accuracy of the prediction method proposed in this application, various state-of-the-art methods were also selected for comparison, including laneGCN, PnPNet, LeapNet, HDGT, HPTR, LLM-MTR, MGTr+, Control MTR, GTR, and MTR++. The specific details of the models involved in the prediction accuracy evaluation are shown in Table 1.

[0113] Table 1

[0114]

[0115] Table 1 shows the performance of our method and the comparison of metrics among different methods in the 2023 Waymo Dynamic Prediction Challenge. Our method demonstrates superior performance on the comprehensive evaluation metric soft mAP, surpassing MTR++, thus fully proving its effectiveness. Furthermore, our method also achieved competitive results on other evaluation metrics. By ensembling five homogeneous models, our method effectively improved all metrics.

[0116] Table 2

[0117]

[0118] Table 2 compares the method of this application with the latest method on WOMD v1.1.0, showing that it outperforms other methods in terms of mAP. Through the DSQ and BDQ designed in this chapter, the method of this application significantly surpasses MTR on all evaluation metrics.

[0119] Table 3

[0120]

[0121] Table 3 evaluates the effectiveness of DSQ and BDQ in the method of this application. Both DSQ and BDQ can improve mAP performance, while the full QE-MTR shows a significant improvement in mAP performance, increasing it from 0.3703 to 0.3991, demonstrating the effectiveness of DSQ and BDQ. Furthermore, the full version reduces prediction errors, namely minADE and minFDE, further illustrating the effectiveness of the method in this chapter. It is worth noting that BDQ increases the error rate to some extent. This may be because dense connections introduce suboptimal intermediate-level dynamic queries into the final dynamic query, resulting in more prediction endpoints deviating from the threshold, thus increasing the error rate.

[0122] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for trajectory prediction of vulnerable road users based on motion query enhancement, characterized in that, The method comprises the following steps: aligning the current position and direction of each motion sample through normalization, clustering the end points of the normalized motion samples, taking the clustering centers obtained by clustering as generated static queries, and adding the observation sequences of the motion samples to the generation process of the static queries to obtain extended static queries; Based on the cascaded Transformer decoding layer, the cross-connection dynamic query is used to establish a connection between the dynamic queries output by the shallow decoder and the deep decoder, and the cross-attention module is combined to aggregate and enhance the trajectory feature O and the map feature C to obtain the updated query feature of each layer, and the optimized dynamic query is obtained; Specifically, the dynamic query output by each decoder is updated according to the following formula: = ( +F c ,F,F)+ where D l (q, k, v) is the lth decoder layer with cross-attention mechanism in the Transformer, 0≤l≤L 1; F c is the feature of the VRU to be predicted; F= is the interaction feature between the VRU to be predicted, the neighboring VRUs and the map polylines; N is the total number of interaction objects, is the kth dynamic query in the lth l decoder layer, and the dynamic query is initialized by the static query. The cross-connection dynamic query is realized by a converging connection or a dense connection; Wherein the enhanced trajectory feature O comprises the following steps: based on the trajectory data and map data after vectorization processing, the VRU trajectory feature and the VRU map feature are extracted and spliced and cross-attention calculation is performed to obtain the map feature C, and the interaction feature of the future trajectory is extracted in combination with the VRU trajectory feature, the map feature and the trajectory history feature; the VRU trajectory feature and the interaction feature of the future trajectory are spliced and three MLP layers to obtain the enhanced trajectory feature O; The optimized dynamic query is taken as the input of the motion decoder module, and in the motion decoder module, an MLP layer is used for each decoding layer to predict the probability and parameters of each Gaussian component of the Gaussian mixture model at each time step, and the predicted trajectory is obtained by extracting the predicted center of the Gaussian component; The prediction trajectories of multiple homogeneous motion decoder modules are integrated, an integration strategy of similar motion patterns is adopted, the distance between all predicted trajectories is calculated based on the prediction trajectories of different motion decoder modules and the scores corresponding to each prediction trajectory using non-maximum suppression, and the trajectories with a distance less than a set threshold are selected as the prediction result; given a model set {M1, M2, …,M J}, each model gives K prediction trajectories and corresponding scores, the prediction trajectory is the kth prediction trajectory of the model M j , the score corresponding to the kth prediction trajectory ∈R1, where T p is the length of the future trajectory, k∈{1,2,…,K}, j∈{1,2,…,J}, and the kth prediction trajectory of each model } j={1,2, …,J} has a similar motion pattern, and the corresponding prediction score is } j={1,2, …,J} ; for confidence integration, the average value of the confidence of the kth trajectory given by all J models is calculated as the confidence of the kth prediction trajectory after integration. For trajectory integration: the end point of the kth prediction trajectory given by J models is calculated } j=1,2,…,J The average trajectory end point is obtained ; the closest end point is taken k as the final integrated target point, while k the corresponding prediction trajectory is k The kth prediction trajectory is selected as the integrated one, and K prediction trajectories result in K trajectories 1, 2, …, k and the corresponding confidence of K trajectories 1, 2, …, k .​ 2. The motion query enhancement based vulnerable road user trajectory prediction method of claim 1, wherein, The current position and direction of each motion sample are aligned through normalization, and the end points of the normalized motion samples are clustered, comprising: For a given motion sample the current position and orientation of each motion sample is aligned by the following process: ← , ← ， ← R( ) · , ← R( ) · , ← R( ) · , ← R( ) · , wherein, and are the position and velocity of the VRU at time t, respectively; T is the length of the observation sequence; R( ) is a rotation matrix with a rotation angle of , i.e., the angle between the position at time T and the coordinate axis, through the above transformation, the current positions of all motion samples are aligned to the origin, and the motion directions are aligned to a relatively limited space. The distance from the target of the motion sample to the origin is referred to as the effective range d, and d=T p , the end points of the normalized motion samples are clustered by using a clustering algorithm, and the obtained clustering centers are taken as the generated static queries , and K is the number of static queries.

3. The motion query enhancement based vulnerable road user trajectory prediction method of claim 2, wherein, And the observation sequence of the motion sample is added to the generation process of the static query to obtain the extended static query, comprising: add the observation sequence of the moving sample to the generation process of the static query and shift the valid range d from T p to T+T p ; translate the start position of the observation sequence of all moving samples to the origin, rotate around the set moving direction, and give a moving sample whose normalization process is as follows: ← , ← , ← R( ) · , ← R( ) · , ← R( ) · , ← R( ) · , wherein, , } is the starting point of the observation sequence.

4. The motion query enhancement based vulnerable road user trajectory prediction method of claim 1, wherein, Based on the trajectory data and map data after vectorization processing, the VRU trajectory feature and the VRU map feature are extracted and spliced and cross-attention calculation is performed to obtain the map feature C, comprising: Adopting history trajectory encoding based on Transformer, adopting polyline encoder to encode each polyline as an input token feature of the Transformer encoder, for aggregating the features of each polyline into VRU trajectory features, adopting similar Transformer-based map encoding as trajectory encoding, encoding the road map, for aggregating the features of each polyline into VRU map features, obtaining VRU map features Cp and VRU trajectory features O p After splicing, input into the Transformer encoder for cross-attention calculation, to obtain map features C.

5. The motion query enhancement based vulnerable road user trajectory prediction method of claim 1, wherein, The converging connection is specifically: all the dynamic queries of the intermediate layers are converged to the last layer decoder, and for the final layer L, the expression is: = ( +F c ,F,F)+ The dense connection is realized by connecting the dynamic query of each layer and all previous layers, which is represented as: = ( +F c ,F,F)+ is the first decoder layer in the Transformer with a cross-attention mechanism l +1 decoder layers, is the balancing coefficient, is the first p decoder layer, k is the first dynamic query in the th decoder layer.

6. The motion query enhancement based vulnerable road user trajectory prediction method of claim 1, wherein, During training of the motion decoder module: Predicting the probability p of each Gaussian component of the Gaussian mixture model at each time step using one MLP layer per decoding layer k and parameters (μ x ,σ x ,μ y ,σ y ,ρ) for each decoding layer, then using the predicted centers of the Gaussian components to obtain the predicted trajectory, the dynamic query of each decoder layer is optimized with the ground truth by a hard assignment strategy, maximizing the cross-entropy loss with the probability of the selected positive Gaussian component, and the final loss is the weighted sum of all Gaussian regression losses of each decoder layer, with equal weights for each decoder layer.

7. A motion query enhancement based vulnerable road user trajectory prediction system, characterized in that, The method comprises a static query acquisition module, a dynamic query acquisition module, and a prediction module; The static query acquisition module is used to align the current position and direction of each motion sample through normalization, cluster the end points of the normalized motion samples, take the clustering centers obtained by clustering as generated static queries, and add the observation sequences of the motion samples to the generation process of the static queries to obtain extended static queries; The dynamic query acquisition module is based on a cascaded Transformer decoding layer, uses cross-connection dynamic queries to establish a connection between the dynamic queries output by the shallow decoder and the deep decoder, combines a cross-attention module to aggregate and enhance the track feature O and the map feature C to obtain the updated query feature of each layer, and obtains the optimized dynamic query; specifically comprising: updating the dynamic query output by each decoder according to the following formula: = ( +F c ,F,F)+ Among them, D l (q,k,v) is the l-th decoder layer in the Transformer that has a cross-attention mechanism, 0≤l≤L 1; F c These are the features of the VRU to be predicted; F= It represents the interaction features between the VRU to be predicted, neighboring VRUs, and map polylines; N is the total number of interacting objects. In the first l The k-th dynamic query in each decoder layer, which is initialized by a static query; The cross-connection dynamic query is achieved through a pooling connection or a dense connection; Wherein, the enhanced track feature O acquisition comprises the following steps: based on the vectorized track data and map data, extracting the VRU track feature and the VRU map feature, and performing splicing and cross-attention calculation to obtain the map feature C, combining the VRU track feature, the map feature and the track history feature, and extracting the interaction feature of the future track; the VRU track feature and the interaction feature of the future track are obtained through feature splicing and three MLP layers to obtain the enhanced track feature O; The prediction module takes the optimized dynamic query as input to the motion decoder module. In the motion decoder module, an MLP layer is used for each decoding layer to predict the probability and parameters of each Gaussian component of the Gaussian mixture model at each time step. The predicted trajectory is obtained by extracting the prediction center of the Gaussian component. The predicted trajectories of multiple homogeneous motion decoder modules are integrated using an integration strategy based on similar motion patterns. Based on the predicted trajectories of different motion decoder modules and the score corresponding to each predicted trajectory, non-maximum suppression is used to calculate the distance between all predicted trajectories, and those with a distance less than a set threshold are selected as the prediction results. Given a model set {M1, M2, …, M…}… J Each model provides K predicted trajectories and corresponding scores. Model M j The k-th predicted trajectory, and the score corresponding to the k-th predicted trajectory. ∈R1, where T p Let k ∈ {1,2,…,K}, j ∈ {1,2,…,J}, and let the k-th predicted trajectory of each model be { } j={1,2, …,J} Having similar movement patterns, their corresponding predicted scores are { } j={1,2, …,J} For confidence ensemble, calculate the average confidence score of the k-th trajectory given by all J models. , which serves as the confidence level of the k-th predicted trajectory after integration; For trajectory integration: the end point of the kth prediction trajectory given by J models is calculated } j=1,2,…,J The average trajectory end point is obtained ; the closest end point is taken k as the final integrated target point, while k the corresponding prediction trajectory k is selected as the kth integrated prediction trajectory, and for K predictions, K trajectories are obtained 1, 2, …, k and the corresponding confidence of the K trajectories 1, 2, …, k .

Images