Track prediction method based on multi-agent heterogeneous graph risk perception coding and continuous parameter decoding

Abstract

The invention discloses a trajectory prediction method based on multi-agent heterogeneous graph risk perception coding and continuous parameter decoding, and relates to the problem that an existing trajectory prediction method neglects perception risk coding such as collision time and motion risk between agents, and trajectory prediction of complex scenes such as crossroads is inaccurate. According to the method, automatic driving multi-agent trajectory prediction is realized through the steps of agent coordinate transformation based on a rotation matrix, multi-agent heterogeneous graph risk perception local coding, map lane line information and agent fusion coding, continuous parameter trajectory decoding and the like. According to the invention, rich coding features of an automatic driving scene are extracted; decoding is different from traditional direct position prediction, but is extracted from features through a decoder, and control points are used for generating a smooth trajectory curve in combination with a primary function and a dynamically adjusted node vector.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding, so as to predict future multi-agent trajectories using historical trajectories. Background Art

[0002] Trajectory represents the location information, state information and lane scene information of multiple agents, which is used as unstructured data input for trajectory prediction. For autonomous driving systems, accurate and reliable trajectory prediction of road agents (such as vehicles and pedestrians) is crucial for downstream decision-making and planning, which can improve driving safety and ride comfort.

[0003] At present, there have been many research results in trajectory prediction. The position and uncertainty are predicted by using multi-layer perceptrons for local encoding of multi-agents, time modeling, global encoding and trajectory decoding. However, the collision time and the encoding of motion information based on speed and heading angle are ignored. In addition, the monomial is numerically unbalanced compared to the polynomial basis function representation. The polynomial basis function represented by continuous parameters can obtain characteristics such as speed through differentiation, thereby improving the trajectory prediction accuracy and robustness in complex scenarios such as intersections.

[0004] This paper proposes a trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding. Trajectory prediction of multi-agents in autonomous driving scenarios lays a solid foundation for downstream decision-making and control links, and improves the safety and reliability of autonomous driving systems. Summary of the invention

[0005] In order to solve the problems that existing trajectory prediction methods ignore the perception risk coding such as collision time and motion risk between intelligent agents, and inaccurate trajectory prediction in complex scenes such as intersections, the present invention proposes a trajectory prediction method based on multi-agent heterogeneous graph risk perception coding and continuous parameter decoding.

[0006] The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding is implemented by the following steps:

[0007] Step 1: Agent coordinate transformation based on rotation matrix; normalize the coordinates of each agent to the local coordinate system of the central agent through the rotation matrix; obtain the feature representation of the central agent and the feature representation between adjacent agents;

[0008] Step 2: Perform risk-aware local encoding of multi-agent heterogeneous graphs; the specific process is as follows:

[0009] Step 21: Construct a local multi-head attention module for a multi-agent heterogeneous graph, use the multi-head attention model to construct risk-aware edges, and calculate the fusion collision time, risk metrics based on direction similarity and speed difference, and multi-agent edge features; finally, output the spatial features of the central agent i at time t through the multi-head attention module;

[0010] Step 22: Time modeling;

[0011] The spatial features of the central agent i at time t output by the local multi-head attention module of the multi-agent heterogeneous graph are processed by the temporal encoder Add learnable time masks to output features after time modeling;

[0012] Step 3: Fusion-encode the map lane information and the agent to obtain fusion features;

[0013] Step 4: Perform continuous parameter trajectory decoding on the fusion features obtained in step 3 to achieve multi-agent trajectory prediction for autonomous driving.

[0014] Beneficial effects of the present invention:

[0015] Compared with the traditional trajectory prediction decoding algorithm, the method of the present invention adds edge features by integrating the risk metrics of motion wind combined with collision time, speed and angle, constructs a multi-head attention local encoding module, and combines time modeling, map lane line information and global interaction encoding modules to extract rich encoding features for autonomous driving scenarios; the decoding module is different from the traditional direct prediction position, but is extracted from the features through the decoder. These control points are combined with basis functions and dynamically adjusted node vectors to generate smooth trajectory curves. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the trajectory prediction method of multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding described in the present invention. DETAILED DESCRIPTION

[0017] Combination Figure 1 This embodiment describes the trajectory prediction method of multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding. The agent is rotated to the relative coordinates of the central agent instead of the global coordinates through the rotation matrix to ensure rotation invariance. By integrating the collision time, speed and angle-based risk metrics into the multi-agent edge features, a multi-agent heterogeneous graph local multi-head attention module is constructed; a low-rank multi-head attention time modeling is established; the map features are integrated to construct global features; and finally, the multi-agent trajectory prediction of autonomous driving is realized based on continuous parameter trajectory decoding.

[0018] The specific steps of this implementation are as follows:

[0019] Step 1: Agent coordinate transformation based on rotation matrix: The coordinates of each agent are normalized to the local coordinate system relative to the central agent i through the rotation matrix, so that the model only focuses on the relative position relationship between agents and is not affected by the rotation of the global coordinate system;

[0020] In this embodiment, for a lane line ζ, its geometric attributes are given by Given, where and are the starting and ending coordinates of the lane line ζ. By converting the point set into a vector set, translation invariance can be guaranteed. However, the relative position information between agents is also lost. Therefore, in order to preserve the spatial relationship, the relative position vectors between agents and between agents and lane lines are introduced.

[0021] The trajectory segments of the central agent i and the adjacent agent j are expressed as According to the heading angle θ of the reference vector i Rotate all local vectors to form the rotation matrix R i Based on the rotated matrix and related semantic attributes a i ,a j , at time t, a multi-layer perceptron MLP is used to obtain the feature representation of the central agent i and the feature representation of any neighboring agent j The formula is as follows:

[0022]

[0023] In the formula, φ nbr and φ center It is a nonlinear transformation of the two-layer perceptron MLP. is parameterized by the heading angle θ i The rotation matrix of is the position of the central agent i at time t, is the position of the central agent i at time t-1; is the position of the neighboring agent j at time t, is the position of the neighboring agent j at time t-1.

[0024] Step 2: Risk-aware local encoding of multi-agent heterogeneous graphs; the specific process is as follows:

[0025] Step 21: Construct a risk-aware multi-head attention module for heterogeneous graphs; construct risk-aware edges through the module by multiplying the road semantic type indicator function, the moving direction risk with speed, and the collision time on the original edge representation. This mechanism dynamically calculates the edge representation between different agents. It not only captures the spatial relationship, but also considers their movement trends, improving the ability to model collision risks.

[0026] The risk edge can be expressed mathematically as:

[0027]

[0028] in, It represents the edge feature between node i (central agent i) and node j (adjacent agent j) at time t. It measures the interaction risk between the two nodes by combining factors such as the direction, speed, and collision time of the nodes. represents the risk measure between nodes i and j based on direction similarity and speed difference, is the collision time between node i and node j, which measures how long two nodes will collide if they maintain their current speed. The smaller the collision time, the higher the collision risk. It is a graph-optimized and clustered edge index set. The edge features between nodes integrate the direction, speed, and collision risk of the nodes, thereby effectively describing the dynamic interaction relationship in the multi-agent system.

[0029] In this embodiment, the risk metric based on direction similarity and speed difference It is expressed as follows:

[0030]

[0031] in, Represents the angular relationship between the directions of motion of node i and node j. α and β represent the heading angles of agent i and agent j, respectively. It provides a smoother measure of directional influence, with a value of 1 indicating that the agents are moving in the same direction and a value of 0 indicating that they are moving in completely opposite directions. Unlike the cosine function, this expression ensures that even when the agents are moving in opposite directions, there is still a certain influence on trajectory prediction. λ1λ2 represent the angle and speed difference weights, respectively, and λ represents the coefficient of the exponential function.

[0032] Quantity The speed difference between the two agents is taken into account. and denote the speed of node i and node j at time t, respectively. It uses exponential decay to smoothly handle the speed difference. As the difference increases, the value gradually decreases, avoiding sudden changes and ensuring natural weight adjustment.

[0033] In this implementation, the collision time calculation formula between the node i and the node j is as follows:

[0034]

[0035] in, and represents the position of node i and node j at time t, and α and β represent the heading angles of the agent.

[0036] In this implementation, the edge index set The construction process is:

[0037] Gaussian kernel is used to calculate edge distance, where p i and p j are the positions of node i and node j respectively. The kernel distances between all trajectory samples are used to construct the similarity matrix W ij According to the degree matrix and similarity matrix, the Laplace matrix is ​​calculated and normalized, and then the eigenvalue and eigenvector are obtained by eigenvalue decomposition, and spectral clustering is performed. Finally, the corresponding edge index set is obtained by minimum cut graph optimization.

[0038] For every pair of nodes (p i ,p j ), calculate the Gaussian kernel similarity:

[0039]

[0040] Where σ is the standard deviation, the adjacency matrix W is initialized to be a zero matrix; the degree matrix and the Laplace matrix are calculated, and the degree matrix is ​​calculated. Calculate the Laplace matrix L: L = D ij -W ij , calculate the normalized Laplacian matrix Cluster the nodes based on the eigenvectors of the normalized Laplacian matrix and assign a cluster C to each node i .

[0041] In this implementation, the embedding features of the central agent are then converted to a query q vector, while the embedding features of the neighboring agents are used to compute the query q and the key k, value v vectors after enhancing the edge attributes:

[0042]

[0043] Among them, W Qspace ,W Kspace , is a learnable matrix, the spatial representation d of agent i in the multi-head attention output h , the resulting query vector Key-value vector and a numeric vector As the input of the multi-head attention module, represents the multi-head attention weight;

[0044]

[0045] Then, the features of neighboring agents are calculated through weighted summation to obtain their contextual representation:

[0046]

[0047] Use a gating mechanism to fuse features from neighboring and central agents:

[0048]

[0049] Ultimately, the output of the central agent is its own features And a weighted combination of neighboring agent features:

[0050]

[0051] in, is the neighbor set of agent i, W gate and W self is a learnable matrix, ⊙ represents the bitwise product of the elements. The risk-aware graph attention uses a gating function to integrate the neighborhood features. Characteristics of the central agent Combined, the expressive power of features is enhanced.

[0052] In order to better control the update of features, the output of the heterogeneous graph risk-aware multi-head attention module is passed through an MLP module to obtain the spatial features of the central agent i at time t In addition, layer normalization is applied before each module and residual connections are added after each module.

[0053] Step 22: Time modeling;

[0054] In order to further capture the temporal information of each local area, a temporal Transformer encoder is used in this implementation. Add a learnable temporal mask M. In the temporal Transformer, spatial features are processed by a temporal attention module. First, the query matrix q, key matrix k, and value matrix v of the input features are calculated. Then, the attention weight is calculated using the dot product between the query and the key, as in formula (7), The weight is obtained by adding a time mask M to obtain the spatiotemporal feature a after time modeling. ξ, represented by a ξ =a ξ +M. d is the scaling factor.

[0055] Step 3: Fusion coding of map lane information and agent; that is, coding the map lane information in broken lines and combining it with the spatiotemporal features a output by temporal modeling ξ Perform fusion coding;

[0056] At the current time t, the local lane line and the relative position vector between the agent and the lane line are rotated to ensure that the changes in direction and position can be captured. The local map structure can indicate the future intentions of the central agent, can better adapt to different driving scenarios, improve the sensitivity to the changes in the position and direction of the central agent, and combine the local map (lane line) information with the spatiotemporal features a ξ To obtain high-dimensional feature embedding, the low-dimensional original features can be transformed into high-dimensional abstract features through encoding, so as to better capture the complex relationship between local map information and the central intelligent agent and obtain the fusion feature z iξ ; It is expressed as follows:

[0057] z iξ =φ lane ([Lane rotation ,a ξ ]) (11)

[0058] Among them, φ lane (·) is the MLP encoder of the lane line, Lane rotation The rotated lane feature is the rotation matrix of the local area of ​​the agent;

[0059]

[0060] In the formula, is the rotation matrix of the local region of the central agent i, They are the starting and ending positions of the lane line respectively.

[0061] Step 4: Continuous parameter trajectory decoding; the specific process is:

[0062] Step 4.1: The fusion feature z iξ Generate preliminary prediction control points P c ; That is: the fusion feature z iξ Input the multilayer perceptron to predict the control point P c ;

[0063] P c =MLP(z iξ ) (13)

[0064] The predicted control point P cUsing curve expression:

[0065]

[0066] Where N f,d (t) is the basis function;

[0067] Step 42: define recursive basis functions;

[0068] In this implementation, a B-spline basis function is used to define how each prediction control point affects the shape of the curve. The recursive formula is as follows:

[0069] Initial basis function (0 times)

[0070]

[0071] Recursive basis function (degree d)

[0072]

[0073] Among them, v f is the fth knot value in the knot vector, and d is the degree of the B-spline.

[0074] Step 43: Define the node vector;

[0075] In this implementation, the initial node vector: In the first few frames (such as the first 5 frames) of trajectory generation, the node vector is uniformly initialized to ensure uniform distribution among control points.

[0076] Dynamically adjust the node vector: Starting from the 6th frame, the node vector is adjusted according to the speed and curvature of the vehicle. The speed of the curve at a certain time t is obtained by taking the derivative of the B-spline basis function and weighted summing it with the control points, as shown in Formula 17. Ensure that when the vehicle speed is high or the trajectory makes a sharp turn, the node adjustment will be smaller, thereby ensuring the smoothness of the trajectory and the naturalness of the transition. When the speed is low or the curvature is small, the node will be adjusted significantly to more finely control the shape of the trajectory.

[0077]

[0078] In the formula, C'(t) represents the velocity of the curve at time t, that is, the first-order derivative of the curve, which describes the rate of change of the curve at that moment. N' f,d (t) is the derivative of the B-spline basis function, which represents the contribution of each control point to the change in the shape of the curve. The derivative of the basis function describes the immediate impact of the control point in the curve change. cis the position vector of the cth predicted control point. The position of the control point determines the shape of the B-spline curve. F is the number of control points. The curvature κ(t) is used to describe the curvature of the curve. The curvature calculation depends on the first and second order derivatives of the curve. The calculation formula is as follows:

[0079]

[0080] The curvature of a curve is measured by the changes in its velocity and acceleration. × represents the cross product of vectors, which is used to calculate the rotation and curvature direction of the curve. C″(t) represents the second-order derivative of the curve at time t, that is, the acceleration vector of the curve. It describes the acceleration or deceleration behavior of the curve. |C′(t)| represents the modulus of the velocity vector, that is, the magnitude of the curve velocity, which is obtained by calculating the length of the velocity vector.

[0081] The curvature is calculated by the cross product of the first-order derivative and the second-order derivative. The cross product gives the direction and degree of curvature of the curve at a specific point. The denominator is the cube of the modulus of the velocity as a normalization factor to ensure that the curvature calculation is consistent at different speeds. By calculating the curvature, we can know the smoothness and curvature of the trajectory, evaluate the stability of the trajectory, and whether there are complex actions such as sharp turns.

[0082] Step 44: Through the dynamic adjustment of basis functions and node vectors, the generated trajectory can adaptively respond to changes in vehicle speed and curvature. This method makes the generated trajectory not only smooth, but also able to accurately reflect the dynamic changes in complex driving environments.

[0083] In this embodiment, an adaptive B-spline basis matrix B is used. u Matrix multiplication with the predicted two-dimensional control point matrix P is calculated as follows:

[0084] Y pos =B u P (19)

[0085] Among them, Y pos It can be calculated by multiplying with the adaptive basis matrix. u is the precomputed B-spline basis matrix, where B is the updated node vector adjusted according to the velocity and curvature changes, and this matrix represents the contribution of each control point to the final trajectory at time T. P contains the predicted two-dimensional control points for trajectory calculation, where each row represents the x and y coordinates of a control point.

[0086]

[0087] in, is a sub-B-spline basis function, at the adaptive node vector value Evaluate. Adaptive node vector It is calculated based on the velocity and curvature of the target at each step, ensuring that the node values ​​are dynamically adjusted to better fit the trajectory, k∈[0,F].

[0088] By transforming the adaptive B-spline basis matrix B u Multiplying with the control point matrix P, we get the complete predicted position set Y of the trajectory pos , the position set Y pos Dynamic changes in curvature and velocity are taken into account. The method proposed in this embodiment ensures that trajectory generation remains adaptive and responsive to real-time conditions, thereby achieving smoother transitions and more accurate trajectory predictions, which is critical for autonomous vehicles.

[0089] This implementation method realizes autonomous driving trajectory prediction through a multi-agent heterogeneous graph encoder and a continuous parameter trajectory decoder. In order to solve the risk collision problem in complex scenarios, risk-aware local coding is designed. The decoding part realizes smooth and robust trajectory representation through continuous parameter trajectory decoding, thereby improving the trajectory prediction precision and accuracy.

[0090] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding, characterized by: The method is implemented by the following steps: Step 1: Agent coordinate transformation based on rotation matrix; normalize the coordinates of each agent to the local coordinate system of the central agent through the rotation matrix; obtain the feature representation of the central agent and the feature representation between adjacent agents; Step 2: Perform risk-aware local encoding of multi-agent heterogeneous graphs; the specific process is as follows: Step 21: Construct a local multi-head attention module for a multi-agent heterogeneous graph, use the multi-head attention model to construct risk-aware edges, and calculate the fusion collision time, risk metrics based on direction similarity and speed difference, and multi-agent edge features; finally, output the spatial features of the central agent i at time t through the multi-head attention module Step 22: Time modeling; The spatial features of the central agent i at time t output by the local multi-head attention module of the multi-agent heterogeneous graph are processed by the temporal encoder Add learnable time masks to output features after time modeling; Step 3: Fusion-encode the map lane information and the agent to obtain fusion features; Step 4: Perform continuous parameter trajectory decoding on the fusion features obtained in step 3 to achieve multi-agent trajectory prediction for autonomous driving.

2. The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding according to claim 1 is characterized by: In step 1, the feature representation of the central agent i is and the feature representation of the neighboring agent j It is expressed as follows: In the formula, φ nbr and φ center It is the nonlinear transformation of the two-layer perceptron MLP; is the rotation matrix, is the position of the central agent i at time t, is the position of the neighboring agent j at time t, a i ,a j are related semantic attributes.

3. The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding according to claim 1 is characterized by: In step 21, the risk-aware edge is constructed through the multi-agent heterogeneous graph local multi-head attention module, which is expressed as follows: In the formula, is the edge feature between node i and node j at time t; is the risk measure between node i and node j based on direction similarity and speed difference; is the collision time between node i and node j; is the edge index set after graph optimization and clustering.

4. The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding according to claim 3 is characterized by: The risk measure between node i and node j based on direction similarity and speed difference It is expressed as follows: In the formula, is the angular relationship between the movement directions of nodes i and j, α and β are the heading angles of nodes i and j respectively; is the speed difference between the two agents; Vit and are the velocities of node i and node j at time t, λ1 and λ2 represent the angle and velocity difference weights, respectively, and λ is the coefficient of the exponential function.

5. The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding according to claim 3 is characterized by: Calculate the collision time between the node i and the node j It is expressed as follows: Where α and β are the heading angles of node i and node j respectively; and are the speeds of node i and node j at time t respectively; is the position of the central agent i at time t, is the position of the neighboring agent j at time t.

6. The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding according to claim 3 is characterized by: The edge index set The construction process is: First, the edge distance is calculated using Gaussian kernel, and the similarity matrix is ​​constructed using the kernel distance between all trajectory samples; Secondly, according to the degree matrix and the similarity matrix, a Laplace matrix is ​​calculated and normalized; Then, the eigenvectors of the normalized Laplacian matrix are used to cluster the nodes; Finally, the corresponding edge index set is obtained by minimum cut graph optimization 7. The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding according to claim 1 is characterized by: The specific process of step three is: The rotated lane feature Lane rotation And the feature a output by temporal modeling ξ Through multi-layer perceptron encoding, the fusion feature z is obtained iξ , which can be expressed as: In the formula, φ lane (·) is the MLP encoder for lane lines, is the rotation matrix of the local region of the central agent i, a ξ And are the starting position and ending position of the lane line respectively.

8. The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding according to claim 7 is characterized by: The rotated lane feature Lane rotation It is expressed as follows: In the formula, is the position of the agent.

9. The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding according to claim 1 is characterized by: The specific process of step 4 is as follows: Step 4.1: The fusion feature z iξ Input decoder, generate predicted control point P through MLP c , and expressing the predicted control point in the form of a curve; Step 42: define recursive basis functions and node vectors; Step 43: By dynamically adjusting the basis functions and node vectors, the generated trajectory can adaptively respond to changes in the speed and curvature of the vehicle to achieve trajectory prediction.

10. The trajectory prediction method based on multi-agent heterogeneous graph risk perception encoding and continuous parameter decoding according to claim 9 is characterized in that: In step 42, the adaptive B-spline basis matrix B is used u Perform matrix multiplication with the predicted control point matrix P to obtain the complete predicted position set Y of the trajectory pos .

Images