Trajectory prediction method based on hierarchical progressive interaction and target lane segment

Abstract

The invention discloses a trajectory prediction method based on hierarchical progressive interaction and a target lane segment, and the method comprises the following steps: firstly obtaining the historical state and local lane segments of vehicles around an automatic driving system, and extracting features through employing a lane segment encoder and an intelligent agent encoder; then, local interaction and global interaction are modeled in a layered progressive mode, local interaction modeling is carried out based on self-attention and cross attention, and global interaction modeling is carried out based on self-attention; and finally, predicting a target lane section, obtaining coarse-grained trajectory features based on the target lane section, introducing learnable embedding to refine the features, and finally decoding the multi-modal trajectory and the probability thereof. According to the method, local and global interaction is modeled in a multi-stage and hierarchical manner, a target lane section is sampled from a drivable area to guide a prediction process, and meanwhile, a self-adaptive refining mechanism is introduced, so that the model can be ensured to realize accurate, interactive perception and scene consistent prediction in various dynamic environments.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving trajectory prediction, and in particular to a trajectory prediction method considering a hierarchical progressive interaction network and target lane segment driving. Background Art

[0002] Accurately predicting the future motion trajectory of surrounding traffic participants (hereinafter referred to as "agents") is crucial to enhancing the risk warning and scene understanding capabilities of autonomous driving systems, thereby ensuring the safety of downstream decision-making and planning tasks. Social interaction is a key factor affecting the motion of agents, so effectively modeling various interactions in traffic is one of the core tasks of trajectory prediction. However, most existing trajectory prediction methods usually simplify the dynamic game and tight coupling process of these interactions, and often use Transformer or graph neural networks for one-time modeling, which often fails to fully capture the diverse interactive behaviors and scene information. In addition, the interaction between the agent and the surrounding environment is composed of local and global interactions. By hierarchically modeling these interactions, higher-order interaction information can be obtained, thereby further improving the agent's comprehensive understanding of the surrounding environment. Another important factor affecting the motion of the agent is the destination (hereinafter referred to as "target"), which contains key prior information, which can significantly reduce the uncertainty of motion and the solution space of the prediction model. However, most current target-driven trajectory prediction methods usually randomly sample target points in an unconstrained two-dimensional space. This method has the following two problems: 1) the target points carry limited information and have a weak supervisory effect on the entire trajectory prediction process; 2) the road topology structure is ignored to constrain and guide the movement of the agent. These two problems may cause the predicted trajectory to violate traffic rules or deviate from the road. In summary, in the trajectory prediction process, through multiple interaction modeling, the dynamic game process of various interactions is gradually captured and the interaction information at different levels is aggregated to more comprehensively understand the interactive behavior and surrounding environment of the agent. At the same time, paying attention to the constraint role of the road network and using the target lane segment with richer information content to guide the prediction process, so as to output interactively perceived, scene-consistent and socially acceptable trajectories, has become an important research topic at present. Summary of the invention

[0003] In order to solve the above problems existing in the prior art, the present invention proposes a trajectory prediction method based on hierarchical progressive interaction and target lane segments. Through multiple interaction modeling, the dynamic game process of various interactions is progressively captured and interaction information at different levels is aggregated. At the same time, the target lane segments with richer information content are used to guide the prediction process, thereby outputting interactively perceived and scene-consistent trajectories.

[0004] In order to achieve the above object, the technical solution of the present invention is as follows: A trajectory prediction method based on hierarchical progressive interaction and target lane segment comprises the following steps:

[0005] A. Extracting Agent and Road Environment Features

[0006] A1. Use the positioning and perception module of the autonomous driving system to obtain M lane segments C consisting of L equidistant points in the local range and the states S of the surrounding N agents in the historical T time steps. The lane segment is the abbreviation of the lane centerline segment. The lane segment C and the state S are expressed as follows:

[0007]

[0008] in, Indicates the information of the mth lane segment at the lth equidistant point, where m is the lane segment number and l is the sequence number of the equidistant point. It represents the state of the nth agent at the tth time, where n is the number of the agent and t represents the time.

[0009] A2. For lane segments, use the lane segment encoder to extract features f lane , the lane segment encoder includes N L The stacked multilayer perceptron is MLP, the layer normalization unit is LN unit and the maximum pooling unit is Pooling unit; for the agent, the agent encoder is used to extract the feature f agent , the agent encoder includes N A The stacked one-dimensional convolutional neural network, namely 1D CNN and LN unit, is represented as follows:

[0010] f lane = LaneEnc(C; W lane )

[0011] f agent =AgentEnc(S;W agent )

[0012] In the formula, f lane is the lane segment feature, f agent is the agent feature, LaneEnc(·) is the lane segment encoder, and W lane is the weight of the lane segment encoder, AgentEnc(·) is the agent encoder, and W agent are the weights of the agent encoder.

[0013] B. Hierarchical and progressive modeling of local and global interactions

[0014] B1. Modeling Local Interactions

[0015] Based on self-attention and cross-attention, local interactions are progressively modeled. The specific steps are as follows:

[0016] B11. Apply the agent-lane segment local interactor to model the relationship between the agent and the lane segment, which is expressed as follows:

[0017]

[0018] Where SA(·) is the self-attention layer, is the lane segment feature extracted by the self-attention layer, f a-l is the interaction feature between the agent and the lane segment captured by the cross-attention layer, is the lane segment feature fused with interactive features, CA(·) is the cross attention layer, It is a single-layer MLP.

[0019] B12. Use lane segment-lane segment local interactors to learn the topological structure between lane segments, which is expressed as follows:

[0020]

[0021] In the formula, φ cat (·) is the concatenation operation, It is the local lane segment interaction feature.

[0022] B13. Use lane segment-agent local interactors to capture the fact that agents change their behaviors or intentions after fully understanding traffic rules and road topology, and adjust the model’s reasoning strategy to better adapt to real-time changing traffic scenarios, as shown below:

[0023]

[0024] In the formula, is the agent feature extracted by the self-attention layer, f l-a is the interaction feature between lane segments and agents captured by the cross-attention layer, It is the agent feature that integrates the interactive features.

[0025] B14. After comprehensively learning the information of the road network through steps B11, B12 and B13, the agent-agent local interactor captures the influence of the surrounding neighbors on the motion trajectory of the target agent and improves the accuracy of trajectory prediction, which is expressed as follows:

[0026]

[0027] In the formula, is the local agent interaction feature.

[0028] B2. Modeling global interactions

[0029] Modeling global interactions based on self-attention, the specific steps are as follows:

[0030] The agent-lane segment global interactor is used to further enhance the information interaction between the agent and the map, so that the agent can fully understand the surrounding environment, as shown below:

[0031]

[0032] f″ g =φ cat (SA(f g ),SA(f g ))

[0033] In the formula, f g is the concatenation of the local agent interaction feature and the local lane segment interaction feature, f″ g is the lane segment-agent global interaction feature.

[0034] C. Decoding multimodal trajectories based on predicted target lane segments

[0035] C1. Predict target lane segment

[0036] First, we use the local agent interaction features in step B14 Interaction features with the local lane segment in step B12 Calculate the agent's attention to the lane segment, and use the attention matrix between the agent and the lane segment as the target lane segment score lookup table score lane , and then use the lane segments with the top K scores as the target lane segment features It is expressed as follows:

[0037]

[0038] Where Softmax(·) is the softmax function, Argtop(·,) is the function of taking the TopK value index, Index(·) is the function of taking the value by index, K is the number of predicted target lane segments, and d lane is the feature K lane The dimension, Q agent is the query value of the attention mechanism, K lane is the key value of the attention mechanism, score lane is the lane segment score query table, and Top_K is the index value of the target lane segment.

[0039] C2. Obtaining coarse-grained trajectory features based on the target lane segment

[0040] First, the agent feature f in step A2 is agent Step B12 Local lane segment interaction features Interaction features with the local agent in step B14 The lane segment-agent global interaction feature f″ in step B2g and the target lane segment features in step C1 They are concatenated together and then input into MLP to generate coarse-grained trajectory features, which are expressed as follows:

[0041]

[0042] In the formula, feat traj is a coarse-grained trajectory feature.

[0043] C3: Refine coarse-grained trajectory features and decode multimodal trajectories

[0044] First, in order to perform high-level and adaptive adjustments to the coarse-grained motion trajectory of step C2, the learnable refined embedding R initialized by sampling from a uniform distribution is tokens Then, the coarse-grained trajectory features and refined embeddings are aggregated using element-wise dot product operations, and the aggregated features are extracted using 1D CNN. Finally, a three-layer MLP is used to decode the multimodal trajectory. The score of the target lane is taken as the probability p of the corresponding trajectory k , which is expressed as follows:

[0045]

[0046] E embs =feat traj ⊙R tokens

[0047]

[0048]

[0049]

[0050] In the formula, U(·) represents uniform distribution, d tokens is the dimension of the refined embedding, ⊙ is the element-wise dot product operation, E embs is a fine-grained trajectory feature, It is the feature obtained by fusion of fine-grained trajectory features and coarse-grained trajectory features. is a one-dimensional convolution operation, is a three-layer MLP, K is the number of predicted target lane segments, and also the number of predicted trajectories, is the predicted multimodal trajectory, p k is the probability of the predicted trajectory, the superscript t represents the time, the subscript k represents the sequence number of the predicted trajectory, and F is the number of future time steps predicted.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention uses local interactors and global interactors constructed by self-attention and cross-attention to model various social interactions in a multi-stage and hierarchical manner, taking into account traffic rules, road structure and information interaction, thereby ensuring that the model can achieve reliable, reasonable and interactively perceived predictions in various dynamic environments.

[0053] 2. The present invention focuses on the constraints and guiding role of the road network, samples target lane segments with richer information content than two-dimensional target points from the drivable area to guide the prediction process, and uses learnable embeddings to adaptively refine the prediction process, thereby promoting accurate, effective and scenario-consistent predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the process of the present invention.

[0055] Figure 2 Figure 2 is a framework diagram of a trajectory prediction model based on hierarchical progressive interaction and target lane segments.

[0056] Figure 3 This is the structure diagram of the lane segment encoder.

[0057] Figure 4 It is the structure diagram of the agent encoder.

[0058] Figure 5 It is a hierarchical progressive interactive network structure diagram. DETAILED DESCRIPTION

[0059] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings. Figure 1 As shown, a trajectory prediction method based on hierarchical progressive interaction and target lane segments includes the following steps:

[0060] A. Extracting Agent and Road Environment Features

[0061] A1. Use the positioning and perception module of the autonomous driving system to obtain M lane segments C consisting of 5 equidistant points within the range of 90 meters in front, 30 meters behind, and 60 meters on the left and right of the vehicle, and the states S of the surrounding N agents in the historical 10 time steps. The lane segment is the abbreviation of the lane centerline segment. The lane segment C and state S are expressed as follows:

[0062]

[0063] in, Indicates the information of the mth lane segment at the lth equidistant point, where m is the lane segment number and l is the sequence number of the equidistant point. Indicates the state of the nth agent at the tth time, n is the number of the agent, and t represents the time. In the embodiment of the present invention, the distance between adjacent equidistant points is 1 meter, and the interval between adjacent time steps is 0.1 second.

[0064] A2. The trajectory prediction model framework of the embodiment of the present invention is as follows Figure 2 As shown. For lane segments, the lane segment encoder is used to extract features f lane The lane segment encoder includes two stacked layers of multi-layer perceptrons (MLP), layer normalization units (LN) and maximum pooling (Pooling). The lane segment encoder structure is as follows: Figure 3 As shown; for the agent, the agent encoder is used to extract the feature f agent , the agent encoder structure is as follows Figure 4 As shown, the agent encoder includes 6 layers of stacked one-dimensional convolutional neural networks, namely 1D CNN and LN, which are expressed as follows:

[0065] f lane = LaneEnc(C; W lane )

[0066] f agent =AgentEnc(S;W agent ;)

[0067] In the formula, f lane is the lane segment feature, f agent is the agent feature, LaneEnc(·) is the lane segment encoder, and W lane is the weight of the lane segment encoder, AgentEnc(·) is the agent encoder, and W agent are the weights of the agent encoder.

[0068] B. Hierarchical and progressive modeling of local and global interactions

[0069] B1. Modeling Local Interactions

[0070] Based on self-attention and cross-attention, local interactions are progressively modeled. The structure is as follows Figure 5 As shown, the specific steps are as follows:

[0071] B11. Apply the agent-lane segment local interactor to model the relationship between the agent and the lane segment, which is expressed as follows:

[0072]

[0073] Where SA(·) is the self-attention layer, is the lane segment feature extracted by the self-attention layer, f a-l is the interaction feature between the agent and the lane segment captured by the cross-attention layer, is the lane segment feature fused with interactive features, CA(·) is the cross attention layer, It is a single-layer multilayer perceptron.

[0074] B12. Use lane segment-lane segment local interactors to learn the topological structure between lane segments, which is expressed as follows:

[0075]

[0076] In the formula, φ cat (·) is the concatenation operation, It is the local lane segment interaction feature.

[0077] B13. Use lane segment-agent local interactors to capture the fact that agents change their behaviors or intentions after fully understanding traffic rules and road topology, and adjust the model’s reasoning strategy to better adapt to real-time changing traffic scenarios, as shown below:

[0078]

[0079]

[0080] In the formula, is the agent feature extracted by the self-attention layer, f l-a is the interaction feature between lane segments and the agent captured by the cross-attention layer, It is the agent feature that integrates the interaction features.

[0081] B14. After comprehensively learning the information of the road network through steps B11, B12 and B13, the agent-agent local interactor captures the influence of the surrounding neighbors on the motion trajectory of the target agent and improves the accuracy of trajectory prediction, which is expressed as follows:

[0082]

[0083] In the formula, is the local agent interaction feature.

[0084] B2. Modeling global interactions

[0085] Based on self-attention modeling of global interactions, the structure is as follows Figure 5 As shown, the specific steps are as follows:

[0086] The agent-lane segment global interactor is used to further enhance the information interaction between the agent and the map, so that the agent can fully understand the surrounding environment, as shown below:

[0087]

[0088] f″ g =φ cat (SA(f g ),SA(f g ))

[0089] In the formula, f g is the concatenation of the local agent interaction feature and the local lane segment interaction feature, f″ g is the lane segment-agent global interaction feature.

[0090] C. Decoding multimodal trajectories based on predicted target lane segments

[0091] C1. Predict target lane segment

[0092] First, we use the local agent interaction features in step B14 Interaction features with the local lane segment in step B12 Calculate the agent's attention to the lane segment, and use the attention matrix between the agent and the lane segment as the target lane segment score lookup table score lane , and then use the lane segments with the highest scores before K=6 as the target lane segment features It is expressed as follows:

[0093]

[0094] Top_K = Argtop(score lane ,K)

[0095]

[0096] Where Softmax(·) is the softmax function, Argtop(·,·) is the function of taking the TopK value index, Index(·) is the function of taking the value by index, K is the number of predicted target lane segments, and d lane =256 is the characteristic K lane The dimension, Q agent is the query value of the attention mechanism, K lane is the key value of the attention mechanism, score lane is the lane segment score query table, and Top_K is the index value of the target lane segment.

[0097] C2. Obtaining coarse-grained trajectory features based on the target lane segment

[0098] First, the agent feature f in step A2 is agent Step B12 Local lane segment interaction features Interaction features with the local agent in step B14 The lane segment-agent global interaction feature f″ in step B2 g and the target lane segment features in step C1 They are concatenated together and then input into MLP to generate coarse-grained trajectory features, which are expressed as follows:

[0099]

[0100] In the formula, feat traj is a coarse-grained trajectory feature.

[0101] C3: Refine coarse-grained trajectory features and decode multimodal trajectories

[0102] First, in order to perform high-level and adaptive adjustments to the coarse-grained motion trajectory of step C2, the learnable refined embedding R initialized by sampling from a uniform distribution is tokens Then, the coarse-grained trajectory features and refined embeddings are aggregated using element-wise dot product operations, and the aggregated features are extracted using 1D CNN. Finally, a three-layer MLP is used to decode the multimodal trajectory. The score of the target lane is taken as the probability p of the corresponding trajectory k , which is expressed as follows:

[0103]

[0104] E embs =feat traj ⊙R tokens

[0105]

[0106]

[0107] In the formula, U(·) represents uniform distribution, d tokens is the dimension of the refined embedding, ⊙ is the element-wise dot product operation, E embs is a fine-grained trajectory feature, It is the feature obtained by fusion of fine-grained trajectory features and coarse-grained trajectory features. is a one-dimensional convolution operation, is a three-layer MLP, K is the number of predicted target lane segments, and also the number of predicted trajectories, is the predicted multimodal trajectory, p k is the probability of the predicted trajectory, the superscript t represents the time, the subscript k represents the sequence number of the predicted trajectory, and F=30 is the number of predicted future time steps. In the embodiment of the present invention, the interval between adjacent time steps is 0.1 seconds.

[0108] The above describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected.

Claims

1. A trajectory prediction method based on hierarchical progressive interaction and target lane segments, Features: The following steps are involved: A. Extracting Agent and Road Environment Features A1. Use the positioning and perception module of the autonomous driving system to obtain M lane segments C consisting of L equidistant points in the local range and the states S of the surrounding N agents in the historical T time steps. The lane segment is the abbreviation of the lane centerline segment. The lane segment C and the state S are expressed as follows: in, Indicates the information of the mth lane segment at the lth equidistant point, where m is the lane segment number and l is the sequence number of the equidistant point. represents the state of the nth agent at the tth time, where n is the agent number and t represents the time; A2. For lane segments, use the lane segment encoder to extract features f lane , the lane segment encoder includes N L The stacked multilayer perceptron is MLP, the layer normalization unit is LN unit and the maximum pooling unit is Pooling unit; for the agent, the agent encoder is used to extract the feature f agent , the agent encoder includes N A The stacked one-dimensional convolutional neural network, namely 1DCNN and LN unit, is represented as follows: f lane =LaneEnc(C;W lane ) f agent =AgentEnc(S;W agent ) In the formula, f lane is the lane segment feature, f agent is the agent feature, LaneEnc(·) is the lane segment encoder, and W lane is the weight of the lane segment encoder, AgentEnc(·) is the agent encoder, and W agent are the weights of the agent encoder; B. Hierarchical and progressive modeling of local and global interactions B1. Modeling local interactions Based on self-attention and cross-attention, local interactions are progressively modeled. The specific steps are as follows: B11. Apply the agent-lane segment local interactor to model the relationship between the agent and the lane segment, which is expressed as follows: Where SA(·) is the self-attention layer, is the lane segment feature extracted by the self-attention layer, f a-l is the interaction feature between the agent and the lane segment captured by the cross-attention layer, is the lane segment feature fused with interactive features, CA(·) is the cross attention layer, It is a single-layer MLP; B12. Use lane segment-lane segment local interactors to learn the topological structure between lane segments, which is expressed as follows: In the formula, φ cat (·) is the concatenation operation, is the local lane segment interaction feature; B13. Use lane segment-agent local interactors to capture the fact that agents change their behaviors or intentions after fully understanding traffic rules and road topology, and adjust the model’s reasoning strategy to better adapt to real-time changing traffic scenarios, as shown below: In the formula, is the agent feature extracted by the self-attention layer, f l-a is the interaction feature between lane segments and agents captured by the cross-attention layer, It is the agent feature that integrates the interactive features; B14. After comprehensively learning the information of the road network through steps B11, B12 and B13, the agent-agent local interactor captures the influence of the surrounding neighbors on the motion trajectory of the target agent and improves the accuracy of trajectory prediction, which is expressed as follows: In the formula, is the local agent interaction feature; B2. Modeling global interactions Modeling global interactions based on self-attention, the specific steps are as follows: The agent-lane segment global interactor is used to further enhance the information interaction between the agent and the map, so that the agent can fully understand the surrounding environment, as shown below: f″ g =φ cat (IN(f g ),SA(f g )) In the formula, f g is the concatenation of the local agent interaction feature and the local lane segment interaction feature, f″ g is the lane segment-agent global interaction feature; C. Decoding multimodal trajectories based on predicted target lane segments C1. Predict target lane segment First, we use the local agent interaction features in step B14 Interaction features with the local lane segment in step B12 Calculate the agent's attention to the lane segment, and use the attention matrix between the agent and the lane segment as the target lane segment score lookup table score lane , and then use the lane segments with the top K scores as the target lane segment features It is expressed as follows: Top_K=Argtop(score lane ,K) Where Softmax(·) is the softmax function, Argtop(·,·) is the function of taking the TopK value index, Index(·) is the function of taking the value by index, K is the number of predicted target lane segments, and d lane is the feature K lane The dimension, Q agent is the query value of the attention mechanism, K lane is the key value of the attention mechanism, score lane is the lane segment score query table, and Top_K is the index value of the target lane segment; C2. Obtaining coarse-grained trajectory features based on the target lane segment First, the agent feature f in step A2 is agent Step B12 Local lane segment interaction features Interaction features with the local agent in step B14 Lane segment-agent global interaction feature f' in step B2 g ' and the target lane segment features of step C1 They are concatenated together and then input into MLP to generate coarse-grained trajectory features, which are expressed as follows: In the formula, feat traj is a coarse-grained trajectory feature; C3: Refine coarse-grained trajectory features and decode multimodal trajectories First, in order to perform high-level and adaptive adjustments to the coarse-grained motion trajectory of step C2, the learnable refined embedding R initialized by sampling from a uniform distribution is tokens ; Then, the coarse-grained trajectory features and refined embeddings are aggregated using element-wise dot product operations, and the aggregated features are extracted using 1D CNN; finally, a three-layer MLP is used to decode the multimodal trajectory The score of the target lane is taken as the probability p of the corresponding trajectory k , which is expressed as follows: E embs =feat traj ⊙R tokens In the formula, U(·) represents uniform distribution, d tokens is the dimension of the refined embedding, ⊙ is the element-wise dot product operation, E enbs is a fine-grained trajectory feature, It is the feature obtained by fusion of fine-grained trajectory features and coarse-grained trajectory features. is a one-dimensional convolution operation, is a three-layer MLP, K is the number of predicted target lane segments, and also the number of predicted trajectories, is the predicted multimodal trajectory, p k is the probability of the predicted trajectory, the superscript t represents the time, the subscript k represents the sequence number of the predicted trajectory, and F is the number of future time steps predicted.

Images