A multi-target trajectory prediction method based on multi-attention mechanism

Abstract

The present invention discloses a multi-target trajectory prediction method based on a multi-attention mechanism, comprising the following steps: vectorizing the trajectory information and road information of traffic participants in a traffic scene, dividing the global traffic scene into N local scenes, and establishing a local coordinate system; based on the local coordinate system, aggregating the spatial, temporal, and interaction information of traffic participants within each local scene, and generating queries containing the spatiotemporal characteristics of traffic participants in parallel; aggregating the queries containing the spatiotemporal characteristics of traffic participants to extract scene-level long-distance interaction features; and generating predicted trajectories based on the queries containing the spatiotemporal characteristics of traffic participants and the scene-level long-distance interaction features. This method can more efficiently extract interaction features between traffic participants in local scenes and adaptively eliminate redundant interactions. Furthermore, by generating queries in parallel within a single inference step, it can achieve fast inference and accelerate trajectory generation.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and in particular relates to a multi-target trajectory prediction method based on a multi-attention mechanism. Background Art

[0002] Trajectory prediction modules, bridging environmental perception and planning decisions, are crucial in high-level autonomous driving systems. Real-time, accurate predictions significantly improve system planning efficiency and overall safety. Currently, deep learning-based methods, which can extract and learn high-dimensional features from data, are expected to achieve optimal prediction results and are widely recognized in academia and industry.

[0003] The essence of the trajectory prediction problem is a sequence-to-sequence problem. Currently, many academic studies use recurrent neural networks or Transformer architectures to deal with trajectory prediction problems. This method uses an encoder to encode the historical trajectory information of vehicles in traffic scenes into high-dimensional feature vectors, and the decoder predicts the future trajectory of the vehicle through the time series features in these high-dimensional feature vectors. Although the above methods have been successful in trajectory prediction tasks, there are still some problems that need to be solved. First, the recurrent neural network solution that is good at processing time series information has the problem of insufficient utilization of other information in traffic scenes, such as spatial information, interaction information between vehicles, and road information, resulting in low prediction accuracy and poor robustness of the prediction results of this method; secondly, the method of aggregating multimodal information through the attention mechanism aggregates too much redundant information, resulting in slow prediction speed and low real-time performance, making it difficult to apply to autonomous driving scenarios. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a multi-target trajectory prediction method based on a multi-attention mechanism to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above objectives, the present invention provides a multi-target trajectory prediction method based on a multi-attention mechanism, comprising the following steps:

[0006] Vectorize the trajectory information and road information of traffic participants in the traffic scene, divide the global traffic scene into N local scenes, and establish a local coordinate system;

[0007] Based on the local coordinate system, spatial information, temporal information, and interaction information of traffic participants within each local scene are aggregated, and queries containing spatiotemporal characteristics of traffic participants are generated in parallel;

[0008] Summarizing the queries containing the spatiotemporal characteristics of traffic participants and extracting scene-level long-distance interaction features;

[0009] A predicted trajectory is generated based on the query containing the spatiotemporal characteristics of traffic participants and the scene-level long-distance interaction features.

[0010] Preferably, the method for establishing a local coordinate system includes:

[0011] The traffic scene is converted into vector information, and the global scene is converted into multiple local scenes. An object to be predicted in the local scene is selected as the central object, the current coordinates of the central traffic participant are used as the origin, and the displacement of the central traffic participant from the previous frame to the current frame is used as the reference direction to establish a local coordinate system.

[0012] Preferably, the method for generating a query containing spatiotemporal characteristics of traffic participants includes:

[0013] A traffic map is constructed with each traffic participant in the local scene as a node, and the interaction information and spatial features greater than a preset impact threshold in the traffic map are extracted based on the multi-head sparse graph attention mechanism; the temporal features of the local scene are aggregated using the multi-head self-attention mechanism; the vectorized map information is mapped to a high-dimensional space through MLP to generate road embeddings, and based on the spatial features, temporal features and road embeddings, potential traffic rule information is extracted through the cross-attention mechanism; based on the spatial features, temporal features, interaction information, potential traffic rule information and traffic participant position encoding, queries containing the spatiotemporal features of traffic participants are generated in parallel.

[0014] Preferably, the method for acquiring the interaction information and spatial features includes:

[0015] Map the historical position information of the central participant and surrounding participants in the local scene to a high-dimensional space to generate position embedding and relative position embedding;

[0016] Calculating the attention coefficients between nodes in the graph based on the position embedding and the relative position embedding;

[0017] Obtain the characteristics of the central traffic participant at a certain moment through the attention coefficient between the nodes in the graph;

[0018] Based on the multi-head attention mechanism, the features at a certain moment are summarized to obtain a multi-head feature vector;

[0019] Perform residual connection and normalization on the multi-head feature vectors to obtain a feature vector set containing interactive information.

[0020] Preferably, the calculation expression of the attention coefficient between nodes in the graph is:

[0021]

[0022] Where, represents the degree of influence of traffic participant j on traffic participant i at time t, N represents the total number of nodes in the graph, α-entmax(·) represents an activation function with sparse characteristics, LeakyReLU represents a leaky rectified linear unit with a negative input slope of 0.1, a represents the weight vector of a single-layer feedforward neural network, W3 and W4 represent learnable parameter matrices, φ represents the embedding function MLP, represents the embedding of traffic participant i at time t, Represents the relative position relationship embedding of the two central participants i and j.

[0023] Preferably, the characteristic expression of the central traffic participant i at time t is:

[0024]

[0025] Where, is the feature vector of traffic participant i at time t containing spatial information and interaction information, ELU is the activation function, represents the degree of influence of traffic participant k on traffic participant i at time t, represents the embedding of traffic participant k at time t.

[0026] Preferably, the method of aggregating temporal features of a local scene includes:

[0027] Add a sinusoidal position marker to the set of feature vectors containing interactive information to obtain the input vector;

[0028] Use a learnable weight matrix to convert the input vector into a query, key, and value, and input the query, key, and value vectors into the self-attention module to obtain the output vector;

[0029] The output vectors are aggregated based on a multi-head self-attention mechanism and residual connections and normalization operations are performed to obtain temporal features.

[0030] Preferably, the query expression containing the spatiotemporal characteristics of traffic participants is:

[0031]

[0032] Where E is the set of position embeddings generated by traffic participants through MLP, G is the vector set of spatiotemporal information and interaction information generated by the temporal attention mechanism module, L is the vector set containing hidden traffic rules and driving intentions, PE is the sinusoidal position embedding, and MLP represents the embedding function.

[0033] Preferably, the method for extracting scene-level long-distance interaction features includes: mapping the relative position relationship between different local coordinate systems to a high-dimensional space through MLP to generate relative position embedding; aggregating features in different local scenes through a multi-head cross-attention mechanism to obtain scene-level long-distance interaction features.

[0034] Preferably, after generating the predicted trajectory, the uncertainty of the trajectory is fitted by Laplace distribution.

[0035] Compared with the prior art, the present invention has the following advantages and technical effects:

[0036] (1) The present invention uses a sparse graph attention learning method to capture the interaction relationship between traffic participants in traffic scenes. This method can more efficiently extract the interaction features between traffic participants in local scenes and can adaptively eliminate redundant interactions.

[0037] (2) The present invention uses a random non-autoregressive query generation method to generate queries in parallel in one reasoning step, which can achieve fast reasoning and accelerate the speed of trajectory generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0039] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0040] Figure 2 A flowchart of an embodiment of the present invention;

[0041] Figure 3 This is a flow chart of a local encoder according to an embodiment of the present invention;

[0042] Figure 4 This is a network structure diagram of the sparse graph attention mechanism, temporal self-attention mechanism, and road attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Example 1

[0046] like Figure 1-4 As shown, this embodiment provides a multi-target trajectory prediction method based on a multi-attention mechanism, including the following steps:

[0047] Vectorize the trajectory information and road information of traffic participants in the traffic scene, divide the global traffic scene into N local scenes, and establish a local coordinate system;

[0048] Based on the local coordinate system, spatial information, temporal information, and interaction information of traffic participants within each local scene are aggregated, and queries containing spatiotemporal characteristics of traffic participants are generated in parallel;

[0049] Summarizing the queries containing the spatiotemporal characteristics of traffic participants and extracting scene-level long-distance interaction features;

[0050] A predicted trajectory is generated based on the query containing the spatiotemporal characteristics of traffic participants and the scene-level long-distance interaction features.

[0051] The specific implementation steps are as follows:

[0052] S1. Preprocessing of scene data: vectorizing the trajectory information of traffic participants and road information in the traffic scene, and dividing the global scene into multiple local scenes.

[0053] S1 includes: converting the input traffic participant trajectory and map information into vector information; converting the global scene into multiple local scenes; selecting an object to be predicted in the local scene as the central object, taking the current coordinates of the central traffic participant as the origin, and taking the displacement of the central traffic participant from the previous frame to the current frame as the reference direction, to establish a local coordinate system.

[0054] In this embodiment, the vectorized traffic scene includes traffic participants, traffic participant historical trajectory information and map information. The scene contains N traffic participants, where the trajectory information of the i-th participant can be expressed as: P i ={P i G ,P i C}. Among them, P i G represents the trajectory information of traffic participant i in the global coordinate system, P i i Represents the trajectory information in the local coordinate system, where the origin is the position of traffic participant i at time t. The direction is the reference direction. The local trajectory information is expressed as: in, represents the location information of traffic participant i at time t, represents the historical location information of traffic participant i at time t-1, Represents the predicted position information of traffic participant i at time t+1. The global coordinate system trajectory is: in, and Similarly, it corresponds to the position information in the global coordinate system. The position information of traffic participant i at each moment can be expressed as: in, Represents the coordinates in the local coordinate system, [x t ,y t ] represents the coordinates in the global coordinate system.

[0055] The displacement of traffic participant i in the local coordinate system is expressed as: Where Δp i represents the displacement of i from time t-1 to time t. The relative position relationship between i and traffic participant j can be expressed as: Where Δp i,j Represents the relative position relationship between traffic participants i and j in the local coordinate system.

[0056] Road segment information is represented as: in, and Indicates the starting point and end point of the road, and the relative position relationship between traffic participants and the road is expressed as

[0057] After vectorizing the information, the scene is divided into local scenes of equal size with the target to be predicted as the central object.

[0058] S2. Local scene encoder aggregates the spatial information, temporal information, and interaction information of traffic participants in a local area and generates queries containing the spatiotemporal features of traffic participants.

[0059] S2 includes: mapping the historical trajectory information in the local scene to a high-dimensional space through MLP to generate an embedding as the input of the subsequent network; constructing a graph with each traffic participant in the local scene as a node, extracting the interactive information and spatial features that have a greater impact on the central traffic participant based on the multi-head sparse graph attention mechanism, and adaptively eliminating the interactive relationships with less impact. The impact size is reflected by the attention coefficient through the α-entmax activation function, and the value reflected by the α-entmax activation function is set as the threshold to judge the impact size; using the multi-head self-attention mechanism to aggregate the temporal features of the local scene; mapping the vectorized map information to a high-dimensional space through MLP to generate a road embedding, which is used together with the spatiotemporal feature vector of the traffic participant as the input of the subsequent network, and extracting the potential traffic rule information through the cross-attention mechanism; the non-autoregressive query generation module generates queries containing the above information features in parallel through spatiotemporal features, interactive information, potential traffic rule information and traffic participant position encoding.

[0060] In this embodiment, the local encoder structure is as follows Figure 3 First, the historical position information of the central actor and surrounding actors in the local scene is mapped to a high-dimensional space to generate position embedding and relative position embedding: in, is the embedding of traffic participant i at time t, φ is the multi-layer perceptron MLP, and W1 is the trainable weight matrix. is the relative position relationship embedding of the two centered participants i and j, and W2 is a trainable weight matrix.

[0061] A graph is constructed with traffic participants as nodes and the relationships between them as edges. The multi-head sparse graph attention mechanism is used to extract important spatial and interactive relationships between participants and propose unimportant relationships. The attention coefficient between nodes in the graph is first calculated:

[0062]

[0063] in, where α-entmax(·) represents the degree of influence of traffic participant j on traffic participant i at time t, N represents the total number of nodes in the graph, α-entmax(·) is an activation function with sparse properties, LeakyReLU represents a leaky rectified linear unit with a negative input slope of 0.1, a is the weight vector of a single-layer feedforward neural network, W3 and W4 are learnable parameter matrices, and φ is an embedding function MLP with trainable weights W4. The feature representation of the central traffic participant i at time t can then be updated based on the corresponding attention score: in, is the feature vector of traffic participant i at time t, which contains spatial information and interaction information. ELU is the activation function. Following the multi-head attention mechanism, the feature aggregation process can be performed in multiple subspaces, and finally the multi-head feature vector is summarized: Among them, φ is the multi-layer perceptron MLP, is the feature vector of the kth head, and W5 is the learnable parameter matrix.

[0064] Perform residual connections: and normalized: Finally, we get a set of feature vectors containing interactive information: The network structure of the sparse graph attention mechanism is as follows Figure 4 shown.

[0065] In H i The vector is followed by a sinusoidal position marker to form the input vector of the temporal self-attention module, S i =H i +T i Among them, T i is the position embedding, generated by:

[0066]

[0067] The resulting vector set Serves as the input of the temporal self-attention module through S i Compute queries, keys, and values: in is a learnable weight matrix. The generated query, key, and value vectors are input into the self-attention module: Among them, M is a mask, which makes the model pay more attention to the information from the previous time. Finally, the multi-head information is summarized and the residual connection and normalization operations are performed on the obtained vector: Among them, W6 is the learnable parameter matrix and LN is the normalization operation. The network structure of the temporal self-attention mechanism is as follows Figure 4 shown.

[0068] The vector containing spatiotemporal features obtained in the previous step will be used as input to the traffic participant-road interaction module. The participant-road interaction module uses a cross-attention mechanism to extract potential traffic rules information from road information. First, the road vector information is mapped to a high-dimensional vector: in, is the lane-related embedding, and are the start and end points of the lane segment, and W7 is a learnable matrix. Then, the query, key, and value are calculated using the road embedding and the vector containing spatiotemporal information obtained in the previous step: in, is a learnable parameter matrix. The query, key, and value are input to the crisscross attention module: Finally, similar to the above operation, the multi-head information is aggregated and normalized to obtain the feature vector containing spatiotemporal information and road interaction information. The network structure diagram is as follows: Figure 4 shown.

[0069] The query generation module takes a vector containing spatiotemporal information and interaction information as input and generates queries in parallel using a non-autoregressive approach:

[0070]

[0071] Among them, E is the set of position embeddings generated by traffic participants through MLP, G is the vector set of spatiotemporal information and interaction information generated by the time attention mechanism module, L is the vector set containing hidden traffic rules and driving intentions, and PE is the sinusoidal position embedding, which is generated in the same way as T i The query generation module can generate multiple queries in parallel during one inference step, and the generated queries serve as the input of the global interaction module.

[0072] S3. Global interaction encoder, used to summarize the features provided by the local encoders and extract scene-level long-range interaction features.

[0073] S3 includes: mapping the relative position relationship between different local coordinate systems to a high-dimensional space through MLP to generate relative position embedding; the position embedding is used together with the local features as the input of the subsequent network, and the features in different local scenes are aggregated through the multi-head cross attention mechanism;

[0074] In this embodiment, the global interaction module is used to aggregate the local features extracted by the local encoder to generate long-range scene-level interaction features. This module aggregates the information provided by the local scene through a cross-attention mechanism. First, the relative position relationship of different local scenes is mapped to a high-dimensional space to generate a local relative position embedding: in, The angle difference between the two local coordinate systems, W8 is a learnable parameter matrix. The query, key and value are calculated by embedding the local feature vector and the relative position: in, The query generated by the local encoder, W Q , W K , W V is a learnable matrix. The query, key, and value are fed into the crisscross attention mechanism layer: Aggregate multi-head features, perform normalization and residual connection to obtain the final feature set used as decoder input:

[0075]

[0076] S4. The decoder generates a predicted trajectory based on the high-dimensional feature vector generated by the encoder.

[0077] The S4 includes: receiving local information and global interaction information and generating a predicted trajectory using MLP; and fitting the uncertainty of the trajectory through Laplace distribution.

[0078] In this example, for each target to be predicted, an MLP receives local and global representations as input and outputs the position of the traffic agent in the local coordinate system at each future time step. The output tensor of the regression head has a shape of [F, N, H, 4], where F is the number of different distributions in the mixture model, N is the total number of traffic agents in the scene, and H is the number of future time steps to predict. Another MLP and a softmax function are used to calculate the mixing coefficients of the mixture model for each agent, which have a shape of [N, F].

[0079] In addition, the loss function in the model training process in this example consists of two parts: in, is the regression loss function, is the classification loss function. Where P(·|·) is the probability density function of the Laplace distribution, and is the position and uncertainty of the best predicted trajectory. The cross entropy loss function is used to calculate The negative log-likelihood function of the Laplace distribution is used as the regression loss function. The cross-entropy loss is used as the classification loss function. Their weights are equal.

[0080] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multi-target trajectory prediction method based on a multi-attention mechanism, characterized in that: The following steps are involved: Vectorize the trajectory information and road information of traffic participants in the traffic scene, divide the global traffic scene into N local scenes, and establish a local coordinate system; Based on the local coordinate system, spatial information, temporal information, and interaction information of traffic participants within each local scene are aggregated, and queries containing spatiotemporal characteristics of traffic participants are generated in parallel; The method for generating a query containing spatiotemporal characteristics of traffic participants includes: A traffic map is constructed with each traffic participant in the local scene as a node. The interaction information and spatial features greater than a preset impact threshold in the traffic map are extracted using a multi-head sparse graph attention mechanism. The temporal features of the local scene are aggregated using a multi-head self-attention mechanism. The vectorized map information is mapped to a high-dimensional space using an MLP to generate a road embedding. Based on the spatial features, temporal features, and road embeddings, the cross-attention mechanism is used to extract potential traffic rule information. Queries containing the spatiotemporal features of traffic participants are generated in parallel based on the spatial features, temporal features, interaction information, potential traffic rule information, and traffic participant position encodings. The query expression containing the spatiotemporal characteristics of traffic participants is: Where E is the set of position embeddings generated by the MLP for traffic participants, G is the vector set of spatiotemporal information and interaction information generated by the temporal attention mechanism module, L is the vector set containing hidden traffic rules and driving intentions, PE is the sinusoidal position embedding, and MLP represents the embedding function. Summarizing the queries containing the spatiotemporal characteristics of traffic participants and extracting scene-level long-distance interaction features; The method for extracting scene-level long-distance interaction features includes: mapping the relative position relationship between different local coordinate systems to a high-dimensional space through MLP to generate relative position embedding; aggregating features in different local scenes through a multi-head cross-attention mechanism to obtain scene-level long-distance interaction features; A predicted trajectory is generated based on the query containing the spatiotemporal characteristics of traffic participants and the scene-level long-distance interaction features.

2. The multi-target trajectory prediction method based on the multi-attention mechanism according to claim 1 is characterized in that: The method for establishing a local coordinate system includes: The traffic scene is converted into vector information, and the global scene is converted into multiple local scenes. An object to be predicted in the local scene is selected as the central object, the current coordinates of the central traffic participant are used as the origin, and the displacement of the central traffic participant from the previous frame to the current frame is used as the reference direction to establish a local coordinate system.

3. The multi-target trajectory prediction method based on the multi-attention mechanism according to claim 1 is characterized in that: The method for acquiring the interactive information and spatial features includes: Map the historical position information of the central participant and surrounding participants in the local scene to a high-dimensional space to generate position embedding and relative position embedding; Calculating the attention coefficients between nodes in the graph based on the position embedding and the relative position embedding; Obtain the characteristics of the central traffic participant at a certain moment through the attention coefficient between the nodes in the graph; Based on the multi-head attention mechanism, the features at a certain moment are summarized to obtain a multi-head feature vector; Perform residual connection and normalization on the multi-head feature vectors to obtain a feature vector set containing interactive information.

4. The multi-target trajectory prediction method based on the multi-attention mechanism according to claim 3 is characterized in that: The calculation expression of the attention coefficient between nodes in the graph is: Where, represents the degree of influence of traffic participant j on traffic participant i at time t, N represents the total number of nodes in the graph, α-entmax(·) represents an activation function with sparse characteristics, LeakyReLU represents a leaky rectified linear unit with a negative input slope of 0.1, a represents the weight vector of a single-layer feedforward neural network, W3 and W4 represent learnable parameter matrices, φ represents the embedding function MLP, represents the embedding of traffic participant i at time t, Represents the relative position relationship embedding of the two central participants i and j.

5. The multi-target trajectory prediction method based on the multi-attention mechanism according to claim 4 is characterized in that: The characteristic expression of the central traffic participant i at time t is: Where, is the feature vector of traffic participant i at time t containing spatial information and interaction information, ELU is the activation function, represents the degree of influence of traffic participant k on traffic participant i at time t, represents the embedding of traffic participant k at time t.

6. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 1 is characterized in that The method for aggregating temporal features of a local scene includes: Add a sinusoidal position marker to the set of feature vectors containing interactive information to obtain the input vector; Use a learnable weight matrix to convert the input vector into a query, key, and value, and input the query, key, and value vectors into the self-attention module to obtain the output vector; The output vectors are aggregated based on a multi-head self-attention mechanism and residual connections and normalization operations are performed to obtain temporal features.

7. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 1 is characterized in that: After generating the predicted trajectory, the uncertainty of the trajectory is fitted by the Laplace distribution.

Images