Multi-target trajectory prediction method based on multi-attention mechanism

Abstract

The invention discloses a multi-target trajectory prediction method based on a multi-attention mechanism, and the method comprises the following steps: carrying out the vectorization processing of the trajectory information and road information of traffic participants in a traffic scene, dividing the global traffic scene into N local scenes, and building a local coordinate system; based on the local coordinate system, aggregating space information, time information and interaction information of the traffic participants in each local scene range, and generating queries containing spatial and temporal characteristics of the traffic participants in parallel; summarizing queries containing spatial-temporal characteristics of traffic participants, and extracting scene-level long-distance interaction characteristics; and generating a prediction trajectory according to the query containing the spatial-temporal characteristics of the traffic participants and the scene-level long-distance interaction characteristics. According to the method, interaction characteristics between traffic participants in a local scene can be extracted more efficiently, redundant interaction can be eliminated in a self-adaptive mode, query is generated in parallel in one reasoning step, rapid reasoning can be achieved, and the track generation speed is increased.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a multi-target trajectory prediction method based on a multi-attention mechanism. Background Art

[0002] As a bridge between environmental perception and planning decisions, the trajectory prediction module is crucial in high-level autonomous driving systems. Real-time and accurate prediction results can significantly improve system planning efficiency and overall system safety. At present, deep learning-based methods can extract and learn high-dimensional features in data and are expected to achieve optimal prediction results, which has been widely recognized by academia and industry.

[0003] The essence of trajectory prediction is a sequence-sequence problem. At present, many academic studies use recurrent neural networks or Transformer architectures to deal with trajectory prediction problems. This method encodes the historical trajectory information of vehicles in traffic scenes into high-dimensional feature vectors through an encoder, and the decoder predicts the future trajectory of the vehicle through the time series features in these high-dimensional feature vectors. Although the above methods have been successful in trajectory prediction tasks, there are still some problems that need to be solved. First, the recurrent neural network solution that is good at processing time series information has the problem of insufficient utilization of other information in traffic scenes, such as spatial information, interaction information between vehicles, and road information, resulting in low accuracy and poor robustness of the prediction results of this method; secondly, the method of aggregating multimodal information through the attention mechanism aggregates too much redundant information, resulting in slow prediction speed and low real-time performance of this method, which is difficult to apply to autonomous driving scenarios. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a multi-target trajectory prediction method based on a multi-attention mechanism to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides a multi-target trajectory prediction method based on a multi-attention mechanism, comprising the following steps:

[0006] Vectorize the trajectory information and road information of traffic participants in the traffic scene, divide the global traffic scene into N local scenes, and establish a local coordinate system;

[0007] Based on the local coordinate system, spatial information, temporal information and interaction information of traffic participants within each local scene are aggregated, and queries containing spatiotemporal characteristics of traffic participants are generated in parallel;

[0008] Aggregating the queries containing the spatiotemporal characteristics of traffic participants to extract scene-level long-distance interaction features;

[0009] A predicted trajectory is generated based on the query containing the spatiotemporal characteristics of traffic participants and the scene-level long-distance interaction features.

[0010] Preferably, the method for establishing a local coordinate system comprises:

[0011] The traffic scene is converted into vector information, and the global scene is converted into multiple local scenes; an object to be predicted in the local scene is selected as the central object, the current coordinates of the central traffic participant are used as the origin, and the displacement of the central traffic participant from the previous frame to the current frame is used as the reference direction to establish a local coordinate system.

[0012] Preferably, the method for generating a query containing spatiotemporal characteristics of traffic participants comprises:

[0013] A traffic map is constructed with each traffic participant in the local scene as a node, and the interactive information and spatial features in the traffic map that are greater than a preset impact threshold are extracted based on a multi-head sparse graph attention mechanism; the temporal features of the local scene are aggregated using a multi-head self-attention mechanism; the vectorized map information is mapped to a high-dimensional space through MLP to generate road embeddings, and based on the spatial features, temporal features and road embeddings, potential traffic rule information is extracted through a cross-attention mechanism; based on the spatial features, temporal features, interactive information, potential traffic rule information and traffic participant position encoding, queries containing the spatiotemporal features of traffic participants are generated in parallel.

[0014] Preferably, the method for acquiring the interaction information and spatial features includes:

[0015] Map the historical position information of the central participant and surrounding participants in the local scene to the high-dimensional space to generate position embedding and relative position embedding;

[0016] Calculate the attention coefficient between nodes in the graph based on the position embedding and the relative position embedding;

[0017] The characteristics of the central traffic participant at a certain moment are obtained through the attention coefficients between the nodes in the graph;

[0018] Based on the multi-head attention mechanism, the features at a certain moment are summarized to obtain a multi-head feature vector;

[0019] Residual connection and normalization are performed on the multi-head feature vectors to obtain a feature vector set containing interactive information.

[0020] Preferably, the calculation expression of the attention coefficient between nodes in the graph is:

[0021]

[0022] In the formula, represents the influence of traffic participant j on traffic participant i at time t, N represents the total number of nodes in the graph, α-entmax(·) represents an activation function with sparse characteristics, LeakyReLU represents a leaky rectified linear unit with a negative input slope of 0.1, a represents the weight vector of a single-layer feedforward neural network, W3 and W4 represent learnable parameter matrices, φ represents the embedding function MLP, represents the embedding of traffic participant i at time t, Represents the relative position relationship embedding of the two central participants i and j.

[0023] Preferably, the characteristic expression of the central traffic participant i at time t is:

[0024]

[0025] In the formula, is the feature vector containing spatial information and interaction information of traffic participant i at time t, ELU is the activation function, represents the influence of traffic participant k on traffic participant i at time t, represents the embedding of traffic participant k at time t.

[0026] Preferably, the method for aggregating temporal features of a local scene comprises:

[0027] Add a sinusoidal position marker after the feature vector set containing the interactive information to obtain the input vector;

[0028] Use a learnable weight matrix to convert the input vector into query, key, and value, input the query, key, and value vectors into the self-attention module to obtain the output vector;

[0029] Based on the multi-head self-attention mechanism, the output vectors are aggregated and residual connections and normalization operations are performed to obtain temporal features.

[0030] Preferably, the query expression containing the spatiotemporal characteristics of traffic participants is:

[0031]

[0032] Where E is the set of position embeddings generated by traffic participants through MLP, G is the vector set of spatiotemporal information and interaction information generated by the temporal attention mechanism module, L is the vector set containing hidden traffic rules and driving intentions, PE is the sinusoidal position embedding, and MLP represents the embedding function.

[0033] Preferably, the method for extracting scene-level long-distance interaction features includes: mapping the relative position relationship between different local coordinate systems to a high-dimensional space through MLP to generate relative position embedding; aggregating features in different local scenes through a multi-head cross-attention mechanism to obtain scene-level long-distance interaction features.

[0034] Preferably, after the predicted trajectory is generated, the uncertainty of the trajectory is fitted by Laplace distribution.

[0035] Compared with the prior art, the present invention has the following advantages and technical effects:

[0036] (1) The present invention uses a sparse graph attention learning method to capture the interaction relationship between traffic participants in traffic scenes. This method can more efficiently extract the interaction features between traffic participants in local scenes and adaptively eliminate redundant interactions.

[0037] (2) The present invention uses a random non-autoregressive query generation method to generate queries in parallel in one reasoning step, which can achieve fast reasoning and speed up trajectory generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0040] Figure 2 is a flowchart of an embodiment of the present invention;

[0041] Figure 3 A flowchart of a local encoder according to an embodiment of the present invention;

[0042] Figure 4 This is a network structure diagram of the sparse graph attention mechanism, temporal self-attention mechanism, and road attention mechanism of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Embodiment 1

[0046] like Figure 1-4 As shown, this embodiment provides a multi-target trajectory prediction method based on a multi-attention mechanism, including the following steps:

[0047] Vectorize the trajectory information and road information of traffic participants in the traffic scene, divide the global traffic scene into N local scenes, and establish a local coordinate system;

[0048] Based on the local coordinate system, spatial information, temporal information and interaction information of traffic participants within each local scene are aggregated, and queries containing spatiotemporal characteristics of traffic participants are generated in parallel;

[0049] Aggregating the queries containing the spatiotemporal characteristics of traffic participants to extract scene-level long-distance interaction features;

[0050] A predicted trajectory is generated based on the query containing the spatiotemporal characteristics of traffic participants and the scene-level long-distance interaction features.

[0051] The specific implementation steps are as follows:

[0052] S1. Preprocessing of scene data: vectorizing the trajectory information of traffic participants and road information in the traffic scene, and dividing the global scene into multiple local scenes.

[0053] S1 includes: converting the input traffic participant trajectory and map information into vector information; converting the global scene into multiple local scenes; selecting an object to be predicted in the local scene as the central object, taking the current moment coordinates of the central traffic participant as the origin, and taking the displacement of the central traffic participant from the previous frame to the current frame as the reference direction, to establish a local coordinate system.

[0054] In this embodiment, the vectorized traffic scene includes traffic participants, historical trajectory information of traffic participants and map information. The scene includes N traffic participants, where the trajectory information of the i-th participant can be expressed as: i = {P i G ,P i C}. Among them, P i G represents the trajectory information of traffic participant i in the global coordinate system, P i i represents the trajectory information in the local coordinate system, where the origin is the position of traffic participant i at time t. The direction is the reference direction. The local trajectory information is expressed as: in, represents the location information of traffic participant i at time t, represents the historical location information of traffic participant i at time t-1, Represents the predicted position information of traffic participant i at time t+1. The global coordinate system trajectory is: in, and Similarly, it corresponds to the position information in the global coordinate system. The position information of traffic participant i at each moment can be expressed as: in, represents the coordinates in the local coordinate system, [x t ,y t ] represents the coordinates in the global coordinate system.

[0055] The displacement of traffic participant i in the local coordinate system is expressed as: Among them, Δp i represents the displacement of i from time t-1 to time t. The relative position relationship between i and traffic participant j can be expressed as: Among them, Δp i,j Represents the relative position relationship between traffic participants i and j in the local coordinate system.

[0056] Road segment information is represented as: in, and represents the starting point and end point of the road, and the relative position relationship between the traffic participants and the road is expressed as

[0057] After vectorizing the information, the scene is divided into local scenes of equal size with the target to be predicted as the central object.

[0058] S2. Local scene encoder aggregates the spatial information, temporal information and interaction information of traffic participants in the local area and generates queries containing the spatiotemporal characteristics of traffic participants.

[0059] S2 includes: mapping the historical trajectory information in the local scene to a high-dimensional space through MLP to generate an embedding as the subsequent network input; constructing a graph with each traffic participant in the local scene as a node, extracting the interactive information and spatial features that have a greater impact on the central traffic participant based on the multi-head sparse graph attention mechanism, and adaptively eliminating the interactive relationships with less impact. The impact size is reflected by the attention coefficient through the α-entmax activation function, and the value reflected by the α-entmax activation function is set as the threshold to judge the impact size; using the multi-head self-attention mechanism to aggregate the temporal features of the local scene; mapping the vectorized map information to a high-dimensional space through MLP to generate a road embedding, which is used together with the spatiotemporal feature vector of the traffic participant as the subsequent network input, and extracting the potential traffic rule information through the cross-attention mechanism; the non-autoregressive query generation module generates queries containing the above information features in parallel through spatiotemporal features, interactive information, potential traffic rule information and traffic participant position encoding.

[0060] In this embodiment, the local encoder structure is as follows: Figure 3 First, the historical position information of the central participant and the surrounding participants in the local scene is mapped to the high-dimensional space to generate position embedding and relative position embedding: in, is the embedding of traffic participant i at time t, φ is the multi-layer perceptron MLP, and W1 is the trainable weight matrix. is the relative position relationship embedding of the two centered participants i and j, and W2 is a trainable weight matrix.

[0061] A graph is built with traffic participants as nodes and the relationships between them as edges. The multi-head sparse graph attention mechanism is used to extract important spatial and interactive relationships between participants and propose unimportant relationships. The attention coefficient between nodes in the graph is first calculated:

[0062]

[0063] in, Where the influence degree of traffic participant j on traffic participant i at time t, N represents the total number of nodes in the graph, α-entmax(·) is an activation function with sparse characteristics, LeakyReLU represents a leaky rectified linear unit with a negative input slope of 0.1, a is the weight vector of a single-layer feedforward neural network, W3 and W4 are learnable parameter matrices, and φ is an embedding function MLP with a trainable weight W4. Afterwards, the feature representation of the central traffic participant i at time t can be updated according to the corresponding attention score: in, is the feature vector of traffic participant i at time t containing spatial information and interaction information, and ELU is the activation function. Following the multi-head attention mechanism, the feature aggregation process can be performed in multiple subspaces, and finally the multi-head feature vector is summarized: Among them, φ is the multi-layer perceptron MLP, is the feature vector of the kth head, and W5 is the learnable parameter matrix.

[0064] Perform a residual connection: And normalized: Finally, we get a set of feature vectors containing interactive information: The network structure of the sparse graph attention mechanism is as follows Figure 4 shown.

[0065] In H i The vector is followed by a sinusoidal position marker to form the input vector of the temporal self-attention module, S i =H i +T i Among them, T i is the position embedding, generated by:

[0066]

[0067] The resulting vector set As the input of the temporal self-attention module, through S i Compute queries, keys, and values: in is a learnable weight matrix. The generated query, key, and value vectors are input into the self-attention module: Among them, M is a mask, which makes the model pay more attention to the information from the previous time. Finally, the multi-head information is summarized and the residual connection and normalization operations are performed on the obtained vector: Among them, W6 is a learnable parameter matrix and LN is a normalization operation. The network structure of the temporal self-attention mechanism is as follows Figure 4 shown.

[0068] The vector containing spatiotemporal features obtained in the previous step will be used as the input of the traffic participant and road interaction module. The participant and road interaction module uses a cross-attention mechanism to extract potential traffic rules information from road information. First, the vector information of the road is mapped into a high-dimensional vector: in, is the lane-related embedding, and is the start and end point of the lane, and W7 is a learnable matrix. Then, the query, key, and value are calculated using the road embedding and the vector containing spatiotemporal information obtained in the previous step: in, is a learnable parameter matrix. The query, key, and value are input to the criss-cross attention module: Finally, similar to the above operation, the multi-head information is summarized and normalized to obtain the feature vector containing spatiotemporal information and road interaction information. The network structure diagram is as follows: Figure 4 shown.

[0069] The query generation module takes a vector containing spatiotemporal information and interaction information as input and generates queries in parallel in a non-autoregressive manner:

[0070]

[0071] Among them, E is the set of position embeddings generated by traffic participants through MLP, G is the vector set of spatiotemporal information and interaction information generated by the temporal attention mechanism module, L is the vector set containing hidden traffic rules and driving intentions, and PE is the sinusoidal position embedding, which is generated in the same way as T i The query generation module can generate multiple queries in parallel during one-step reasoning, and the generated queries serve as the input of the global interaction module.

[0072] S3. Global interaction encoder, used to summarize the features provided by the local encoders and extract scene-level long-range interaction features.

[0073] S3 includes: mapping the relative position relationship between different local coordinate systems to high-dimensional space through MLP to generate relative position embedding; the position embedding is used as the input of the subsequent network together with the local features, and the features in different local scenes are summarized through the multi-head cross attention mechanism;

[0074] In this embodiment, the global interaction module is used to summarize the local features extracted by the local encoder to get the long-range scene-level interaction features. This module summarizes the information provided by the local scene through the cross-attention mechanism. First, the relative position relationship of different local scenes is mapped to a high-dimensional space to generate a local relative position embedding: in, The angle difference between the two local coordinate systems, W8 is the learnable parameter matrix. The query, key and value are calculated by embedding the local feature vector and relative position: in, is the query generated by the local encoder, W Q , W K , W V is a learnable matrix. The query, key, and value are fed into the criss-cross attention mechanism layer: Aggregate multi-head features, perform normalization and residual connection to obtain the final feature set used as decoder input:

[0075]

[0076] S4. The decoder generates a prediction trajectory based on the high-dimensional feature vector generated by the encoder.

[0077] The S4 includes: receiving local information and global interaction information and generating a predicted trajectory using MLP; and fitting the uncertainty of the trajectory through Laplace distribution.

[0078] In this example, for each target to be predicted, the MLP receives local and global representations as input and outputs the position of the traffic actor at each future time step in the local coordinate system. The output tensor of the regression head has the shape of [F, N, H, 4], where F is the number of different distributions in the mixture model, N is the total number of traffic actors in the scene, and H is the number of future time steps predicted. Another MLP and a softmax function are used to calculate the mixing coefficients of the mixture model for each agent, which has the shape of [N, F].

[0079] In addition, the loss function in the model training process in this example consists of two parts: in, is the regression loss function, is the classification loss function. Where P(·|·) is the probability density function of the Laplace distribution, and is the position and uncertainty of the best predicted trajectory. The cross entropy loss function is used to calculate The negative log-likelihood function of the Laplace distribution is used as the regression loss function. The cross entropy loss is used as the classification loss function. Their weights are equal.

[0080] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-target trajectory prediction method based on a multi-attention mechanism, characterized in that: The following steps are involved: Vectorize the trajectory information and road information of traffic participants in the traffic scene, divide the global traffic scene into N local scenes, and establish a local coordinate system; Based on the local coordinate system, spatial information, temporal information and interaction information of traffic participants within each local scene are aggregated, and queries containing spatiotemporal characteristics of traffic participants are generated in parallel; Aggregating the queries containing the spatiotemporal characteristics of traffic participants to extract scene-level long-distance interaction features; A predicted trajectory is generated based on the query containing the spatiotemporal characteristics of traffic participants and the scene-level long-distance interaction features.

2. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 1 is characterized in that: The method for establishing a local coordinate system comprises: The traffic scene is converted into vector information, and the global scene is converted into multiple local scenes; an object to be predicted in the local scene is selected as the central object, the current coordinates of the central traffic participant are used as the origin, and the displacement of the central traffic participant from the previous frame to the current frame is used as the reference direction to establish a local coordinate system.

3. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 1 is characterized in that: The method for generating a query containing spatiotemporal characteristics of traffic participants includes: A traffic map is constructed with each traffic participant in the local scene as a node, and the interactive information and spatial features in the traffic map that are greater than a preset impact threshold are extracted based on a multi-head sparse graph attention mechanism; the temporal features of the local scene are aggregated using a multi-head self-attention mechanism; the vectorized map information is mapped to a high-dimensional space through MLP to generate road embeddings, and based on the spatial features, temporal features and road embeddings, potential traffic rule information is extracted through a cross-attention mechanism; based on the spatial features, temporal features, interactive information, potential traffic rule information and traffic participant position encoding, queries containing the spatiotemporal features of traffic participants are generated in parallel.

4. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 3 is characterized in that: The method for acquiring the interactive information and spatial features includes: Map the historical position information of the central participant and surrounding participants in the local scene to the high-dimensional space to generate position embedding and relative position embedding; Calculate the attention coefficient between nodes in the graph based on the position embedding and the relative position embedding; The characteristics of the central traffic participant at a certain moment are obtained through the attention coefficients between the nodes in the graph; Based on the multi-head attention mechanism, the features at a certain moment are summarized to obtain a multi-head feature vector; Residual connection and normalization are performed on the multi-head feature vectors to obtain a feature vector set containing interactive information.

5. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 4 is characterized in that: The calculation expression of the attention coefficient between nodes in the graph is: In the formula, represents the influence of traffic participant j on traffic participant i at time t, N represents the total number of nodes in the graph, α-entmax(·) represents an activation function with sparse characteristics, LeakyReLU represents a leaky rectified linear unit with a negative input slope of 0.1, a represents the weight vector of a single-layer feedforward neural network, W3 and W4 represent learnable parameter matrices, φ represents the embedding function MLP, represents the embedding of traffic participant i at time t, Represents the relative position relationship embedding of the two central participants i and j.

6. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 5 is characterized in that: The characteristic expression of the central traffic participant i at time t is: In the formula, is the feature vector containing spatial information and interaction information of traffic participant i at time t, ELU is the activation function, represents the influence of traffic participant k on traffic participant i at time t, represents the embedding of traffic participant k at time t.

7. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 3 is characterized in that: The method for aggregating the temporal features of a local scene comprises: Add a sinusoidal position marker after the feature vector set containing the interactive information to obtain the input vector; Use a learnable weight matrix to convert the input vector into query, key, and value, input the query, key, and value vectors into the self-attention module to obtain the output vector; Based on the multi-head self-attention mechanism, the output vectors are aggregated and residual connections and normalization operations are performed to obtain temporal features.

8. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 3 is characterized in that: The query expression containing the spatiotemporal characteristics of traffic participants is: Where E is the set of position embeddings generated by traffic participants through MLP, G is the vector set of spatiotemporal information and interaction information generated by the temporal attention mechanism module, L is the vector set containing hidden traffic rules and driving intentions, PE is the sinusoidal position embedding, and MLP represents the embedding function.

9. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 1, characterized in that: The method for extracting scene-level long-distance interaction features includes: mapping the relative position relationship between different local coordinate systems to a high-dimensional space through MLP to generate relative position embedding; aggregating features in different local scenes through a multi-head cross-attention mechanism to obtain scene-level long-distance interaction features.

10. The multi-target trajectory prediction method based on multi-attention mechanism according to claim 1, characterized in that: After generating the predicted trajectory, the uncertainty of the trajectory is fitted by the Laplace distribution.

Images