Trajectory prediction method and device based on attention mechanism fusion multi-modal information

Abstract

The invention relates to a trajectory prediction method and device based on attention mechanism fusion multi-modal information, belongs to the technical field of automatic driving, and solves the problem of large trajectory prediction error in the prior art. The method comprises the following specific steps: preprocessing traffic scene video data to obtain a to-be-predicted target at different moments, trajectory data of other traffic participants and lane center line data; coding interaction information between the to-be-predicted target and other traffic participants based on embedded features obtained by coding the trajectory data and the lane center line data to obtain traffic participant interaction features; coding interaction information between the to-be-predicted target and the lane to obtain lane interaction features; utilizing a cross attention module and a time self-attention module to fuse lane interaction features and traffic participant interaction features; and on the basis of the obtained spatial-temporal characteristics of the to-be-predicted target, trajectory prediction is performed on the to-be-predicted target by using the optimized trajectory prediction network, so that the trajectory prediction precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a trajectory prediction method and device based on an attention mechanism fusing multimodal information. Background Art

[0002] Trajectory prediction, a core module of autonomous ground-based unmanned platforms, predicts future driving trajectories by leveraging structured and unstructured road scene information and historical data. Its quality directly impacts the safety and reliability of autonomous ground-based unmanned platforms and is crucial for improving safety, efficiency, and decision-making capabilities. Currently, deep learning methods are widely considered by the industry to have the potential to achieve optimal trajectory prediction due to their exceptional performance in aggregating multimodal data features.

[0003] Current research results often employ an encoder-decoder paradigm, encoding environmental and historical information in traffic scenes through networks such as long short-term memory (LSTM) or attention (Transformer) networks, aggregating spatiotemporal features, and then using these high-dimensional features to generate predicted trajectories. While these methods have achieved success in trajectory prediction tasks, several challenges remain. First, existing technologies generally underutilize and incompletely consider scene prior information, typically aggregating only scene interactions for a single or short time step. Furthermore, methods that aggregate scene information based on attention mechanisms often aggregate excessively redundant information, often caused by minor interactions within the scene. This information not only slows down model inference but also introduces additional errors. Finally, the decoding stage suffers from insufficient utilization of data features, resulting in large errors and poor rationality in the generated results. Summary of the Invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a trajectory prediction method based on the attention mechanism and the fusion of multimodal information to solve the problem of large trajectory prediction error in the prior art.

[0005] The purpose of the present invention is mainly achieved through the following technical solutions:

[0006] On the one hand, an embodiment of the present invention provides a trajectory prediction method based on the attention mechanism to fuse multimodal information, comprising the following steps:

[0007] Preprocess the traffic scene video data to obtain the trajectory data of the target to be predicted, other traffic participants, and the lane centerline data of the target to be predicted at different times;

[0008] Encoding the trajectory data and lane centerline data to obtain embedded features;

[0009] Based on the embedded features, the interaction information between the target to be predicted and other traffic participants is encoded to obtain traffic participant interaction features; the interaction information between the target to be predicted and the lane is encoded to obtain lane interaction features;

[0010] The lane interaction features and traffic participant interaction features are fused using a cross-attention module and a temporal self-attention module to obtain the spatiotemporal features of the target to be predicted;

[0011] Based on the spatiotemporal characteristics of the target to be predicted, the trajectory of the target to be predicted is predicted using the optimized trajectory prediction network.

[0012] Furthermore, the embedded features include position embedding and speed embedding of the target to be predicted, position embedding and speed embedding of other traffic participants, lane position embedding, relative position embedding of the target to be predicted and other traffic participants, relative position embedding between the target to be predicted and the lane, and relative speed embedding of the target to be predicted and other traffic participants.

[0013] Furthermore, obtaining the traffic participant interaction features includes:

[0014] The positions of the target to be predicted at different times and the positions of other traffic participants at the corresponding times are used as nodes, and the relative speed relationship and relative position relationship between the two are used as edges to form a traffic participant interaction graph;

[0015] Calculating the position attention coefficient and speed attention coefficient between nodes in the traffic participant interaction graph based on the position embedding and speed embedding of the target to be predicted, the position embedding and speed embedding of other traffic participants, and the relative position and relative speed embedding of the target to be predicted and other traffic participants;

[0016] Based on the position attention coefficient and the speed attention coefficient, updating the characteristics of the interaction between the target to be predicted and other traffic participants at different times;

[0017] The multi-head attention mechanism is used to aggregate features at different times in multiple subspaces, and the interaction features of traffic participants are obtained after residual connection and normalization.

[0018] Furthermore, the position attention coefficient and speed attention coefficient between nodes in the traffic participant interaction graph are calculated based on the following formulas:

[0019]

[0020] Among them, β i,j and γ i,jThey represent the position attention coefficient and speed attention coefficient of other traffic participants j to the predicted target i at time t; π() is the activation function; LeakyReLU represents the leakage rectified linear unit; a2, a3 represent the weight vectors of the single-layer feedforward neural network, W2, W3 represent the learnable parameter matrix, φ represents the embedding function; N represents the total number of nodes in the traffic participant interaction graph; represents the position embedding of the target i to be predicted at time t, represents the position embedding of other traffic participants j at time t, represents the relative position embedding of the target i to be predicted and other traffic participants j, represents the velocity embedding of the target i to be predicted at time t, represents the speed embedding of other traffic participants j at time t, Represents the relative speed embedding between the target i to be predicted and other traffic participants j.

[0021] Furthermore, a multi-head sparse graph attention mechanism is used to encode the spatial interaction information between the predicted target and the lanes, including:

[0022] The lane interaction graph is constructed by using the position of the target to be predicted at different times and the lane segment closest to the target to be predicted at the corresponding time as nodes and the spatial relationship between the two as edges;

[0023] Calculating lane attention coefficients between nodes in the lane interaction graph based on the target position embedding, the lane position embedding, and the relative position embedding between the target position and the lane;

[0024] Based on the lane attention coefficient, updating the characteristics of the interaction between the target to be predicted and the lane position at different times;

[0025] The multi-head attention mechanism is used to aggregate features at different times in multiple subspaces. After residual connection and normalization, the lane interaction features are obtained.

[0026] Furthermore, the optimized trajectory prediction network includes a decoding query module, a trajectory reference point prediction module, and a trajectory prediction module; using the optimized trajectory prediction network to perform trajectory prediction on the target to be predicted includes:

[0027] Based on the spatiotemporal features of the target to be predicted, the decoding query module generates decoding queries in parallel using a non-autoregressive method;

[0028] Based on the decoding query, using a trajectory reference point prediction module to estimate the trajectory reference point of the target point to be measured in the future time step;

[0029] Based on the trajectory reference point and the spatiotemporal characteristics of the target to be predicted, a trajectory prediction module is used to predict the trajectory of the target to be predicted.

[0030] Furthermore, after the trajectory of the target to be predicted is obtained, the method further includes optimizing the trajectory using a constraint loss function to obtain a final trajectory; wherein the constraint loss function L tr Expressed as:

[0031]

[0032] L tr =L off +L ang ,

[0033] Among them, L off is the offset constraint function of the space, L ang is the angle constraint function, N′ is the total number of targets to be predicted, H′ is the number of future time steps to be predicted, ‖‖2 is the mean square error loss function, is the two-dimensional coordinate information of the predicted trajectory, is the coordinate information of the real trajectory,

[0034] and are the heading angle of the predicted trajectory and the actual heading angle, respectively.

[0035] Furthermore, obtaining the spatiotemporal characteristics of the target to be predicted includes:

[0036] Aggregating the lane interaction features and the traffic participant interaction features based on the cross-attention module to obtain a temporal feature;

[0037] The time feature is spliced ​​with the position mark to obtain an input vector;

[0038] Based on the input vector, the temporal self-attention module is used for fusion to obtain the spatiotemporal features of the target to be predicted.

[0039] Furthermore, the traffic scene video data is preprocessed, including:

[0040] The lane centerline data of the target to be predicted in the traffic scene and the trajectory data of all traffic participants are discretized into vector information, where the trajectory vector information includes the position, speed, heading angle and acceleration of the traffic participant;

[0041] A local coordinate system is established with the position of the target to be predicted as the origin, and the vector information of the lane centerline data and trajectory data are normalized respectively to obtain the trajectory data of the target to be predicted, other traffic participants and the lane centerline data of the target to be predicted at different times.

[0042] On the other hand, an embodiment of the present invention provides a trajectory prediction device based on the attention mechanism to fuse multimodal information, including:

[0043] The acquisition module is used to pre-process the traffic scene video data to obtain the trajectory data of the target to be predicted, other traffic participants and lane centerline data at different times;

[0044] a feature extraction module for encoding the trajectory data and lane centerline data to obtain embedded features; and further for encoding, based on the embedded features, interaction information between the target to be predicted and other traffic participants to obtain traffic participant interaction features; and encoding interaction information between the target to be predicted and the lane to obtain lane interaction features;

[0045] A feature fusion module is used to fuse the lane interaction features and the traffic participant interaction features using a cross-attention module and a temporal self-attention module to obtain the spatiotemporal features of the target to be predicted;

[0046] The prediction module is used to predict the trajectory of the target to be predicted using the optimized trajectory prediction network based on the spatiotemporal characteristics of the target to be predicted.

[0047] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0048] 1. The present invention proposes encoding the trajectory data of the target to be predicted, other traffic participants, and the lane centerline data of the target to be predicted under different traffic scenarios at different times. Based on the obtained embedded features, feature extraction is performed to obtain traffic participant interaction features and lane interaction features. The two features are then fused and the final trajectory of the target to be predicted is predicted using a trajectory prediction network, thereby improving the accuracy of trajectory prediction.

[0049] 2. A multi-stage prediction trajectory generation method is proposed. By designing a multi-stage decoder, the trajectory reference points of the target to be measured are estimated in advance. Then, a multimodal trajectory is generated based on the trajectory reference points, historical interaction relationships, and spatial information. Spatial and posture constraints are imposed on the generated predictions to improve the accuracy and rationality of the predicted trajectory.

[0050] 3. Use the sparse graph attention mechanism to aggregate the interaction between the target to be predicted and the lanes in the traffic scene, and extract the spatial interaction relationship between the target to be predicted and the lanes in multiple time steps. This can more efficiently utilize the information in the scene. The extracted interaction features are fused using the cross-attention module and the temporal self-attention module to obtain the spatiotemporal characteristics of the target to be predicted.

[0051] 4. The sparse graph attention mechanism is combined with multimodal information to aggregate the interaction between the target to be predicted and other traffic participants, and speed information and posture information are introduced to improve the utilization of multimodal information in traffic scenes.

[0052] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0054] Figure 1 This is a flowchart of a trajectory prediction method based on the attention mechanism fusing multimodal information according to an embodiment of the present invention;

[0055] Figure 2 This is a logical block diagram of a trajectory prediction method based on an attention mechanism fusing multimodal information according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the coding network for predicting the interaction relationship between targets and lanes according to an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of a coding network for the interaction relationship between a prediction target and other traffic participants according to an embodiment of the present invention;

[0058] Figure 5 Schematic diagram of an interactive information aggregation network based on a cross-attention mechanism according to an embodiment of the present invention;

[0059] Figure 6 Schematic diagram of a time feature coding network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0061] Example 1

[0062] A specific embodiment of the present invention discloses a trajectory prediction method based on the attention mechanism to fuse multimodal information, such as Figure 1 As shown, the following steps are included:

[0063] Step S1: pre-processing the traffic scene video data to obtain trajectory data of the target to be predicted, other traffic participants, and lane centerline data of the target to be predicted at different times;

[0064] Step S2: Encode the trajectory data and lane centerline data to obtain embedded features;

[0065] Step S3: Based on the embedded features, encoding the interaction information between the target to be predicted and other traffic participants to obtain traffic participant interaction features; encoding the interaction information between the target to be predicted and the lane to obtain lane interaction features;

[0066] Step S4: using a cross-attention module and a temporal self-attention module to fuse the lane interaction features and the traffic participant interaction features to obtain the spatiotemporal features of the target to be predicted;

[0067] Step S5: Based on the spatiotemporal characteristics of the target to be predicted, the trajectory of the target to be predicted is predicted using the optimized trajectory prediction network.

[0068] Through the above method, the trajectory data of the target to be predicted, other traffic participants, and the lane centerline data of the target to be predicted in traffic scenarios at different times are encoded and represented. Based on the obtained embedded features, feature extraction is performed to obtain traffic participant interaction features and lane interaction features. The two features are then fused and the final trajectory of the target to be predicted is predicted using a trajectory prediction network, avoiding large errors in trajectory prediction.

[0069] Specifically, in step S1, the traffic scene includes traffic participant and lane information. Traffic participants include the target to be predicted and other traffic participants in the scene other than the target to be predicted (hereinafter referred to as other traffic participants). Vectorization and normalization preprocessing are performed on the traffic scene video data to obtain trajectory data and lane centerline data at different times after preprocessing, specifically including:

[0070] S11. Discretize the lane centerline data of the target to be predicted in the traffic scene and the trajectory data of all traffic participants into vector information, expressed as:

[0071]

[0072] Among them, L i The vector information of the lane centerline where the target i is located or closest to the target i is located (referred to as the lane centerline vector information of the target i), which contains multiple discretized road segment information; is the discretized road section information of the lane centerline at time t in the global coordinate system, and and a t are the starting coordinates, end coordinates and lane attribute information of the lane segment at time t, respectively. Lane attribute information includes road type, such as straight lane, left turn lane and right turn lane; P xis the trajectory vector information of traffic participant x (i.e., the target to be predicted and other traffic participants), including the state information of traffic participants at different times; is the state information of the traffic participant at the current time t in the global coordinate system, is the predicted state of the future time step T, and the state information of traffic participants includes position, speed, heading angle and attributes, that is, is the position of the traffic participant in the global coordinate system, v is the speed of the traffic participant, θ is the heading angle of the traffic participant, and a is the attribute information of the traffic participant, which includes the type of traffic participant, such as car, truck, bus, etc.

[0073] S12. After vectorizing the information, a local coordinate system is established with the position of the target to be predicted as the origin and the forward direction and the axis perpendicular to the forward direction as the coordinate axis in the motion plane. The data in the world coordinate system (global coordinate system) is converted to the local coordinate system for normalization. The obtained normalized lane centerline data and trajectory data are expressed as follows:

[0074]

[0075] Among them, L i ′ is the lane centerline data of the target i to be predicted after normalization; P x ′ is the normalized trajectory data of traffic participants; and where M is the discretized lane centerline section information and traffic participant status information at time t in the normalized local coordinate system I. M is the coordinate transformation matrix, which transforms the scene information (traffic participant position information, lane information, etc.) from the global coordinate system to the local coordinate system centered on the target to be predicted at time t, and converts the absolute position information into relative position information, the angle of which is determined by the angle of the line connecting the positions at time t-1 to time t (i.e., the heading angle of the target at time t).

[0076] Specifically, in step S2, the preprocessed lane information trajectory information is mapped to a high-dimensional space through a multilayer perceptron (MLP) to generate embedded features for use as subsequent network input.

[0077] For example, let's take vehicles as an example of traffic participants, the target to be predicted is the target vehicle, and other traffic participants are other vehicles. The embedding features include the position embedding and speed embedding of the target to be predicted in the scene, the position embedding and speed embedding of other traffic participants in the scene except the target to be predicted, the lane position embedding, the relative position embedding of the target to be predicted and other traffic participants, the relative position embedding between the target to be predicted and the lane, and the relative speed embedding of the target to be predicted and other traffic participants, which are expressed as:

[0078]

[0079] Where φ represents the embedding function MLP; is the position embedding of the trajectory of target i to be predicted at time t, is the position embedding of the trajectory of other traffic participant j at time t, is the relative position embedding between the target i and other traffic participants j, is the lane position embedding of the road centerline L closest to the target to be predicted at time t, is the relative position embedding of the target i to be predicted and the road centerline L at time t; represents the velocity embedding of the target i to be predicted at time t; represents the velocity embedding of other traffic participant j at time t; Represents the relative speed embedding between the target i to be predicted and other traffic participants j.

[0080] Specifically, in step S3, the graph attention mechanism is used to encode the interaction information between traffic participants and the interaction information between the target to be predicted and the lane in the traffic scene, and the corresponding interaction features are generated in parallel. The specific steps include:

[0081] S31. Encode using the lane attention module to obtain lane interaction features;

[0082] For example, Figure 3 As shown in the figure, the spatial interaction relationship between the predicted target and the lane is encoded and aggregated based on the multi-head sparse graph attention mechanism to obtain lane interaction features, including:

[0083] S311, constructing a lane interaction graph using the positions of the target to be predicted at different times and the lane segments closest to the target to be predicted at the corresponding times as nodes and their spatial relationships as edges;

[0084] S312: Calculate the lane attention coefficients between nodes in the lane interaction graph based on the position embedding of the target to be predicted, the lane position embedding, and the relative position embedding between the target to be predicted and the lane, expressed as:

[0085]

[0086] Where, α i,L represents the lane attention coefficient (i.e., influence) of lane L on the predicted target i at time t, N represents the total number of nodes in the lane interaction graph, π() = α - entmax(), represents an activation function with sparse characteristics, LeakyReLU represents a leaky rectified linear unit with a negative input slope of 0.1, a1 represents the weight vector of a single-layer feedforward neural network, W1 represents the learnable parameter matrix, and φ represents the embedding function MLP.

[0087] S313. Based on the lane attention coefficients between the nodes in the lane interaction graph, update the features of the interaction between the target i to be predicted and the lane position at time t. The expression of the updated features is:

[0088]

[0089] in, is the feature vector of the interaction relationship between the target i to be predicted and the lane at time t, ELU is the activation function, and W represents the learnable parameter matrix.

[0090] S314, using the multi-head attention mechanism, summarize the feature vectors of the interaction relationship between lanes at a certain moment in multiple subspaces to obtain a multi-head feature vector Expressed as:

[0091]

[0092] Among them, φ is the multi-layer perceptron MLP, is the feature vector of the interaction relationship between the kth head and the lane, and H is the total number of heads in the attention mechanism.

[0093] S315: Perform residual connection and normalization on the summarized multi-head feature vectors to obtain a feature vector set containing spatial interaction information between the target to be predicted and the lanes. i , that is, the lane interaction feature, is expressed as:

[0094]

[0095] S32, using the traffic participant attention module to perform encoding to obtain traffic participant interaction features;

[0096] For example, Figure 4 As shown in Figure 2, the interaction relationship between the predicted target and other traffic participants is encoded and aggregated based on the multi-head sparse graph attention mechanism, including:

[0097] S321, constructing a traffic participant interaction graph with the target position to be predicted and the positions of other traffic participants as nodes and the relative speed relationship and relative positions of the two as edges.

[0098] S322, based on the position embedding and speed embedding of the predicted target, the position embedding and speed embedding of other traffic participants, and the relative position and relative speed embedding between the target to be predicted and other traffic participants, respectively calculate the position attention coefficient and speed attention coefficient between the nodes in the graph, expressed as:

[0099]

[0100] Where, β i,j and γ i,j They represent the position attention coefficient and speed attention coefficient of other traffic participants j to the predicted target i at time t, respectively. N represents the total number of nodes in the traffic participant interaction graph. a2 and a3 represent the weight vectors of a single-layer feedforward neural network, respectively. W2 and W3 represent the learnable parameter matrices, respectively.

[0101] S323. Based on the position attention coefficients and speed attention coefficients between the nodes in the traffic participant interaction graph, update the features of the interaction between the target i to be predicted and other traffic participants at time t. The expression of the updated features is:

[0102]

[0103] in, is the characteristic vector of the interaction relationship between the target i to be predicted and other traffic participants at time t, W ′ represents the learnable parameter matrix.

[0104] S324, using the multi-head attention mechanism, summarize the feature vectors of the interaction relationship between a certain moment and other traffic participants in multiple subspaces to obtain the multi-head feature vector After residual connection and normalization, the feature vector set h containing the interaction information between the target to be predicted and other traffic participants is obtained i , that is, the interaction characteristics of traffic participants, expressed as:

[0105]

[0106] in, is the feature vector of the k-th head’s interaction with other traffic participants.

[0107] In this way, the relative position and relative speed embedding features are used to introduce the spatial interaction information of traffic participants, thereby improving the utilization of multimodal information in traffic scenes.

[0108] Specifically, in step S4, fusing the lane interaction features and the traffic participant interaction features includes:

[0109] S41. Based on the cross-attention mechanism, the interaction between the predicted target and the lane, and the interaction between the predicted target and surrounding traffic participants are summarized to obtain temporal features.

[0110] For example, Figure 5 As shown, the interactive relationship features between the predicted target and surrounding traffic participants are used as query Q G =h i W Q , using the predicted interaction relationship between the target and the lane as the key K G=[h i ,o i ]W K Sum V G =[h i ,o i ]W V , summarizing two interactive relationships through the multi-head cross attention mechanism: Finally, the feature vector of the interaction relationship between the predicted target and the traffic scene at different times is obtained, that is, the time feature.

[0111] S42, based on the temporal self-attention mechanism, aggregate the temporal features to obtain the spatiotemporal features of the target to be predicted, such as Figure 6 As shown, specifically including:

[0112] S421, concatenate the time feature vector set containing the interactive information and add a sine position marker to obtain an input vector;

[0113] For example, the interaction relationship feature vector g at a certain moment i A sinusoidal position marker is added to form the input vector s of the temporal self-attention module i , expressed as:

[0114] s i =g i +t i ,

[0115]

[0116] Among them, t i is the position embedding.

[0117] S422, projecting the input vector into query, key, and value using a learnable parameter matrix;

[0118] For example, the query, key, and value are calculated based on the following formula:

[0119]

[0120] in, are the query vector, key vector, and value vector corresponding to the target i to be predicted; W Q , W K , W V are the learnable weight matrices respectively.

[0121] S423: Input the generated query vector, key vector, and value vector into the temporal self-attention module for fusion and aggregation to obtain a fused feature vector, which is expressed as:

[0122]

[0123] in, is the fused feature vector; M is the mask, which makes the model pay more attention to the information from the previous time; d k is the dimension of the key vector.

[0124] Summarize multiple information and obtain spatiotemporal features z through residual connection and normalization operations i , expressed as:

[0125]

[0126] in, is the fused feature vector of the k-th head.

[0127] Through the above method, the extracted lane spatial features and traffic participant interaction features are fused to obtain the time domain features of the predicted target trajectory, and the spatial interaction relationship between the predicted target and the traffic scene can be extracted in multiple time steps, which can make more efficient use of the information in the scene.

[0128] Specifically, in step S5, the trajectory of the target to be predicted is predicted using the optimized trajectory prediction network. The optimized trajectory prediction network includes a decoding query module, a trajectory reference point prediction module, and a trajectory prediction module. The specific steps include:

[0129] S51. Based on the spatiotemporal features of the target to be predicted, a non-autoregressive method is used to generate decoding queries in parallel to obtain a query set, which is expressed as:

[0130] Q=[MLP(Z)×MLP(E)×MLP(PE)]+n,

[0131] Where Z is the context feature containing spatiotemporal features generated by the temporal encoder, and Z = {z i}; E is the reason The trajectory embedding of the target to be predicted is generated by MLP; PE is the sinusoidal position embedding, and n is the introduced noise that follows a Gaussian distribution.

[0132] By combining the parallel decoding of non-autoregressive models with the feature processing capabilities of MLP, we can efficiently generate decoding queries while capturing spatiotemporal dependencies. Compared with the autoregressive decoding method that generates decoding queries step by step, the non-autoregressive query generation method generates decoding queries in parallel, reducing cumulative errors and improving decoding efficiency.

[0133] S52, using a trajectory reference point prediction module to estimate the trajectory reference points of the target to be predicted in the future time steps based on the query set;

[0134] For example, a three-layer MLP network is used to receive the query set and generate trajectory reference points, which can be expressed as:

[0135]

[0136] in, is the reference point of the predicted trajectory of the target i; q i is the decoding query of the target i in the query set Q.

[0137] During the training phase, the loss function of the trajectory reference point prediction module is composed of the classification loss function and the regression loss function, which can be expressed as:

[0138]

[0139] in, is the classification loss function of the trajectory reference point prediction module; is the regression loss function of the trajectory reference point prediction module; d() is the Smooth L1 loss function (Huber loss function), N′ is the total number of targets to be predicted, and H′ is the number of predicted future time steps.

[0140] S53, combining the trajectory reference point and the context features including the spatiotemporal features, and using the trajectory prediction module to predict the trajectory of the target to be predicted in the future time steps;

[0141] For example, a new query is generated by a cross-attention module using the reference point embedding as a query and a query containing the spatiotemporal features of the predicted target as a key and value; the generated new query is decoded by MLP to generate a predicted trajectory; the predicted trajectory is fitted with the uncertainty of the prediction estimate using the Laplace distribution, that is, it is fitted into a mixture model that conforms to the Laplace distribution to obtain the predicted trajectory Expressed as:

[0142]

[0143] Among them, concat() is a cross-attention module; t is the trajectory tensor generated by the decoder, whose shape is [K, N′, H′, 4], K is the number of output trajectories of the predicted target, N′ is the total number of targets to be predicted, H′ is the number of predicted future time steps, and the trajectory information is represented by the predicted position coordinates [x, y] and their associated uncertainties μ and b, where μ and b represent the position and scale parameters of each Laplace component, respectively.

[0144] To help with training, two side-by-side MLPs are used to predict μ and b. The trajectory prediction loss function consists of a classification loss function and a regression loss function:

[0145]

[0146] in, is the classification loss function for trajectory prediction; is the regression loss function for trajectory prediction.

[0147] Furthermore, a constrained loss function is used for model training to optimize the predicted trajectory using the trained model.

[0148] Exemplarily, the constraint loss function includes spatial constraints and posture constraints, and the constraint loss function is established based on the following formula:

[0149]

[0150] L tr =L off +L ang ,

[0151] Among them, L off is the offset constraint function of the space, that is, the space constraint; L ang is the angle constraint function, i.e., the posture constraint; ‖‖2 is the mean square error loss function (L2 least squares loss function), is the two-dimensional coordinate information of the predicted trajectory, is the coordinate information of the real trajectory, and They are the heading angle of the predicted trajectory and the actual heading angle information respectively.

[0152] Compared with the existing technology, this embodiment provides a trajectory prediction method based on the attention mechanism to fuse multimodal information, such as Figure 2 As shown, by encoding the trajectory data of the target to be predicted and other traffic participants in the preprocessed traffic scene, and the lane centerline data of the target to be predicted, the interaction features between the target to be predicted and the lane centerline and the interaction features between the target to be predicted and other traffic participants are extracted respectively, and then the extracted interaction features are fused using the cross attention and temporal attention modules to obtain the spatiotemporal features of the target to be predicted, obtain the interaction features in more time steps, improve the utilization rate of multimodal information in the traffic scene, use the multi-stage decoder to estimate the trajectory reference point of the target to be predicted, and then predict the multimodal trajectory information, impose spatial constraints and posture constraints on the generated prediction, and improve the accuracy and rationality of the predicted trajectory.

[0153] Example 2

[0154] Another specific embodiment of the present invention discloses a trajectory prediction device based on an attention mechanism fusing multimodal information, comprising:

[0155] The acquisition module is used to pre-process the traffic scene video data to obtain the trajectory data of the target to be predicted, other traffic participants and the lane centerline data of the target to be predicted at different times;

[0156] a feature extraction module for encoding the trajectory data and lane centerline data to obtain embedded features; and further for encoding, based on the embedded features, interaction information between the target to be predicted and other traffic participants to obtain traffic participant interaction features; and encoding interaction information between the target to be predicted and the lane to obtain lane interaction features;

[0157] A feature fusion module is used to fuse the lane interaction features and the traffic participant interaction features using a cross-attention module and a temporal self-attention module to obtain the spatiotemporal features of the target to be predicted;

[0158] The prediction module is used to predict the trajectory of the target to be predicted using the optimized trajectory prediction network based on the spatiotemporal characteristics of the target to be predicted.

[0159] The system can perform trajectory prediction according to the method described in any one of the solutions in Example 1. The relevant parts are referenced from each other and are not described repeatedly in this embodiment.

[0160] Compared with the existing technology, this embodiment provides a trajectory prediction device based on the attention mechanism to fuse multimodal information, which integrates an acquisition module, a feature extraction module, a feature fusion module and a prediction module to predict the trajectory of the target to be predicted in the preprocessed traffic scene video data, thereby improving the accuracy of the device's trajectory prediction.

[0161] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0162] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A trajectory prediction method based on attention mechanism fusing multimodal information, characterized in that: The steps include: Preprocess the traffic scene video data to obtain the trajectory data of the target to be predicted, other traffic participants, and the lane centerline data of the target to be predicted at different times; Encoding the trajectory data and lane centerline data to obtain embedded features; Based on the embedded features, the interaction information between the target to be predicted and other traffic participants is encoded to obtain traffic participant interaction features; the interaction information between the target to be predicted and the lane is encoded to obtain lane interaction features; The lane interaction features and traffic participant interaction features are fused using a cross-attention module and a temporal self-attention module to obtain the spatiotemporal features of the target to be predicted; Based on the spatiotemporal characteristics of the target to be predicted, the trajectory of the target to be predicted is predicted using the optimized trajectory prediction network.

2. A trajectory prediction method based on attention mechanism fusion of multimodal information according to claim 1, characterized in that: The embedding features include the position embedding and speed embedding of the target to be predicted, the position embedding and speed embedding of other traffic participants, the lane position embedding, the relative position embedding of the target to be predicted and other traffic participants, the relative position embedding between the target to be predicted and the lane, and the relative speed embedding of the target to be predicted and other traffic participants.

3. The trajectory prediction method based on attention mechanism fusion of multimodal information according to claim 2 is characterized in that: Obtaining the traffic participant interaction features includes: The positions of the target to be predicted at different times and the positions of other traffic participants at the corresponding times are used as nodes, and the relative speed relationship and relative position relationship between the two are used as edges to form a traffic participant interaction graph; Calculating the position attention coefficient and speed attention coefficient between nodes in the traffic participant interaction graph based on the position embedding and speed embedding of the target to be predicted, the position embedding and speed embedding of other traffic participants, and the relative position and relative speed embedding of the target to be predicted and other traffic participants; Based on the position attention coefficient and the speed attention coefficient, updating the characteristics of the interaction between the target to be predicted and other traffic participants at different times; The multi-head attention mechanism is used to aggregate features at different times in multiple subspaces, and the interaction features of traffic participants are obtained after residual connection and normalization.

4. The trajectory prediction method based on attention mechanism fusion of multimodal information according to claim 3 is characterized in that: The position attention coefficient and speed attention coefficient between nodes in the traffic participant interaction graph are calculated based on the following formulas: Among them, β i,j and γ i,j They represent the position attention coefficient and speed attention coefficient of other traffic participants j to the predicted target i at time t; π() is the activation function; LeakyReLU represents the leakage rectified linear unit; a2, a3 represent the weight vectors of the single-layer feedforward neural network, W2, W3 represent the learnable parameter matrix, φ represents the embedding function; N represents the total number of nodes in the traffic participant interaction graph; represents the position embedding of the target i to be predicted at time t, represents the position embedding of other traffic participants j at time t, represents the relative position embedding of the target i to be predicted and other traffic participants j, represents the velocity embedding of the target i to be predicted at time t, represents the speed embedding of other traffic participants j at time t, Represents the relative speed embedding between the target i to be predicted and other traffic participants j.

5. The trajectory prediction method based on attention mechanism fusion of multimodal information according to claim 2 is characterized in that: A multi-head sparse graph attention mechanism is used to encode the spatial interaction information between the predicted target and the lanes, including: The lane interaction graph is constructed by using the position of the target to be predicted at different times and the lane segment closest to the target to be predicted at the corresponding time as nodes and the spatial relationship between the two as edges; Calculating lane attention coefficients between nodes in the lane interaction graph based on the target position embedding, the lane position embedding, and the relative position embedding between the target position and the lane; Based on the lane attention coefficient, updating the characteristics of the interaction between the target to be predicted and the lane position at different times; The multi-head attention mechanism is used to aggregate features at different times in multiple subspaces. After residual connection and normalization, the lane interaction features are obtained.

6. The trajectory prediction method based on attention mechanism fusion of multimodal information according to claim 1 is characterized in that: The optimized trajectory prediction network includes a decoding query module, a trajectory reference point prediction module, and a trajectory prediction module. Using the optimized trajectory prediction network to perform trajectory prediction on the target to be predicted includes: Based on the spatiotemporal features of the target to be predicted, the decoding query module generates decoding queries in parallel using a non-autoregressive method; Based on the decoding query, using a trajectory reference point prediction module to estimate the trajectory reference point of the target to be predicted in the future time step; Based on the trajectory reference point and the spatiotemporal characteristics of the target to be predicted, a trajectory prediction module is used to predict the trajectory of the target to be predicted.

7. The trajectory prediction method based on attention mechanism fusion of multimodal information according to claim 6 is characterized in that: After the trajectory of the target to be predicted is obtained, the method further includes optimizing the trajectory using a constraint loss function to obtain the final trajectory; wherein the constraint loss function L tr Expressed as: L tr =L off +L ang , Among them, L off is the offset constraint function of the space, L ang is the angle constraint function, N′ is the total number of targets to be predicted, H′ is the number of future time steps to be predicted, ‖‖2 is the mean square error loss function, is the two-dimensional coordinate information of the predicted trajectory, is the coordinate information of the real trajectory, and are the heading angle of the predicted trajectory and the actual heading angle, respectively.

8. A trajectory prediction method based on attention mechanism fusion of multimodal information according to any one of claims 1 to 7, characterized in that: Obtaining the spatiotemporal characteristics of the target to be predicted includes: Aggregating the lane interaction features and the traffic participant interaction features based on the cross-attention module to obtain a temporal feature; The time feature is spliced ​​with the position mark to obtain an input vector; Based on the input vector, the temporal self-attention module is used for fusion to obtain the spatiotemporal features of the target to be predicted.

9. The trajectory prediction method based on attention mechanism fusion of multimodal information according to claim 1 is characterized in that: Preprocess traffic scene video data, including: The lane centerline data of the target to be predicted in the traffic scene and the trajectory data of all traffic participants are discretized into vector information, where the trajectory vector information includes the position, speed, heading angle and acceleration of the traffic participant; A local coordinate system is established with the position of the target to be predicted as the origin, and the vector information of the lane centerline data and trajectory data are normalized respectively to obtain the trajectory data of the target to be predicted, other traffic participants and the lane centerline data of the target to be predicted at different times.

10. A trajectory prediction device based on attention mechanism fusing multimodal information, characterized in that: include: The acquisition module is used to pre-process the traffic scene video data to obtain the trajectory data of the target to be predicted, other traffic participants and the lane centerline data of the target to be predicted at different times; a feature extraction module, configured to encode the trajectory data and lane centerline data to obtain embedded features; It is also used to encode the interaction information between the target to be predicted and other traffic participants based on the embedded features to obtain traffic participant interaction features; it is also used to encode the interaction information between the target to be predicted and the lane to obtain lane interaction features; A feature fusion module is used to fuse the lane interaction features and the traffic participant interaction features using a cross-attention module and a temporal self-attention module to obtain the spatiotemporal features of the target to be predicted; The prediction module is used to predict the trajectory of the target to be predicted using the optimized trajectory prediction network based on the spatiotemporal characteristics of the target to be predicted.

Images