A method, device and medium for predicting multimodal non-autoregressive trajectory of intelligent vehicles

Abstract

The present invention discloses a method, device and medium for predicting multimodal non-autoregressive trajectories of intelligent vehicles. The method comprises the steps of obtaining driving trajectory data of a vehicle, obtaining a trajectory sequence based on the driving trajectory data, and dividing the trajectory sequence into an observation sequence and a prediction sequence; constructing and training a trajectory prediction model, inputting the data to be tested into the trained trajectory prediction model, and obtaining a trajectory prediction result. The present invention can not only improve the prediction accuracy of the vehicle trajectory, but also improve the prediction speed of the vehicle trajectory.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, and in particular relates to a method, device and medium for predicting multimodal non-autoregressive trajectories of an intelligent vehicle. Background Art

[0002] With the continuous development of smart car technology, the vision of achieving high-level autonomous driving and mass production of smart cars has attracted more and more attention from academia and industry. The safety of smart cars is related to the possibility of mass production. In order to further improve the safety of smart car travel, the key is to enable it to predict the status of other vehicles in the future based on the surrounding driving environment like a human driver, so as to help it make safer and more reasonable decisions and plans.

[0003] However, predicting the future trajectory of a vehicle in a real traffic scenario is challenging. There are three main challenges: ① Dynamic interaction challenge. The dynamic interaction between vehicles will affect the vehicle's driving trajectory, and when the number and type of traffic participants around the vehicle change dynamically, whether the interaction mode at this time can be accurately modeled directly affects the accuracy of trajectory prediction; ② Uncertainty challenge. The vehicle's driving trajectory naturally has multimodal properties. At present, vehicle driving is mainly controlled by the driver and is affected by many factors, such as driving style, driving status, driving intention, road conditions, etc. The same historical trajectory may have multiple reasonable future trajectories; ③ Accuracy and speed trade-off challenge. The demand of smart cars for trajectory prediction models is to balance prediction accuracy and prediction speed under the limited computing power of on-board chips, so the design of the model needs to reasonably balance the two.

[0004] The commonly used vehicle trajectory prediction methods are mainly data-driven methods, among which the two main types are based on recurrent neural networks (RNN) and their variants, long short-term memory networks (LSTM), gated recurrent units (GRU), and transformers. Thanks to the great success of these two types of methods in the field of natural language processing, scholars have migrated the encoder-decoder architecture of Sequence to Sequence to the field of trajectory prediction, and used the network's powerful feature extraction ability for sequence information in the encoder to model the spatiotemporal interaction in the trajectory data, and input the obtained intermediate representation into the decoder to decode and generate future trajectories. Although good prediction results have been achieved, these two types of methods still have problems such as large prediction cumulative errors (corresponding to model prediction accuracy) and low prediction real-time performance (corresponding to model inference speed).

[0005] Liu Miaomiao et al. disclosed in Chinese invention patent CN117930847A a “vehicle trajectory prediction algorithm based on an improved Transformer model”. In order to improve the prediction accuracy of the model, a processing layer that considers the spatiotemporal interaction of vehicles is added to the encoder part of the Transformer model. However, it processes the time-space attention in a stacked manner, and the decoding method is the same as the traditional Transformer. Both are autoregressive decoding architectures, and the prediction speed is slow. Summary of the invention

[0006] In order to solve the problem that the existing trajectory prediction cannot take into account both prediction accuracy and reasoning speed, the present invention proposes a multimodal non-autoregressive trajectory prediction method, device and medium for intelligent vehicles, which can not only improve the prediction accuracy of vehicle trajectories, but also improve the prediction speed of vehicle trajectories. The former is mainly reflected in the use of a time-space attention mechanism that takes into account both to mine the spatiotemporal interaction characteristics of vehicles; the latter is mainly reflected in the design of a non-autoregressive Transformer decoding architecture, which improves the prediction speed compared to the traditional Transformer autoregressive decoding architecture. In order to compensate for the loss of temporal dependency of the prediction sequence caused by the non-autoregressive decoding architecture, a loss function that considers temporal dependency is proposed at the same time, and the temporal dependency in the sequence is predicted through an explicit modeling method, which effectively improves the prediction accuracy of the vehicle trajectory.

[0007] In order to achieve the purpose of the present invention, the present invention provides a method for predicting multimodal non-autoregressive trajectories of intelligent vehicles considering temporal dependency decoding, comprising the steps of:

[0008] Acquire the driving trajectory data of the vehicle, obtain a trajectory sequence based on the driving trajectory data, and divide the trajectory sequence into an observation sequence and a prediction sequence;

[0009] Inputting the driving trajectory data of the vehicle into the trajectory prediction model to obtain the trajectory prediction result, the trajectory prediction model includes an embedding coding module of the observation sequence, a feature extraction encoder based on spatiotemporal attention, a fully connected layer, an embedding coding module of the prediction sequence, a decoder query vector generation module and a decoder, the embedding coding module of the observation sequence is used to embed the observation sequence into a vector to obtain a high-dimensional representation of the observation sequence and positionally encode the high-dimensional representation of the observation sequence to obtain a position encoding result of the observation sequence, the feature extraction encoder based on spatiotemporal attention includes a joint consideration of spatiotemporal attention weight calculation module, which is used to encode the spatiotemporal interaction features of the vehicles in the traffic scene and output the spatiotemporal attention, the fully connected layer is used to map the spatiotemporal attention to a hidden state vector, the embedding coding module of the prediction sequence is used to embed the prediction sequence into a vector to obtain a high-dimensional representation of the prediction sequence and positionally encode the high-dimensional representation of the prediction sequence to obtain a position encoding result of the prediction sequence, the decoder query vector generation module is used to consider the influence of the observation sequence, the prediction sequence and the spatiotemporal interaction, and generate a decoder query vector with the same length as the prediction sequence in a non-autoregressive decoding method, and the decoder is used to obtain a future trajectory of the length of the prediction sequence;

[0010] Among them, the loss function used when training the trajectory prediction model includes a temporal dependency loss, which is used to calculate a coarse-grained temporal dependency loss and a fine-grained temporal dependency loss between the output future trajectory and the true label.

[0011] Furthermore, the method of obtaining the vehicle's driving trajectory data includes: using a publicly available vehicle trajectory open source data set or determining a target vehicle, and collecting the driving trajectory data of the target vehicle and its surrounding vehicles through on-board sensors.

[0012] The vehicle's driving trajectory data is obtained through the on-board sensor, and the environment perception modules such as target detection and target tracking are used to obtain historical trajectory coordinates, speed, acceleration, lane number, vehicle number, and vehicle type information. Generally, the perception range can be selected as 200m.

[0013] Preferably, the driving trajectory data is preprocessed and noise and outliers are removed from the acquired driving trajectory data. The noise and outliers removed include but are not limited to large error data and wild points generated by the target detection and target tracking modules, and finally the trajectory data is smoothed. The implementation method can be one or more of first-order Savitzky-Golay filtering, wavelet decomposition, and Leyda's rule.

[0014] Furthermore, a fixed-length sequence is extracted based on the trajectory sequence, and the fixed-length sequence is divided into an observation sequence and a prediction sequence; and for each fixed-length sequence, a coordinate system is constructed with the prediction object, i.e., the target vehicle, as the origin, with the orientation of the target vehicle at the end of the observation sequence as the positive direction of the vertical axis, and a 90° counterclockwise rotation as the positive direction of the horizontal axis, and the data is unified into the coordinate system through coordinate conversion for representation.

[0015] Furthermore, before coordinate transformation, the variables in the trajectory sequence data are first subjected to correlation analysis on the trajectory prediction task, the possible combinations of optimal variables are listed, the optimal variable combination is screened, and the trajectory sequence data composed of the optimal variable combination is subjected to fixed-length sequence extraction and coordinate transformation.

[0016] Among them, for the vehicle trajectory prediction task, different data sets can provide different data variables, but all include the most basic horizontal and vertical coordinates of the vehicle trajectory, in addition to horizontal and vertical speeds, horizontal and vertical accelerations, vehicle types, road information, map information, etc. In order to achieve a balance between the features provided to the model being rich enough but not redundant, the variables of the trajectory sequence data are analyzed for correlation and then optimized. The method for performing correlation analysis on the variables contained in the trajectory sequence can adopt any of the methods of principal component analysis, recursive feature elimination, and Warpper Methods, and train the model on the data subset containing these preferred variables and evaluate the performance to find the best variable combination. The Latin hypercube test optimization design method is used to optimize the combination of different variables that can be provided by the data set, and the evaluation index is used to judge the quality of the variable combination after the model is trained.

[0017] By analyzing variable sensitivity and performing optimal sequence data variables, it is helpful to achieve a balance between improving the prediction accuracy and prediction speed of the model.

[0018] After obtaining the best variable combination, coordinate transformation is performed on the trajectory sequence data composed of the best variable combination and the data set is divided.

[0019] Furthermore, in order to make more efficient use of the existing trajectory sequence data, the sliding window method is used to process the trajectory sequence, the window length is set to 8s, and the collected trajectory sequence is divided into observations of length 3s, predicted length The prediction length is 5s, and the trajectory within the prediction length is used as the true label of the prediction task. Finally, all trajectory sequences are divided into training, validation and test sets.

[0020] Furthermore, for each fixed-length sequence, a coordinate system is constructed with the prediction object, i.e., the target vehicle, as the origin. The orientation of the target vehicle at the end of the observation sequence is the positive direction of the x-axis, and the y-axis is rotated 90° counterclockwise. The data in the entire trajectory sequence is unified into this coordinate system through coordinate transformation. The specific transformation process formula is as follows:

[0021] ;

[0022] For vehicles With vehicle The xy coordinate difference between them; For vehicles The xy coordinates of For vehicles The xy coordinates of For vehicles Coordinate transformation to vehicle The rotation matrix of the coordinate system; For vehicles With vehicle The heading angle difference between For vehicles After the rotation matrix transformation, the vehicle Coordinate representation in coordinate system.

[0023] Furthermore, the feature extraction encoder based on spatiotemporal attention is used to encode the spatiotemporal interaction features of vehicles in the traffic scene. The input of the feature extraction encoder based on spatiotemporal attention is:

[0024] The observation sequence is vector-embedded according to the following formula to obtain a high-dimensional representation of the observation sequence ,in Respectively represent the length of the input sequence, the number of vehicles in each sequence, and the set model dimension:

[0025] ;

[0026] in represents the fully connected layer;

[0027] High-dimensional representation of observation sequences The position coding result is obtained after using sine and cosine position coding PE :

[0028] ;

[0029] in, Represents a specific position in the observation sequence; Indicates a specific dimension index.

[0030] Furthermore, in the feature extraction encoder based on spatiotemporal attention, the calculation process performed by the joint spatiotemporal attention weight calculation module is as follows:

[0031] Calculate the time attention weight and output the time value vector. The formula of the time attention weight is:

[0032] ;

[0033] ;

[0034] ;

[0035] In the formula, is the calculated time attention weight, is the time query vector of vehicle n, that is, the time query vector Focusing on the specific vehicle n, is the transpose operation, is the time key vector for vehicle n, is a high-dimensional representation of vehicle n, is the position encoding result of vehicle n, is the time value vector of vehicle n;

[0036] Calculate the spatial attention weight, the calculation formula is:

[0037] ;

[0038] ;

[0039] In the formula, is the calculated spatial attention weight for each time step, is the spatial query vector at time step t; is the spatial key vector at time step t, is the model dimension being set;

[0040] The time influence weight matrix is ​​constructed to obtain the spatiotemporal attention weight, and the expression of the spatiotemporal attention weight is:

[0041] ;

[0042] In the formula, is the spatiotemporal attention weight, is the time influence weight matrix, is the time attention weight set The last dimension, the set of time attention weights , is the number of all vehicles in the observation sequence;

[0043] Get encoder output: For each vehicle in the observation sequence , define the final high-dimensional representation of each vehicle as , based on the set of spatial attention weights and the time impact weight matrix The final high-dimensional representation of the vehicle is , the calculation formula is as follows:

[0044] ;

[0045] In the formula, , is the linear spatial dimensional representation of the encoder output of vehicle n, is the length of the observation sequence input to the spatiotemporal attention weight calculation module, is the spatiotemporal attention of vehicle n, is the time value vector of vehicle n.

[0046] Furthermore, for each vehicle in the observation sequence , define three learnable matrices, namely the time query vector matrix , time key vector matrix and the time value vector matrix , a high-dimensional representation of the input observation sequence With position encoding results The result of the addition is multiplied by these three matrices respectively to obtain the corresponding time query vector , time key vector and the time value vector , is to treat the spatial dimension as BatchSize when calculating temporal attention, where BatchSize is the batch size. For each time step in the observation sequence , define two learnable matrices, namely the spatial query vector matrix and the spatial key vector matrix , a high-dimensional representation of the input observation sequence Multiply these two matrices respectively to get the corresponding spatial query vector , spatial key vector .

[0047] Furthermore, in the spatiotemporal attention-based feature extraction encoder, the execution process of the decoder query vector generation module includes:

[0048] For each target vehicle in the observation sequence , its hidden state , is the linear space dimension representation of the hidden state, the hidden state The calculation formula is as follows:

[0049] ;

[0050] In the formula, is a learnable matrix, is a learnable bias parameter, is a matrix The linear space dimension representation of is a learnable bias parameter Linear space dimension representation of ;

[0051] Calculate the position encoding result of the predicted sequence after sine and cosine position encoding The generated spatial influence weights , the calculation formula is:

[0052] ;

[0053] In the formula, , is the linear spatial dimension representation of the spatial influence weight, is a learnable matrix, is a learnable bias parameter, is the position encoding result of the prediction sequence of each vehicle after sine-cosine position encoding;

[0054] Define the decoder query vector as , the expression is:

[0055] ;

[0056] In the formula, , yes The linear space dimension representation of , yes The linear space dimension representation of is a high-dimensional representation of the observation sequence of each vehicle, is the length of the observation sequence, is the length of the prediction sequence.

[0057] Furthermore, the decoder includes a spatiotemporal attention weight calculation module and a cross-attention weight calculation module. The calculation process of the spatiotemporal attention weight calculation module is basically the same as the calculation process performed by the joint consideration spatiotemporal attention weight calculation module of the encoder. The difference is that the sequence in the spatiotemporal attention weight calculation module of the decoder is a prediction sequence; the input of the cross-attention weight calculation module includes two parts, namely the output of the encoder and the output of the spatiotemporal attention weight calculation module in the decoder. For the output of the encoder, matrix multiplication is performed with two learnable parameter matrices to serve as the key vector and value vector of the cross-attention weight calculation; for the output of the spatiotemporal attention weight calculation module in the decoder, matrix multiplication is performed with a learnable parameter matrix to serve as the query vector of the cross-attention weight calculation, and then cross-attention calculation is performed. The output obtained is mapped to the future trajectory of the predicted sequence length through a fully connected layer.

[0058] Furthermore, the loss function used in the training process of the trajectory prediction model is:

[0059] ;

[0060] In the formula, is the total loss, including regression loss, classification loss and temporal dependency loss, is the timing dependency loss, is the soft target cross entropy loss, The loss for the regression task;

[0061] Among them, the timing dependency loss for:

[0062] ;

[0063] ;

[0064] ;

[0065] In the formula, represents the prediction loss of the trajectory prediction model, represents the fine-grained timing dependency loss, is a weight used to reflect the coarse-grained timing dependency loss.

[0066] The temporal dependency loss is calculated between the output future trajectory and the true label. At the same time, the coarse-grained and fine-grained temporal dependency losses are distinguished, and the coarse-grained and fine-grained temporal dependency losses together constitute the final loss. The coarse-grained temporal dependency loss is defined as the difference between the predicted value and the true value at each time step in the prediction time domain, and the fine-grained temporal dependency loss is defined as the first-order difference between the predicted value and the true value at each time step in the prediction time domain.

[0067] The present invention also provides a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0068] The present invention also provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0069] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0070] 1. Model prediction accuracy

[0071] In order to improve the prediction accuracy of the trajectory prediction model, the main focus is on strengthening the extraction of spatiotemporal interaction features, avoiding cumulative errors and

[0072] Improvements are made by considering three aspects of decoding timing dependency.

[0073] (1) Strengthening spatiotemporal interaction feature extraction

[0074] In real traffic scenarios, there is often a complex time-space interaction relationship between the predicted target vehicle and other surrounding traffic participants. The previous practice is usually to use an attention mechanism to model the interaction relationship. In order to capture deeper spatiotemporal interaction features, stacked or parallel spatiotemporal attention is often used. The stacked method alternately processes the correlation between time and space, and the parallel method extracts the features of time and space separately and then aggregates the two. None of them jointly considers the interdependence of time and space. In the process of extracting spatiotemporal interaction features, time and space often interact in a bidirectional and simultaneous manner. For this reason, the present invention proposes an attention mechanism that jointly considers spatiotemporal interaction. This method can better extract spatiotemporal interaction features, enhance the model's understanding of vehicle trajectory data, and improve the model's prediction accuracy.

[0075] (2) Avoid cumulative errors

[0076] In the traditional Transformer trajectory prediction model, the decoder is based on an autoregressive decoding method, that is, in the process of predicting future trajectories, the result of each time step is recursively predicted, and the prediction of the current time step needs to rely on the prediction result of the previous time step. Such a decoding structure inevitably introduces cumulative errors, which increase with the growth of the prediction time domain, and ultimately manifests as a decrease in the prediction accuracy of the model. For this reason, the present invention proposes a decoding method based on non-autoregression, the implementation of which mainly depends on a constructed decoder query vector generation module, which is intended to generate a query vector consistent with the length of the prediction sequence and input it into the decoder, so that the decoding process is freed from the dependence on the prediction results of the previous moment, and can directly generate future trajectories, which structurally avoids the generation of cumulative errors and is conducive to improving the prediction accuracy of the model.

[0077] (3) Consider decoding timing dependencies

[0078] After using the non-autoregressive decoding method, since the future trajectory is directly output during the decoding process, the time-series dependency of the future trajectory is lost. For the vehicle trajectory sequence data that changes over time, the motion characteristics contained therein are consistent with the vehicle kinematics and cannot undergo mutations, that is, the observation value at each time step will be affected by the observation value at the previous time step, and may also show other complex nonlinear time dependencies, such as trends. For this reason, the present invention proposes an auxiliary task embedded in the training process, that is, introducing a coarse-grained and fine-grained time-series dependency loss during the model training process, using the loss to measure the relationship between the prediction results and establish the dependency between the decoding time steps, which is beneficial to improving the prediction accuracy of the model.

[0079] 2. Model prediction speed

[0080] In the traditional Transformer trajectory prediction model, due to the limitation of the autoregressive decoding structure, it is necessary to make predictions at every time step within the length of the prediction sequence, which results in a long prediction time. For this reason, the present invention proposes a decoding structure based on non-autoregressive, which only needs one step in the decoding process to predict the future trajectory. It should be noted that this structure is consistent with the above-mentioned non-autoregressive decoding structure, that is, this decoding method can improve the model prediction speed while avoiding the cumulative error.

[0081] In summary, the present invention proposes a Transformer intelligent vehicle multimodal trajectory prediction method based on non-autoregressive and time-dependent decoding, which can provide a reference for overcoming the problems of low real-time performance and large cumulative prediction error of existing trajectory prediction models. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1It is a flowchart of the steps of a method for predicting multimodal non-autoregressive trajectories of an intelligent vehicle provided in an embodiment.

[0083] Figure 2 is a flow chart of the trajectory prediction model in the embodiment.

[0084] FIG3( a ) is a schematic diagram of a fine-grained timing dependency loss calculation process in an embodiment.

[0085] FIG3( b ) is a schematic diagram of the coarse-grained timing dependency loss calculation process in an embodiment.

[0086] Figure 4 It is a schematic diagram comparing other methods in the embodiments with the method of the present invention.

[0087] Figure 5 It is a displacement error comparison diagram of the results obtained by the method of the present invention and other methods.

[0088] Figure 6 This is a comparison chart of the reasoning time consumption of the method of the present invention and other methods. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] The German highway data described by the HighD dataset is taken as an example to illustrate the effectiveness and accuracy of the trajectory prediction method proposed in this embodiment.

[0091] The multimodal non-autoregressive trajectory prediction method of an intelligent vehicle considering temporal dependency decoding of this embodiment includes: processing a public human natural driving trajectory data set; converting all data points into a coordinate system with a prediction target as the origin; decomposing the trajectory prediction task into two stages: a regression task and an auxiliary task; constructing a feature extraction encoder based on spatiotemporal attention, mapping the temporal attention influence map to the spatial attention, and realizing the interactive fusion of spatiotemporal features; passing the fused features through a decoder query vector generation module to obtain a decoder query vector with the same length as the prediction sequence; passing the fused features and the decoder query vector through a cross-attention weight calculation module to obtain a future trajectory output; using the future trajectory and the true value label to calculate the coarse and fine-grained temporal dependency loss, and using the temporal dependency loss to assist the trajectory prediction model training; using the binary Laplace negative log-likelihood loss function to train the regression task, aiming to encourage the generation of multimodal trajectories, and at the same time, for the probability of each modality, using the soft target cross entropy loss function to train this classification task.

[0092] See also Figure 1 This embodiment provides a method for predicting a multimodal non-autoregressive trajectory of an intelligent vehicle, wherein any one of the vehicles is taken as a prediction object, referred to as a target vehicle, and other vehicles around the target vehicle are collectively referred to as surrounding vehicles. The method specifically includes the following steps:

[0093] Step 1: Select a vehicle around the autonomous vehicle as the prediction subject, i.e., the target vehicle, and collect the driving trajectory data of the target vehicle and its surrounding vehicles.

[0094] Among them, the original perception information of the target vehicle and its surrounding vehicles (surrounding vehicles) is obtained through the on-board sensors of the autonomous driving vehicle and based on the target detection module. The original perception information may include: bounding box information of the target vehicle and surrounding vehicles, lane number information, etc. The target tracking module can be used to collect the driving trajectory data of the target vehicle and its surrounding vehicles, including the vehicle's historical trajectory coordinates, heading angle, lane number, vehicle number, and vehicle type. In some embodiments of the present invention, only 200 meters around the autonomous driving vehicle are considered for each acquisition segment. The two modules, the target detection module and the target tracking module, are inherent modules in the environmental perception system of the autonomous driving vehicle. They are professional terms in the industry and will not be elaborated on here.

[0095] Step 2: Perform data preprocessing on the driving trajectory data obtained in step 1 to obtain trajectory sequence data.

[0096] The preprocessing includes removing data noise and outliers. In some embodiments of the present invention, the trajectory sampling frequency is set to 10 Hz, the trajectory is smoothed using wavelet decomposition, and the trajectory sequence data is divided into several trajectory sequences according to a preset time length (such as 8 seconds), and the trajectory sequences whose total time length does not meet 8 seconds are removed.

[0097] Step 3: Analyze the correlation of variables and select the optimal variables to select the best combination of variables.

[0098] For the obtained trajectory sequence data, including the historical trajectory coordinates, heading angle and other variables mentioned in step 1, under the limited data collection capability, the variable correlation is analyzed and the best variable combination is screened out to achieve the expected balance between the prediction accuracy and prediction speed of the model.

[0099] In one embodiment of the present invention, the aforementioned steps 1, 2, and 3 can directly use the data of the public large-scale human natural driving trajectory dataset HighD. The HighD dataset is mainly a large-scale natural vehicle trajectory data of German highways, which includes the post-processed trajectories of 110,000 vehicles (including cars and trucks). A total of 60 recordings were made at six different locations, with an average duration of 17 minutes (a total of 16.5 hours), covering a road section of about 420 meters. In this embodiment, the data that the HighD dataset can provide include: vehicle historical trajectory coordinates, vehicle speed, vehicle acceleration, vehicle type, vehicle number, weekly vehicle number, etc. Up to 25 variables, first of all, these variables are analyzed for correlation with the trajectory prediction task, and the possible combinations of the optimal variables are listed by enumeration, and sequence data is constructed on the basis of each variable combination for training and verifying the trajectory prediction model, and the evaluation index of the trajectory prediction model is used to quantitatively analyze the pros and cons of each variable combination, and finally the variable optimization is completed.

[0100] Step 4: Perform coordinate transformation on the trajectory sequence data composed of the optimal variables and divide the data set. Each trajectory sequence includes an observation sequence and a prediction sequence.

[0101] In some embodiments of the present invention, in order to more efficiently utilize the existing trajectory sequence data, a sliding window method is used to process the trajectory sequence, and the window length is set to 8s. The trajectory sequence (fixed-length sequence) with a total duration of 8s for all vehicles (including the target vehicle and surrounding vehicles) is extracted, and the extracted trajectory sequence is divided into an observation sequence and a prediction sequence according to the timestamp. For each fixed-length sequence, a coordinate system is constructed with the prediction object, i.e., the target vehicle, as the origin, and the orientation of the target vehicle at the end of the observation sequence is taken as the positive direction of the x-axis (vertical axis), and rotated 90° counterclockwise as the positive direction of the y-axis (horizontal axis). The data is unified into the coordinate system through coordinate conversion for representation. Specifically, the fixed-length sequence is divided into the length of the observation sequence according to the timestamp. is 3s, the length of the predicted sequence The prediction length is 5s, and the trajectory within the prediction length is used as the true label of the prediction task. Finally, the trajectory sequence is divided into training, verification and test sets in a ratio of 6:2:2.

[0102] In some embodiments of the present invention, coordinate conversion is performed by the following formula:

[0103] ;

[0104] In the formula, For vehicles With vehicle The difference between the x and y coordinates; For vehicles The xy coordinates of For vehicles The xy coordinates of For vehicles Coordinate transformation to vehicle The rotation matrix of the coordinate system; For vehicles With vehicle The heading angle difference between For vehicles After the rotation matrix transformation, the vehicle Coordinate representation in coordinate system.

[0105] Step 5: Build and train the trajectory prediction model, which includes an embedding coding module for the observation sequence, a feature extraction encoder based on spatiotemporal attention, a fully connected layer, an embedding coding module for the prediction sequence, a decoder query vector generation module, and a decoder. The embedding coding module of the observation sequence is used to perform vector embedding on the observation sequence to obtain a high-dimensional representation of the observation sequence and position encoding on the high-dimensional representation of the observation sequence to obtain a position encoding result of the observation sequence. The feature extraction encoder based on spatiotemporal attention includes a joint spatiotemporal attention weight calculation module, which is used to perform spatiotemporal interaction feature encoding on vehicles in traffic scenes and output spatiotemporal attention. The fully connected layer is used to map the spatiotemporal attention to a hidden state vector. The embedding coding module of the prediction sequence is used to perform vector embedding on the prediction sequence to obtain a high-dimensional representation of the prediction sequence and position encoding on the high-dimensional representation of the prediction sequence to obtain a position encoding result of the prediction sequence. The decoder query vector generation module is used to consider the influence of the observation sequence, the prediction sequence and the spatiotemporal interaction, and generate a decoder query vector with the same length as the prediction sequence in a non-autoregressive decoding method. The decoder is used to obtain a future trajectory of the length of the prediction sequence; wherein the trajectory prediction model is trained by a loss function, and the loss function includes a dependency loss, and the coarse-grained temporal dependency loss and the fine-grained temporal dependency loss between the output future trajectory and the true label are calculated. Please refer to Figure 2 , the operations performed on the trajectory prediction model are described as follows:

[0106] Step 5.1: For the embedding coding module of the observation sequence, it performs the following operations:

[0107] a. The observation sequence is vector-embedded according to the following formula to obtain a high-dimensional representation of the observation sequence ,in Respectively represent the length of the input observation sequence, the number of vehicles contained in each observation sequence, and the set model dimension:

[0108] ;

[0109] in represents the fully connected layer, represents the input observation sequence;

[0110] b. High-dimensional representation of observation sequences The position coding result is obtained after using sine and cosine position coding PE :

[0111] ;

[0112] in, Represents a specific position in the observation sequence; Indicates a specific dimension index;

[0113] The processing process of the embedding coding module of the prediction sequence is basically the same as the aforementioned process of the embedding coding module of the observation sequence. The difference is that the embedding coding module of the prediction sequence is used to perform vector embedding and position coding on the prediction sequence to obtain the position coding result of the prediction sequence. .

[0114] Step 5.2: For a feature extraction encoder based on spatiotemporal attention, the feature extraction encoder of the spatiotemporal attention includes a joint spatiotemporal attention weight calculation module for performing temporal and spatial attention weight calculation. The execution process of the joint spatiotemporal attention weight calculation module is as follows:

[0115] In parallel, each of them uses the high-dimensional representation of the observation sequence through their own learnable parameter matrices to generate a time query vector, a time key vector, a time value vector, a spatial query vector, and a spatial key vector. Although the following operations are arranged in sequence, it should be noted that the operations are all performed in parallel:

[0116] a. Calculate the time attention weight and output time value vector: for each vehicle in the observation sequence , define three learnable matrices, namely the time query vector matrix , time key vector matrix and the time value vector matrix , a high-dimensional representation of the input observation sequence With position encoding results The result of the addition is multiplied by these three matrices respectively to obtain the corresponding time query vector , time key vector and the time value vector , when calculating temporal attention, the spatial dimension is regarded as BatchSize, where BatchSize is the batch size, and the relationship is as follows:

[0117] ;

[0118] ;

[0119] ;

[0120] in, is the time query vector of vehicle n, that is, the time query vector Focus on the specific vehicle n; is the calculated time attention weight, is the time value vector of vehicle n; is a high-dimensional representation of vehicle n; is the time key vector for vehicle n; is the position encoding result of vehicle n, with superscript is the transpose operation.

[0121] In this step, the time query vector is used to perform matrix multiplication with the time key vector. To align the dimensions, the time query vector needs to be transposed first, and then the time attention weight is calculated using formula (6). The time value vector is not subjected to any operation and is directly output to be used as one of the inputs for the subsequent joint calculation of spatiotemporal attention.

[0122] b. Calculate spatial attention weights: For each time step in the observation sequence, define two learnable matrices, namely the spatial query vector matrix and the spatial key vector matrix , a high-dimensional representation of the input observation sequence Multiply these two matrices respectively to get the corresponding spatial query vector , spatial key vector When calculating spatial attention, the time dimension is regarded as BatchSize, where BatchSize is the batch size, and the relationship is as follows:

[0123] ;

[0124] ;

[0125] in, is the calculated spatial attention weight for each time step, is the spatial query vector at time step t; is the spatial key vector at time step t; superscript Represents a transpose operation.

[0126] In this step, the spatial query vector is used to perform matrix multiplication with the spatial key vector. To align the dimensions, the spatial query vector needs to be transposed first, and then the spatial attention weight is calculated using formula (8).

[0127] c. Construct the time influence weight matrix: The time attention weight and the spatial attention weight are each represented by a vector, which is defined as the time attention weight set and the set of spatial attention weights , , The linear space dimension representing the temporal attention set is ,in is the number of all vehicles in the observation sequence, The length of the observation sequence input into the spatiotemporal attention weight calculation module. It should be noted that the spatiotemporal attention weight calculation modules used in the encoder and decoder are the same, but the lengths of the sequences input to the two are different. The observation sequence length is input in the encoder part, and the predicted sequence length is input in the decoder part. , represents the linear spatial dimension of the spatial attention set; for the time value vector set, it is , , is the linear space dimension representation of the time value vector, and for the time attention weight set The last dimension of is defined as the time influence weight matrix , is the linear spatial dimension representation of the time influence weight matrix, which will act on the spatial attention at the same time, as follows:

[0128] ;

[0129] in, It is defined as the spatial attention weight affected by time, that is, the spatiotemporal attention weight. It is also represented by a vector and defined as the spatiotemporal attention weight set , ， ; is the linear spatial dimension representation of the set of spatiotemporal attention weights.

[0130] d. Get encoder output: For each vehicle in the sequence , define the final high-dimensional representation of each vehicle as , based on the set of spatial attention weights and the time impact weight matrix The final high-dimensional representation of the vehicle is , and its calculation formula is as follows:

[0131] ;

[0132] in, , is the linear spatial dimensional representation of the encoder output of vehicle n, is the spatiotemporal attention of vehicle n, is the time value vector of vehicle n.

[0133] In summary, in view of the spatiotemporal interaction of vehicles during driving, in addition to considering the historical trajectory information of the target vehicle, the spatiotemporal interaction characteristics between vehicles are also comprehensively considered. The data contained in the observation sequence is first converted into a high-dimensional representation through vector embedding, and then sine and cosine position encoding is performed to obtain a high-dimensional representation with position encoding, which is input into the spatiotemporal attention feature extraction encoder, and the spatiotemporal attention is output through the spatiotemporal attention feature extraction encoder.

[0134] Step 5.3: Construct a decoder query vector generation module for generating decoder query vectors. The input of the decoder query vector generation module consists of three parts: the high-dimensional representation E of the observation sequence, the output spatiotemporal attention of the encoder, and the The result of the full connection layer and transposition, and the predicted sequence through the prediction sequence embedding coding module to obtain the position encoding result of the prediction sequence. The operations included in each part of the input are as follows:

[0135] a. Encoder output: The obtained encoder output is mapped to a hidden state vector through a fully connected layer. The hidden vector represents the influence of the features contained in the observation sequence on the generation of the decoder query vector;

[0136] b. Output of the predicted sequence after sine-cosine position encoding (the operation of the embedding encoding module of the predicted sequence is the same as that of the predicted sequence): After a fully connected layer, it is mapped to the predicted sequence position vector, which represents the influence of the position relationship of each time step of the predicted sequence on the generation of the decoder query vector; it also reflects the length of the decoder query vector, which is consistent with the length of the predicted sequence;

[0137] c. High-dimensional representation of the observation sequence: It adopts an idea similar to residual connection and incorporates the influence of the original observation sequence.

[0138] The specific process of the decoder query vector generation module is as follows:

[0139] 1) Calculate the hidden state For each target vehicle in the observation sequence , its hidden state , the calculation formula is as follows:

[0140] ;

[0141] in, is a learnable matrix, is a learnable bias parameter, is the linear space dimension representation of the hidden state, is the linear space dimension representation, which is a constant. is the bias parameter The linear space dimension representation of .

[0142] 2) Calculate the position encoding result of the predicted sequence after sine and cosine position encoding The resulting spatial influence weights are defined as , which means to characterize the influence of the predicted sequence on the decoder query vector after position encoding. The calculation formula is as follows:

[0143] ;

[0144] in, , is a learnable matrix, is a learnable bias parameter, is the linear spatial dimension representation of the spatial influence weight, is the linear space dimension representation, which is a constant. It is the position encoding result of the predicted sequence of each vehicle after sine and cosine position encoding.

[0145] 3) Generate a decoder query vector. Define the decoder query vector as , which is used as the query vector input to the decoder for attention weight calculation. Here, the influence of the observation sequence, prediction sequence and spatiotemporal interaction is comprehensively considered. The calculation formula is as follows:

[0146] ;

[0147] in, , , is a high-dimensional representation of the observation sequence of each vehicle, yes The linear space dimension representation of is the length of the observation sequence, yes The linear space dimension representation of is the length of the prediction sequence.

[0148] Finally, the decoder query vector generation module can generate a decoder query vector with the same length as the predicted sequence as the input of the spatiotemporal attention weight calculation module in the decoder. At the same time, the decoder query vector generation module avoids the situation in which the generation of the subsequent query vector in the autoregressive decoding method depends on the previous prediction result. Such methods are collectively referred to as non-autoregressive decoding methods.

[0149] Step 5.4: Construct a decoder. The decoder consists of two parts: the spatiotemporal attention weight calculation module and the cross attention weight calculation module, which are used for spatiotemporal attention weight calculation and cross attention weight calculation respectively. The decoder query vector is input into the spatiotemporal attention weight calculation module for processing, and the attention weight that comprehensively considers the spatiotemporal influence is obtained as one of the inputs of the cross attention calculation module. The cross attention weight calculation module in the decoder finally outputs the future trajectory.

[0150] The spatiotemporal attention weight calculation module in the decoder is consistent with the spatiotemporal attention calculation method in the encoder, and will not be elaborated here. Therefore, the following mainly describes the cross-attention weight calculation module. The input of the cross-attention weight calculation module includes two parts, namely the output of the encoder and the output of the spatiotemporal attention weight calculation module in the decoder; wherein, for the output of the encoder, it is used as the key vector and value vector of the cross-attention weight calculation by matrix multiplication with two learnable parameter matrices; for the output of the spatiotemporal attention weight calculation module in the decoder, it is used as the query vector of the cross-attention weight calculation by matrix multiplication with a learnable parameter matrix. The cross-attention calculation is performed according to formula (5) or formula (7) (the parameters in the formula are replaced with the relevant parameters of the prediction sequence, such as the query vector, key vector, value vector, and high-dimensional representation of the vehicle n of the prediction sequence in the cross-attention calculation mentioned above), and the final output is mapped to the future trajectory of the length of the prediction sequence through a fully connected layer.

[0151] Step 5.5: The loss function used in the training process of the trajectory prediction model is as follows:

[0152] ;

[0153] In the formula, is the total loss, including regression loss, classification loss and temporal dependency loss, is the timing dependency loss.

[0154] Among them, the binary Laplace negative log-likelihood loss function is used to train the regression task. For the future trajectory of the predicted sequence length, at each time step in the prediction time domain, the binary Laplace negative log-likelihood is used to calculate its location parameter and scale parameter loss. The formula is as follows:

[0155] ;

[0156] In the formula, is the loss of the regression task, is the predicted mode number; is the length of the predicted sequence; is the predicted Laplace distribution, the subscript mode indicates the specific mode, are the predicted value and true value at mode mode and time step t respectively, are the location parameter and scale parameter of the Laplace distribution at mode mode and time step t, respectively. In some embodiments of the present invention, The value of is 6. The value is 5 seconds.

[0157] Among them, for the probability of each mode, since the labels are all one-hot encoded, the probability corresponding to each mode does not actually have a true value. Therefore, the soft target cross entropy loss function is used to train the classification task to prevent the probability of the mode from appearing only 0 or 1. The calculation formula of the soft target cross entropy loss function is as follows:

[0158] ;

[0159] in, is the soft target cross entropy loss, is the length of the predicted sequence, is the predicted value of mode mode.

[0160] In order to enable the non-regressive decoding method to take into account the temporal dependency in the prediction sequence during decoding, the temporal dependency loss is introduced as an auxiliary task. This loss is added to the total loss of model training to further help optimize the model performance. The temporal dependency loss can be divided into coarse-grained temporal dependency loss and fine-grained temporal dependency loss. The coarse-grained temporal dependency loss is defined as the value obtained by performing a logical "not" operation on the predicted value and the true value at each time step in the prediction time domain after the sign function is passed, as shown in Figure 3(b). , ..., To predict the first-order difference value between two adjacent time steps in the time domain, the specific calculation example is: ..., and so on; is the set of true valued first-order differences, , ,……, ; is the set of first-order differences of the predicted values, , ..., ; Sign(.)is the sign function; The fine-grained temporal dependency loss is defined as the first-order difference between the predicted value and the true value at each time step in the prediction time domain, as shown in Figure 3(a). , ,……, To predict the true trajectory value of each time step in the time domain; , ,……, To predict the first-order difference value between two adjacent time steps in the time domain, the specific calculation example is: …and so on; , ..., is the predicted trajectory value for each time step in the prediction domain.

[0161] In the calculation process of coarse-grained temporal dependency loss, the sign function and XOR operation are used. The final loss calculation formula is as follows:

[0162] ;

[0163] ;

[0164] ;

[0165] In the formula, is the length of the predicted sequence, represents the prediction loss of the trajectory prediction model, Represent the true value and predicted value of the trajectory respectively; represents the fine-grained timing dependency loss, Represent the first-order difference of the true value of the trajectory at time t and the first-order difference of the predicted value at time t respectively; represents the temporal dependency loss of the trajectory prediction model, is the weight, which is used to reflect the coarse-grained temporal dependency loss. The total loss of the trajectory prediction model is determined by the weight Control the proportion of prediction loss and time-series dependency loss. As the prediction accuracy of the trajectory prediction model increases, the weight It gradually decreases, which means that the temporal dependency loss is reduced, that is, the trajectory prediction model models the temporal dependency better during the decoding process.

[0166] In addition, the following formula is used during the training process to verify and evaluate the accuracy and inference speed of the trajectory prediction model:

[0167] ;

[0168] In the formula, To predict the endpoint displacement error in the time domain, , Represents the true value and predicted value in the y-axis direction at time step t.

[0169] In some embodiments of the present invention, a 3-second historical trajectory is used to predict the trajectory of the next 5 seconds, and the evaluation index of the model prediction accuracy is the end point displacement error (Displacement Error, DE) in the prediction time domain from 1 second to 5 seconds.

[0170] In some embodiments of the present invention, the evaluation index of the model prediction speed is the average time taken for a preset number of model inferences, such as 100 times, and the unit is ms. It should be noted that the inference time is related to the machine used in the experiment. All time consumption data in the embodiments of the present invention are obtained on the CPU intel 13700 i7 and GPU NVIDIA 4060Ti O16G.

[0171] The present invention decomposes the trajectory prediction task into two stages: regression task and auxiliary task, which correspond to the forward reasoning and back propagation in the trajectory prediction model training process. The regression task is used to output the predicted future trajectory. The auxiliary task introduces the temporal dependency loss, calculates the coarse and fine-grained temporal dependency loss between the predicted future trajectory and the trajectory true value label, and optimizes the back propagation of the auxiliary trajectory prediction model.

[0172] Step 6: Input the driving trajectory data of the vehicle to be tested into the trained trajectory prediction model to obtain the trajectory prediction result.

[0173] In order to verify the superiority of the method proposed in the present invention, an ablation test is also set up. In some embodiments of the present invention, three models are obtained by training the PyTorch deep learning framework. Figure 4 , respectively the method proposed in the present invention, the method based on autoregressive decoding and the method based on MLP decoding. The model based on the MLP decoding method is the same as the models of the other two methods in the encoder part, only the decoder is replaced, and the decoder part is replaced by a multi-layer perceptron (which includes two fully connected layers, ReLU as the activation function, and a layer of LayerNorm). The following analyzes the superiority of the trajectory prediction model based on non-autoregressive and considering time series dependency of the present invention through the results of two comparative experiments. The comparison indicators are the displacement error (minimum unit is second) of each time step in the prediction time domain and the inference time of the model (unit is milliseconds). The comparison result is the result of selecting the best mode from the predicted multi-mode. The selection strategy is to regard the mode with the smallest displacement error at the end point as the best mode. The comparison result is the value obtained by averaging after inference for each model 100 times.

[0174] Comparative experiment 1: The proposed method Vs based on autoregressive decoding method to verify the influence of autoregressive and non-autoregressive decoding methods on model prediction speed

[0175] Analysis of similarities and differences between the two models: The similarity is that both are models that consider time dependency for decoding, and the difference is that the method proposed in the present invention uses a decoder query vector generation module to transform the autoregressive decoding method into a non-autoregressive decoding method. The comparison results are shown in Table 1.

[0176] Table 1 Comparison results of the method proposed in this invention, the method based on autoregressive decoding, and the method based on MLP decoding

[0177]

[0178] As can be seen from Table 1, the proposed method based on non-autoregressive decoding can generate a decoder query vector with the same length as the predicted sequence, so that the decoding at the current moment in the decoding process no longer depends on the prediction result at the previous moment, which greatly speeds up the inference time. After changing the model decoding mode to non-autoregressive decoding, the temporal dependency of the predicted sequence will be lost, but the cumulative error caused by autoregressive decoding is also avoided. From Table 1, the displacement error and Figure 5 From the above, the method of the present invention improves the average speed by 38.46%, 36.36%, 32.53%, 20.83% and 8.47% per second respectively. At the same time, this result is also due to the decoding scheme that considers the timing dependency proposed by the present invention, which makes up for the dependency of non-autoregressive decoding on timing compared with autoregressive decoding. Figure 6 The single inference time of the method proposed in the present invention is only 13.86ms, while the single inference time of the autoregressive decoding method is 394.71ms, and the inference time will increase with the increase of the prediction time domain, because the autoregressive decoding method depends on the prediction results of the previous moment. The method proposed in the present invention has greatly improved inference, and the inference time is only 2.53% of the original.

[0179] Comparative experiment 2: The proposed method vs. the MLP decoding method to verify the impact of timing dependency on model prediction accuracy

[0180] Analysis of the similarities and differences between the two methods: the similarity is that both are based on non-autoregressive decoding. The feasibility of non-autoregressive decoding of the method proposed in the present invention comes from the decoder query vector generation module proposed in the present invention, and the non-autoregressive decoding based on the MLP decoding method comes from the commonly used multi-layer perceptron decoding; the difference is that the method proposed in the present invention considers timing dependency in the decoding process, while the MLP decoding method does not consider timing dependency in the decoding process.

[0181] The method proposed in the present invention and the method based on MLP decoding are both based on non-autoregressive decoding, and the reasoning time of both is much less than that based on autoregressive decoding (such as the method based on autoregressive decoding mentioned in Table 1), wherein the reasoning time of the method proposed in the present invention is only 13.86ms, compared with 50.01ms based on MLP decoding method, the reasoning time is only 72.29% of the original, indicating that the method proposed in the present invention is faster than the commonly used MLP decoding method. At the same time, the most important thing is that the displacement error of the method proposed in the present invention in the prediction time domain of the next 5s is smaller than that of the method based on MLP decoding, and the average improvement is 35.16%, 43.66%, 48.15%, 49.67% and 47.06% per second, especially in the long-term prediction (3s-5s) segment, it can be seen that the lack of explicit modeling of temporal dependency and the direct output from feature mapping to the prediction sequence length leads to a rapid increase in the model error in the long-term prediction segment and a large absolute error.

[0182] In summary, the intelligent vehicle multimodal trajectory prediction method based on non-autoregressive and temporal dependency decoding proposed in the present invention can provide a reference for achieving accurate and fast real-time trajectory prediction of intelligent vehicles.

[0183] In some embodiments of the present invention, a computer device is further provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the aforementioned embodiment.

[0184] In some embodiments of the present invention, a computer-readable storage medium is further provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the method described in the aforementioned embodiment are implemented.

[0185] The present invention improves the original Transformer decoding method from autoregressive to non-autoregressive and considers time-dependent decoding in a vehicle multimodal trajectory prediction model, which can generate a decoder query vector with the same length as the prediction sequence, that is, decoding can be performed in parallel, which greatly improves the decoding speed and effectively suppresses the cumulative error caused by autoregressive decoding, thereby improving the prediction accuracy of the model.

[0186] While adopting the non-autoregressive decoding method, a loss that considers time dependence is designed as an auxiliary task, which explicitly models the decoding process of the prediction sequence in time series. This is helpful to solve the problem of decreased model accuracy caused by the previous non-autoregressive decoding-based models directly using nonlinear models such as MLP to fit time series dependence.

[0187] In summary, the present invention can provide a reference for achieving accurate and fast real-time trajectory prediction of smart cars.

[0188] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the present invention may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present invention, but will conform to the widest scope consistent with the principles and novel features disclosed in the present invention.

Claims

1. A multimodal non-autoregressive trajectory prediction method for an intelligent vehicle, characterized in that: The following steps are involved: Acquire the driving trajectory data of the vehicle, obtain a trajectory sequence based on the driving trajectory data, and divide the trajectory sequence into an observation sequence and a prediction sequence; Inputting the vehicle's driving trajectory data into the trained trajectory prediction model to obtain a trajectory prediction result, wherein the trajectory prediction model includes an observation sequence embedding coding module, a spatiotemporal attention-based feature extraction encoder, a fully connected layer, a prediction sequence embedding coding module, a decoder query vector generation module, and a decoder; The embedding coding module of the observation sequence is used to perform vector embedding on the observation sequence to obtain a high-dimensional representation of the observation sequence and position encoding on the high-dimensional representation of the observation sequence to obtain a position encoding result of the observation sequence; the feature extraction encoder based on spatiotemporal attention includes a joint spatiotemporal attention weight calculation module, which is used to perform spatiotemporal interaction feature encoding on vehicles in the traffic scene and output spatiotemporal attention; the fully connected layer is used to map the spatiotemporal attention to a hidden state vector; the embedding coding module of the prediction sequence is used to perform vector embedding on the prediction sequence to obtain a high-dimensional representation of the prediction sequence and position encoding on the high-dimensional representation of the prediction sequence to obtain a position encoding result of the prediction sequence; The decoder query vector generation module is used to consider the influence of the observation sequence, the prediction sequence and the spatiotemporal interaction, and generate a decoder query vector with the same length as the prediction sequence in a non-autoregressive decoding method; the decoder is used to obtain the future trajectory of the prediction sequence length; Among them, the loss function used when training the trajectory prediction model includes a temporal dependency loss, and the temporal dependency loss is used to calculate a coarse-grained temporal dependency loss and a fine-grained temporal dependency loss between the output future trajectory and the true label; In the feature extraction encoder based on spatiotemporal attention, the execution process of the joint spatiotemporal attention weight calculation module includes: Calculate the time attention weight and output the time value vector. The formula of the time attention weight is: In the formula, is the calculated time attention weight, It is a vehicle The time query vector, is the transpose operation, It is a vehicle The time key vector of It is a vehicle The high-dimensional representation of It is a vehicle The position encoding result is It is a vehicle A vector of time values, is the model dimension being set; Calculate the spatial attention weight, the calculation formula is: In the formula, is the calculated spatial attention weight for each time step, is at the time step The spatial query vector of is at the time step The spatial key vector of ; The time influence weight matrix is ​​constructed to obtain the spatiotemporal attention weight, and the expression of the spatiotemporal attention weight is: In the formula, is the spatiotemporal attention weight, is the time influence weight matrix, is the time attention weight set The last dimension, the set of time attention weights , is the number of all vehicles in the observation sequence; Get encoder output: For each vehicle in the observation sequence, define the final high-dimensional representation of each vehicle as , based on the set of spatial attention weights and the time impact weight matrix The final high-dimensional representation of the vehicle is , the calculation formula is as follows: In the formula, , It is a vehicle The linear space dimension representation of the encoder output is, is the length of the observation sequence input to the spatiotemporal attention weight calculation module, It is a vehicle The spatial and temporal attention It is a vehicle A vector of time values.

2. The method for predicting multimodal non-autoregressive trajectories of an intelligent vehicle according to claim 1, characterized in that: Methods for obtaining vehicle driving trajectory data include: using an open source vehicle trajectory dataset or identifying a target vehicle, and collecting driving trajectory data of the target vehicle and its surrounding vehicles through on-board sensors.

3. The method for predicting multimodal non-autoregressive trajectories of an intelligent vehicle according to claim 1, characterized in that: A fixed-length sequence is extracted based on the trajectory sequence, and the fixed-length sequence is divided into an observation sequence and a prediction sequence. For each fixed-length sequence, a coordinate system is constructed with the prediction object, i.e., the target vehicle, as the origin, and the orientation of the target vehicle at the end of the observation sequence is taken as the positive direction of the vertical axis, and the positive direction of the horizontal axis is taken as 90° counterclockwise rotation. The data is unified into the coordinate system through coordinate conversion for representation.

4. The method for predicting multimodal non-autoregressive trajectories of an intelligent vehicle according to claim 3, characterized in that: Before coordinate transformation, the variables in the trajectory sequence are first analyzed for correlation with the trajectory prediction task, the possible combinations of optimal variables are listed, the optimal combination of variables is screened, and the trajectory sequence data composed of the optimal combination of variables is subjected to fixed-length sequence extraction and coordinate transformation.

5. The method for predicting multimodal non-autoregressive trajectories of an intelligent vehicle according to claim 1, characterized in that: The execution process of the decoder query vector generation module includes: For each target vehicle in the observation sequence, its hidden state is , is the linear space dimension representation of the hidden state, the hidden state The calculation formula is as follows: In the formula, is a learnable matrix, is a learnable bias parameter, is a matrix The linear space dimension representation of is a learnable bias parameter Linear space dimension representation of ; Calculate the position encoding result of the predicted sequence after sine and cosine position encoding The generated spatial influence weights , the calculation formula is: In the formula, , is the linear spatial dimension representation of the spatial influence weight, is a learnable matrix, is a learnable bias parameter, is the position encoding result of the prediction sequence of each vehicle after sine-cosine position encoding; Define the decoder query vector as , the expression is: In the formula, , yes The linear space dimension representation of , yes The linear space dimension representation of is a high-dimensional representation of the observation sequence of each vehicle, is the length of the observation sequence, T pred is the length of the prediction sequence.

6. The method for predicting multimodal non-autoregressive trajectories of an intelligent vehicle according to claim 1, characterized in that: The decoder includes a spatiotemporal attention weight calculation module and a cross attention weight calculation module, and the calculation process of the spatiotemporal attention weight calculation module is different from the calculation process performed by the joint consideration spatiotemporal attention weight calculation module of the encoder in that the sequence in the spatiotemporal attention weight calculation module of the decoder is a prediction sequence; The input of the cross-attention weight calculation module includes two parts, namely the output of the encoder and the output of the spatiotemporal attention weight calculation module in the decoder. For the output of the encoder, the output of the encoder is matrix multiplied with two learnable parameter matrices as the key vector and value vector of the cross-attention weight calculation; for the output of the spatiotemporal attention weight calculation module in the decoder, it is matrix multiplied with a learnable parameter matrix as the query vector of the cross-attention weight calculation, and then the cross-attention calculation is performed. The output is mapped to the future trajectory of the predicted sequence length through a fully connected layer.

7. A method for predicting multimodal non-autoregressive trajectories of an intelligent vehicle according to any one of claims 1 to 6, characterized in that: The loss function used in the training process of the trajectory prediction model is: In the formula, is the total loss, including regression loss, classification loss and temporal dependency loss, is the timing dependency loss, is the soft target cross entropy loss, The loss for the regression task; Among them, the timing dependency loss for: In the formula, represents the prediction loss of the trajectory prediction model, represents the fine-grained timing dependency loss, is a weight used to reflect the coarse-grained timing dependency loss.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of any one of claims 1-7.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images