Multi-modal vehicle trajectory prediction method driven by target and distribution heat map output

Abstract

The application discloses a multi-modal vehicle trajectory prediction method of target driving and distribution heat map output, comprising the following steps: constructing a trajectory data set containing high-precision map data; decomposing a trajectory prediction task into two stages of target point prediction and trajectory regression; constructing a target point prediction network encoder to respectively extract features of historical motion and a rasterized map; fusing and interacting the extracted features through a multi-stage attention mechanism; obtaining a target point heat map distribution through a decoder based on transposed convolution; training the target point prediction network according to trajectory data in the data set, and setting hyperparameters and a training strategy; after the training of the target point prediction network is completed, using a sampling algorithm to sample multiple target points from the heat map to correspond to different modes in reality; building a trajectory regression network with the sampled target points and fused features as inputs; and training the trajectory regression network, so that the application can improve the scene coverage of a trajectory prediction model.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving perception technology and relates to a trajectory prediction method, specifically a multimodal vehicle trajectory prediction method based on target-driven and distributed heatmap output. Background Technology

[0002] Autonomous driving technology is of great significance for improving traffic safety and efficiency, promoting innovation and upgrading in the automotive industry, and meeting consumer travel needs. The trajectory prediction module is a crucial component of autonomous driving systems, enabling vehicles to anticipate the future states of surrounding road users and make safe and rational decisions and plans. Without the prediction module, environmental perception can only capture the current state of the surrounding environment; focusing solely on the current state is insufficient for safety, especially in highly interactive scenarios (such as busy intersections).

[0003] Because causal inference is involved, prediction problems are difficult, and future outcomes often face a huge solution space. Natural language processing addresses this by training large models on massive amounts of data to cover multiple feasible solutions. The field of autonomous driving lacks such massive amounts of data, and real-time requirements limit model size. However, vehicle behavior is rule-based; vehicles on roads mostly follow traffic rules and travel along designated lanes. Furthermore, vehicle state variables such as heading and speed adhere to vehicle dynamics constraints and do not undergo sudden changes in a short period. These specific structures and behaviors reduce the solution space of trajectory prediction problems, making it unlike natural language processing, which requires massive amounts of data to cover the solution space.

[0004] Trajectory prediction involves the vehicle itself and other road users, with the vehicle acting as the driver and predicting the behavior of the other road users. Since vehicles interact with each other extensively, effective methods for modeling these interactions are necessary. High-definition maps (HD maps) contain a wealth of data influencing vehicle behavior, such as lane lines, lane boundaries, drivable areas, and pedestrian crossings, which are crucial for trajectory prediction. Currently, trajectory prediction still faces significant challenges, primarily including: 1) Interactions among road users. Roads are not always confined to a single vehicle; busy sections may have dozens. Different vehicle behaviors can lead to different outcomes; for example, a vehicle slowing down can cause following vehicles to slow down or overtake. Accurate prediction requires accurate interaction modeling; 2) Utilizing HD map information. HD maps contain a wealth of useful information that plays a vital role in constraining vehicle behavior and guiding vehicle direction. For instance, placing yield signs at intersections can completely alter the behavior of approaching vehicles. Integrating HD map information into the model is also a challenge; 3) Predicted trajectories are inherently multimodal. Cars approaching an intersection may exhibit multiple behaviors, such as going straight, turning, or changing lanes. Predictive models need to generate various possible outcomes and output the probability or confidence level of each. Therefore, it is necessary to propose an effective multimodal prediction method that considers the interactions of multiple features. (Multimodal TrajectoryPredictions for Autonomous Driving using Deep Convolutional Networks)(Socialgan: Socially acceptable trajectories with generative adversarial networks)(GRIP: Graph-based Interaction-aware Trajectory Prediction) Most existing methods lack modeling of feature interactions or fail to consider the impact of high-precision map information, severely limiting trajectory prediction performance. Furthermore, some methods do not consider the impact of multimodal prediction, resulting in a lack of diversity in prediction results. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a multimodal vehicle trajectory prediction method based on target-driven and distributed heatmap output, which can improve the scene coverage of the trajectory prediction model and ensure that the predicted trajectory conforms to traffic rules and human driving behavior.

[0006] The present invention is achieved by at least one of the following technical solutions.

[0007] A target-driven, multimodal vehicle trajectory prediction method based on distributed heatmap output includes the following steps:

[0008] (1) Select a vehicle around the autonomous vehicle as the prediction subject and collect the trajectory data of the prediction subject. Collect the driving trajectory of the surrounding vehicle through the target tracking module and record the type of each traffic participant. Divide the collected trajectory into segments of length T according to the timestamp. obs The historical trajectory and length are T pred The future trajectory of the subject being predicted is used as the true label for the prediction task.

[0009] (2) Clean the collected trajectories, mine the corresponding prediction scenarios, and divide the collected trajectories into training, validation and test sets;

[0010] (3) Preprocess driving trajectory and high-precision map data: Represent the information of the predicted subject and surrounding vehicles in the motion vector, and represent the high-precision map information in a grid map of a specific size, as the known road information for the trajectory prediction task;

[0011] (4) The trajectory prediction task is decomposed into two stages: target point prediction and trajectory regression, in order to correspond to the motivational uncertainty and motion uncertainty in the real driving process.

[0012] (5) Build a target point prediction network to predict the terminal point of the future trajectory, and define a loss function to guide the training of the target point prediction network;

[0013] (6) After training the target point prediction network, different strategies are used to obtain dissimilar multimodal target points from the target point prediction network.

[0014] (7) Based on the target point and context information, a trajectory regression network is built to complete the trajectory regression.

[0015] Furthermore, for each prediction scenario, the position of all trajectory data is transformed into a coordinate system with the prediction subject as the origin and the orientation of the prediction subject at the last observation time as the x-axis.

[0016] Furthermore, the motion vector includes the type of traffic participant, historical trajectory coordinates, and time interval; the time interval is represented by the difference between the current time and the previous time; the rasterized map includes lane centerline data, lane boundary, and drivable area information, where lane centerline data occupies 3 channels and is represented by HSV encoding, where H is hue, ranging from 0° to 360°; S is saturation, ranging from 0 to 1; and V is brightness; lane boundary and drivable area data occupy 1 channel, and if there is data at a certain location, the pixel value at the corresponding location is set to 255, otherwise it is 0.

[0017] Furthermore, the construction and training of the target point prediction network includes the following steps:

[0018] 1) Assuming the target point distribution follows a two-dimensional discrete distribution, a deep neural network is used to fit it, and the network output is called a distribution heatmap; a target point prediction network is built based on an encoder-decoder architecture:

[0019] a. Motion feature encoder based on U-LSTM: Trajectory history data is first fused through one-dimensional convolutional blocks, and then fused again through U-LSTM to output a feature vector of a specific dimension;

[0020] b. Swing Transformer-based map feature encoder: Extract raster map features using Swing Transformer to obtain a larger receptive field;

[0021] c. Feature fusion module based on multi-stage attention mechanism: It fuses historical motion and high-precision map features through attention mechanism, and explicitly adds relative distance information as a supplement to enhance feature representation ability;

[0022] d. Transposed convolution-based decoder: The feature map is upsampled by transposed convolution blocks to the original raster map size. The original high-precision map is then stitched onto the feature map to compensate for the information lost during downsampling. Finally, the number of channels is reduced to 1 by convolution blocks, and the output of each pixel is constrained to the range [0,1] by the Sigmoid function.

[0023] 2) The network is trained using pixel-wise focal loss. The true distribution Y is a two-dimensional Gaussian distribution centered on the true target point. Let the value at position (i,j) on the output heatmap be... The value at position (i,j) in the true distribution is Y. ij The loss function is expressed as follows:

[0024]

[0025]

[0026] in, The true distribution Y and the output heatmap distribution The loss function; To output the value in the i-th row and j-th column of the heatmap; Y ij represents the value in the i-th row and j-th column of the true distribution; H and W are the height and width of the heatmap, respectively; β is the weighting coefficient.

[0027] 3) Use the data processed in step (3) to train the target point prediction network, set hyperparameters and training strategies, select the model network with the best training effect through the validation set, and evaluate its performance in the test set.

[0028] Furthermore, at the output of each transposed convolutional block of the decoder, the original image size is restored through an auxiliary transposed convolutional block, the number of channels is converted to 1, and then a heatmap distribution is output through a sigmoid function. Supervised learning is then performed using pixel-by-pixel focus loss. The total loss function will include the output loss after passing through multiple transposed convolutional blocks and the auxiliary loss within the transposed convolutional blocks, as shown in the following equation:

[0029]

[0030] in, is the output of the l-th transposed convolutional block, and α is a hyperparameter used to balance the two losses, used to control the coefficients before the auxiliary loss.

[0031] Furthermore, the motion feature encoder based on U-LSTM is an improved Long Short-Term Memory (LSTM) network structure. It first performs a reverse LSTM process to obtain future information, and then passes its output to the forward LSTM.

[0032] Furthermore, the Swin Transformer-based map feature encoder comprises three stages, each stage performing a downsampling of the image once, and each stage includes two Swin Transformer blocks.

[0033] Furthermore, the MR (Miss Rate) sampling algorithm and the FDE (Final Displacement Error) sampling algorithm are used to obtain dissimilar multimodal target points from the target point prediction network;

[0034] The MR sampling algorithm includes the following steps:

[0035] The K target points obtained by the MR sampling algorithm are used as the initial target points: the region with the maximum probability integral value is selected iteratively, and after the corresponding region is selected, the probability value of all positions in the region is set to 0, and the center of the region is used as the prediction target point. This process is repeated K times to obtain K target points, which are the prediction modes.

[0036] The FDE sampling algorithm uses K target points obtained by the MR sampling algorithm as initial target points and iteratively optimizes them to obtain multimodal target points. The main steps include:

[0037] Calculate position x on the heatmap i and center point distance Where the subscript i represents the i-th position;

[0038] Then assume position x i The nearest center point is Calculate x for each point i and the nearest center point distance m i ;

[0039] For the k-th center point Filter out neighborhood points within a meter range, perform a weighted average to obtain the new center point coordinates, and proceed to the next iteration. The weights of the weighted average are: and The product of p, where p i For position x i By repeating the above process L times on the heatmap, a more accurate sampling target point can be obtained.

[0040] Furthermore, the trajectory regression network includes the following modules:

[0041] Information pooling layer: The intermediate layer features of the target point prediction network are flattened into a one-dimensional vector through global average pooling and concatenated with the target point features. The intermediate layer features of the target point prediction network refer to the output feature map after the scene input is processed by U-LSTM, Swin Transformer and multi-stage attention mechanism, which contains rich road structure information.

[0042] Linear residual blocks consist of fully connected layers, layer normalization layers, and ReLU activation functions, which perform multi-level transformations on the input dimension.

[0043] Furthermore, the trajectory regression network based on target point and context information is trained using Huber Loss. The loss function is expressed as:

[0044]

[0045] in, This indicates the coordinates of the points predicted by the trajectory prediction model, including the x and y coordinates. Represents the actual coordinates; The predicted coordinate x at time t F,t and actual coordinates Huber losses between; The predicted coordinates y at time t F,t and actual coordinates Huber losses between; a total of T pred At each prediction time, variables The Huber loss with respect to variable y is defined as follows:

[0046]

[0047] Where δ is a hyperparameter in the loss.

[0048] The multimodal vehicle trajectory prediction method based on target-driven and distributed heatmap output described in this invention has the following advantages compared with existing technologies:

[0049] 1. The target-driven multimodal vehicle trajectory prediction framework proposed in this invention decomposes trajectory prediction into two stages: 1) target point prediction; 2) trajectory regression based on target points. This decomposition has a realistic basis: target point prediction corresponds to the uncertainty of motivation and models behavioral intention; trajectory regression corresponds to the uncertainty of motion and considers the errors in the vehicle motion control process. Therefore, this framework provides a certain degree of interpretability.

[0050] 2. The multimodal vehicle trajectory prediction method proposed in this invention, which uses target-driven and distributed heatmap output, can effectively model the distribution of target points through heatmap output, thereby reducing the trajectory loss rate (MR). The loss rate is an important trajectory prediction evaluation metric, representing the proportion of scenarios where all multimodal predicted trajectories deviate from the true trajectory. Compared to the final displacement error, the loss rate is often more important in practice because it better reflects the generalization ability of the prediction model in complex and ever-changing real-world scenarios. Reducing the loss rate can improve scene coverage, thus providing a greater advantage in real-world scenarios.

[0051] 3. The map feature encoder based on Swing Transformer proposed in this invention expands the receptive field of the image through the attention mechanism of the moving window, effectively combines road network information, improves the performance of target point prediction, and reduces the size of the model through lightweight configuration, thus providing a guarantee for real vehicle applications.

[0052] 4. The feature fusion module based on a multi-stage attention mechanism proposed in this invention during the target point prediction stage effectively solves the feature interaction problem in the vehicle trajectory prediction process, making features no longer independent of each other and thus improving the model's expressive power. The fusion process first models the interaction between map information and the agent. Because lane lines guide the agent's movement and lane boundaries constrain the agent's movement range, a cross-attention mechanism is used to enable the agent to perceive map information. Next, the interaction between agents is modeled. Considering that agents can also infer lane information from the behavior of other agents, a self-attention mechanism is used to achieve the fusion of features between agents.

[0053] 5. In the A2A feature fusion module, this invention explicitly introduces relative distance information, constructs relative distance features between other agents and the target agent, and integrates an attention mechanism module to make the influence of relative distance more significant, further improving trajectory prediction performance. In human driving, there is also a tendency to prioritize the influence of nearby vehicles, giving them greater weight; therefore, considering relative distance information aligns with driving experience.

[0054] 6. The heatmap output loss based on pixel-wise focal loss proposed in this invention extends focal loss to dense prediction tasks, effectively solving the problem of imbalanced positive and negative samples during model training. This loss not only addresses sample imbalance but also the problem of hard samples, increasing the proportion of poorly predicted pixels in the loss. For the location of the true target point, the weight coefficient β forces the predicted value at that location to be close to the actual value, ensuring accurate coverage of the target point by the heatmap output.

[0055] 7. The target-driven, distribution heatmap output multimodal vehicle trajectory prediction method proposed in this invention, in addition to supervised training using pixel-wise focal loss during the training phase, also sets up an auxiliary task. At the output of each transposed convolutional block of the decoder, an auxiliary transposed convolutional block restores the original image size, converting the number of channels to 1. Then, a heatmap distribution is output using the sigmoid function, and supervised learning is conducted using pixel-wise focal loss. This forces each layer of the decoder to contain information capable of recovering the true distribution, improving the training speed and performance of the model.

[0056] 8. The trajectory regression network proposed in this invention, based on target point and context information, considers the impact of vehicle-to-vehicle interaction and map information on the predicted trajectory after the target point is known. The input includes fused features and sampled target point information, making the predicted trajectory more reasonable. The fused features are generated by the encoder of the target point prediction network. In order not to affect the target point prediction network, the parameters of the target point prediction network are frozen to achieve decoupling between target point prediction and trajectory regression. Attached Figure Description

[0057] Figure 1 This is a flowchart of a multimodal vehicle trajectory prediction method based on target-driven and distributed heatmap output according to the present invention;

[0058] Figure 2 This is a diagram of the target point prediction network model architecture based on heatmap output in the method of the present invention;

[0059] Figure 3 This is a structural diagram of a motion feature encoder based on U-LSTM according to an embodiment of the present invention;

[0060] Figure 4 This is a diagram of the U-LSTM structure according to an embodiment of the present invention;

[0061] Figure 5 This is a structural diagram of a map feature encoder based on Swing Transformer according to an embodiment of the present invention;

[0062] Figure 6 This is a schematic diagram of the M2A process according to an embodiment of the present invention;

[0063] Figure 7 This is a schematic diagram of the A2A process according to an embodiment of the present invention;

[0064] Figure 8 This is a diagram of the attention layer structure according to an embodiment of the present invention;

[0065] Figure 9 This is a structural diagram of a transposed convolutional block according to an embodiment of the present invention;

[0066] Figure 10 This is a graph showing the qualitative results of the test set in an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific implementation steps. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] The present invention discloses a multimodal vehicle trajectory prediction method based on target-driven distribution heatmap output, comprising: constructing a trajectory dataset containing high-precision map data; transforming all data points into a coordinate system with the predicted target as the origin, and converting the high-precision map information into a raster map; decomposing the trajectory prediction task into two stages: target point prediction and trajectory regression; constructing a target point prediction network encoder to extract features from historical motion and the raster map respectively; fusing the extracted features through a multi-stage attention mechanism; passing the fused features through a transposed convolution-based decoder to obtain the target point heatmap distribution; training the target point prediction network using pixel-wise focal loss based on the trajectory data in the dataset, and setting appropriate hyperparameters and training strategies; after completing the training of the target point prediction network, sampling multiple target points from the heatmap using a specific sampling algorithm to correspond to different modalities in reality; constructing a trajectory regression network composed of residual connection blocks, using the sampled target points and fused features as input; training the trajectory regression network using Huber Loss; and setting appropriate hyperparameters and training strategies to train the trajectory regression network.

[0069] In this embodiment, traffic participants are collectively referred to as agents, and a vehicle of interest around the autonomous vehicle is selected as the prediction subject; this vehicle is called the target agent. Figure 1 The target-driven multimodal vehicle trajectory prediction framework shown includes the following steps:

[0070] (1) Select a vehicle of interest around the autonomous vehicle as the prediction subject and collect its trajectory data. Collect the driving trajectories of surrounding traffic participants through a target tracking module and record the type of each traffic participant (e.g., vehicle, pedestrian, bicycle, etc.). Obtain road network information through a high-precision map and combine it with the historical motion information such as position and speed of all traffic participants to construct a trajectory prediction task and predict the future trajectory of the target agent. Divide the collected trajectory into segments of length T according to timestamps. obs The historical trajectory and length are T pred The future trajectory of the target agent is used as the true label for the prediction task.

[0071] (2) The collected trajectory set is cleaned to identify interesting prediction scenarios, including: a) intersections; b) left or right turns; c) turning into adjacent lanes; d) vehicles in dense traffic. The collected trajectory set is divided into training, validation, and test sets. For each prediction scenario, only the influence of traffic participants within a 100-meter radius of the target agent is considered. The agent's position is represented by coordinate points, with the target agent as the origin and the orientation of the target agent at the last observation time as the x-axis. All agent positions are transformed into the defined coordinate system.

[0072] As a preferred embodiment, the data acquisition and cleaning processes in steps (1) and (2) will directly utilize data from the large public trajectory prediction dataset Argoverse. The Argoverse dataset, primarily collected from Pittsburgh and Miami, contains trajectories of over 300,000 target agents and rich semantic map data, covering 290 kilometers. Ultimately, 324,557 5-second sequences were collected, with a fixed sampling frequency of 10Hz. A prediction task is constructed based on the collected 5-second sequences, and T is set... obs =2,T pred =3. To evaluate the performance of multimodal prediction, K output trajectories will be considered for a single scene. This embodiment uses K=6 as an example, which also corresponds to the Argoverse standard. For 324,557 sequences, 205,942 are divided into the training set, 39,472 into the validation set, and 78,143 into the test set. To ensure data diversity, the training, validation, and test sets are taken from disjoint portions of the cities, that is, approximately 1 / 8 and 1 / 4 of each city are used as validation and test data, respectively.

[0073] For trajectory data and road network data in each scene, coordinate system transformation is performed, with the target agent as the origin and the orientation of the target agent at the last observation time as the x-axis, transforming all data into the defined coordinate system.

[0074] (3) Preprocess the driving trajectory and high-precision map data, representing the information of the target agent and other traffic participants into a vector of a specific length. The motion vector includes features such as the type of traffic participant, historical trajectory coordinates, and time interval. The time interval is represented by the difference between the current time and the previous time. The high-precision map information is represented into a raster map of a specific size. The raster map contains information such as lane centerline data, lane boundaries, and drivable areas. Among them, the lane centerline data occupies 3 channels and is represented by HSV encoding to consider the directionality of the centerline. In HSV representation, H is hue, with a value range of 0° to 360°, which corresponds exactly to the direction angle of the centerline; S is saturation, with a value range of 0 to 1, and its value is 1; V is value, with a value of 1. Lane boundary and drivable area data occupy 1 channel. If there is data at a certain position, the pixel value at the corresponding position is set to 255; otherwise, it is 0.

[0075] (4) The trajectory prediction task is decomposed into two stages: target point prediction and trajectory regression, to correspond to the motivational and motion uncertainties in the actual driving process. Target point prediction predicts the coordinates of the last moment of the future trajectory; trajectory regression uses the predicted target point as a condition to regress and complete the predicted trajectory.

[0076] (5) For the target point prediction stage, a target point prediction method based on heatmap output is proposed. Assuming the target point distribution follows a two-dimensional discrete distribution, a target point prediction network based on the heatmap distribution is constructed to fit this distribution. The target point prediction network uses an encoder-decoder architecture, as shown in the following structure... Figure 2 As shown, it includes the following modules:

[0077] a. U-LSTM-based motion feature encoder: Motion history data is first fused using a one-dimensional convolutional block (Conv1dBlock), then fused again using U-LSTM to output a feature vector of a specific dimension, as shown in the structure below. Figure 3 As shown, Conv1dBlock consists of a 5×5 one-dimensional convolutional Conv1d kernel, a batch normalization (BN) layer, and a Rectified Linear Unit (ReLU). U-LSTM is an improved long short-term memory structure, as shown in the diagram. Figure 4As shown, a reverse LSTM process is first performed to obtain future information, and then its output is passed to a forward LSTM to accumulate information. This also helps the forward process understand which information will be useful in the future.

[0078] b. Swin Transformer-based map feature encoder: Extracts rasterized map features using a lightweight Swin Transformer. It consists of three stages, each performing a downsampling operation on the image. Internally, it contains two Swin Transformer blocks, such as... Figure 5 As shown. The patch partitioning operation divides the input data into blocks; the linear embedding layer performs dimensionality transformation on the partitioned blocks; the Swin Transformer blocks further extract features without changing the feature dimensions; the downsampling process is completed by the patch merging operation. Considering that the same lane will have different effects when located at different positions in the raster map, it is beneficial to explicitly introduce location coordinates. The coordinate information of each pixel is represented by concatenating coordinate information in the channel dimension of the raster map through reference coordinate convolution to obtain a larger receptive field.

[0079] c. Feature fusion module based on multi-stage attention mechanism: This module extracts salient parts of map features from different agents using M2A (Map to Agent). The feature vectors of all traffic participants in the scene are used as the query (Q) of the attention mechanism, and the rasterized map features after feature extraction are used as the key (K) and value (V). Q, K, and V are all obtained through a 1×1 2D convolution Conv2d. A schematic diagram of the M2A process is shown below. Figure 6 As shown in the diagram. Next, the output of M2A is used to achieve information exchange between agents via A2A (Agent to Agent). The feature vectors of all traffic participants in the scene are used as queries (Q), keys (K), and values ​​(V). The entire process is equivalent to calculating a self-attention mechanism. A schematic diagram of the A2A process is shown below. Figure 7 As shown, by fusing historical motion and high-precision map features through attention mechanism modules such as M2A (Map to Agent) and A2A (Agent to Agent), and explicitly adding relative distance information as a supplement, the feature representation capability is enhanced.

[0080] Considering that relative distance information needs to be explicitly considered after A2A, the relative distance information of the target agent and other traffic participants is passed through a multilayer perceptron and then concatenated into the feature vector of the A2A interaction. Then, the target agent is used as the query (Q), and the other traffic participants are used as the key (K) and value (V) to calculate cross-attention.

[0081] As a preferred embodiment, the above attention mechanisms are all multi-head attention mechanisms, with the number of heads set to 4. The target point prediction network structure also includes modules such as LayerNorm, Multiple Perceptron (MLP), and residual structures, such as... Figure 8 As shown.

[0082] d. Transposed Convolution-Based Decoder: The feature map is upsampled using transposed convolution blocks. Each transposed convolution block contains transposed convolutions, batch normalization (BN), and ReLU layers. The kernel size is 4×4, with padding of 1 and stride of 2 to achieve the upsampling effect. The structure is as follows: Figure 9 As shown. After upsampling to the original rasterized map size, the original high-precision map is stitched onto the feature map to compensate for the information lost during downsampling, such as... Figure 2 The decoder is shown on the right. Finally, the number of channels is reduced to 1 through a convolutional block, and the output of each pixel is constrained to the range [0,1] by the Sigmoid function.

[0083] Define focal loss (pixel-wise focal loss) to train the target point prediction network. Assume the true distribution Y is a two-dimensional Gaussian distribution centered on the true target points, and let the value at position (i,j) on the output heatmap be... The value at position (i,j) in the true distribution is Y. ij The loss function can be expressed as follows:

[0084]

[0085]

[0086] in, The true distribution Y and the output heatmap distribution The loss function; To output the value in the i-th row and j-th column of the heatmap; Y ij β represents the value in the i-th row and j-th column of the true distribution; β is the weighting coefficient; the variance of the two-dimensional Gaussian distribution is set to 4 pixels; H and W are the height and width of the heatmap, respectively. Multiplication term This can solve the hard sample problem and increase the proportion of poorly predicted pixels in the loss. For the true target point location (Y... ij =1), the weighting coefficient β forces the predicted value at this location to be close to the official value. For the remaining locations, the multiplication term (1-Y) ij ) 4 It can reduce Y ij The proportion of loss values ​​close to 1 is used to prevent the model from being over-constrained.

[0087] An auxiliary task is introduced to improve model performance. At the output of each transposed convolutional block of the decoder, an auxiliary transposed convolutional block is used to restore the original image size, converting the number of channels to 1. Then, a heatmap distribution is output using a sigmoid function, and learning is supervised by pixel-wise focal loss. The total loss function includes the output loss after passing through multiple transposed convolutional blocks and the auxiliary loss within the transposed convolutional blocks, as shown in the following equation:

[0088]

[0089] in, is the output of the l-th transposed convolutional block, numbered starting from the closest to the encoder, with a total of 4 transposed convolutional blocks; Y represents the true distribution; To predict the heatmap distribution, α is a hyperparameter used to balance the two losses. It controls the coefficient before the auxiliary loss and decays exponentially to give higher weight to the loss closer to the output.

[0090] (6) The target point prediction network was trained using the Adam optimizer with an initial learning rate of 0.001, a batch size of 32, and a total of 20 epochs. To stabilize training, a warm-up strategy was used to adjust the learning rate in the early stages of training, linearly increasing it from a small learning rate to the initial learning rate, which was set to reach the initial learning rate at the 800th iteration. Furthermore, the learning rate was halved at epochs 3, 6, 9, and 13. For the weight β in the pixel-wise focal loss, it was set to 1000 in the first 16 epochs and 1 in the last 4 epochs. To enhance the robustness of the model, the channel data of the raster map was modified with a probability of 0.2 to achieve data augmentation. The channel data was modified in the following ways: 1) Randomly masked 50% of the pixels in 3 random channels; 2) Added 2D Gaussian noise with a mean of 0.05 and a standard deviation of 0.05 to the 3 random channels; 3) Added salt-and-pepper noise to the 3 random channels; 4) Randomly selected one channel was filled with 0; 5) Randomly selected one channel was filled with a standard Gaussian distributed value. During training, the network with the best prediction metrics was selected as the final network model using the validation set, and its performance was evaluated on the test set.

[0091] (7) After training the target point prediction network, two sampling algorithms were used to sample the distribution heatmap, each considering different metrics: one was the MR sampling algorithm considering the Miss Rate (MR), and the other was the FDE sampling algorithm considering the Final Displacement Error (FDE). The algorithm flowchart is shown in the table below:

[0092] Table 1 MR Sampling Algorithm

[0093]

[0094]

[0095] The MR sampling algorithm iteratively selects the region with the highest probability integral value from a circular region of radius R. After selecting the corresponding region, the probability values ​​of all x within the region are set to 0, and the center of the region is taken as the prediction target point. Setting the probability values ​​of points within the region to 0 effectively ensures the spacing between sampling points and improves trajectory diversity. In the implementation process, to improve computational efficiency, the region is treated as a square region instead of a circular region. The summation of probabilities within the region can be regarded as a convolution operation. Therefore, a convolutional layer with a fixed kernel parameter (fixed to 1) is used to process the heatmap, and a deep learning framework is used to accelerate the computation.

[0096] Table 2 FDE Sampling Algorithm

[0097]

[0098] The FDE sampling algorithm flow is shown in the table above. Its principle is inspired by the KMeans clustering algorithm. Each sampled target point has a neighborhood, and each point within that neighborhood has its own coordinates and probability value. Therefore, the sampling target point can be more accurate by weighting the points within the neighborhood. First, the FDE sampling algorithm uses the results of MR sampling to initialize K center points; then, for each center point, it selects neighborhood points within a 3m range. The process involves... It is an indicator function that, when the condition is met... The function outputs 1 if the value is 1, otherwise it outputs 0. Finally, a weighted sum is calculated for all points in the neighborhood to form the new center point, and the next iteration begins. The weight of each point in the neighborhood is not only related to its probability, but also to its distance from the center point; the greater the distance, the smaller the weight.

[0099] (8) Based on the target point and context information, a trajectory regression network is built to complete trajectory regression, including the following modules:

[0100] a. Information pooling layer: The intermediate layer features of the target point prediction network are flattened into a one-dimensional vector by global average pooling (GAP) and then concatenated with the target point features.

[0101] b. Linear residual block: Consists of fully connected layers, layer normalization layers (LayerNorm), and ReLU activation functions, which perform multi-level transformations on the input dimension.

[0102] The trajectory regression network is trained using Huber Loss, and Huber Loss uses... The loss function can be expressed as:

[0103] in, This indicates the coordinates of the points predicted by the trajectory prediction model, including the x and y coordinates. Represents the actual coordinates; The predicted coordinate x at time t F,t and actual coordinates Huber losses between; The predicted coordinates y at time t F,t and actual coordinates Huber losses between; a total of T pred At each prediction time, variables The Huber loss with respect to variable y is defined as follows:

[0104]

[0105] Where δ is a hyperparameter in the loss.

[0106] The trajectory regression network was trained using the Adam optimizer with an initial learning rate of 0.001, a batch size of 32, for a total of 10 epochs. The learning rate was adjusted to 0.0001 on the 5th epoch. The intermediate layer features of the target point prediction network were used only as input; gradients were not backpropagated to the already trained target point prediction network during training.

[0107] The model was evaluated using evaluation metrics widely used in Argoverse, taking into account both single-modal and multi-modal prediction performance. These metrics primarily include the following:

[0108] Minimum Final Displacement Error (minFDE): The L2 distance between the endpoints of the best predicted trajectory and the endpoints of the true trajectory. In the case of multimodal prediction, the best trajectory is the trajectory with the minimum endpoint error.

[0109] Minimum Average Displacement Error (minADE): The average L2 distance between the best predicted trajectory and the true trajectory. In the case of multimodal prediction, the best trajectory is the trajectory with the minimum endpoint error.

[0110] Miss Rate (MR): The number of scenarios where the predicted trajectory is not within 2 meters of the actual trajectory, based on endpoint error.

[0111] p-minADE: Similar to minADE, but the difference is that it takes into account the influence of probability. It adds min(-log(p),-log(0.05)) to minADE, where p corresponds to the probability of the best predicted trajectory.

[0112] p-minFDE: Similar to minFDE, but the difference is that it takes into account the influence of probability. It adds min(-log(p),-log(0.05)) to minFDE, where p corresponds to the probability of the best predicted trajectory.

[0113] 1. Quantitative comparison of prediction methods

[0114] Based on the above metrics, the proposed method will be compared with the following baseline models on the Argoverse test set. Following the conventions of the Argoverse dataset, the experiments will include predictions for both unimodal K=1 and multimodal K=6.

[0115] Argo-CV: A physics prediction model that uses observations in the time domain T. obs The coordinates within the range are used to calculate different average speeds, assuming that the vehicle travels at a constant speed within the prediction time domain.

[0116] Argo-NN+map: A weighted nearest neighbor regression model that obtains future trajectories by querying nearest neighbors based on vectorized map information.

[0117] Argo-LSTM+map: A prediction model based on an LSTM encoder-decoder architecture, with input including motion history data and vectorized map information.

[0118] uulm-mrm: A raster map-based method that extracts map features through a CNN backbone network and stitches together state features. It directly generates multimodal trajectories and their confidence scores through a regression head and trains using a winner-take-all loss.

[0119] MotionCNN: A raster map-based approach that explicitly incorporates motion history into the raster map and directly generates multimodal trajectories and their confidence scores through a pre-trained CNN backbone network.

[0120] TNT: A phased trajectory prediction method that integrates target point prediction and trajectory regression within the same network to generate multimodal trajectories in an end-to-end manner.

[0121] Multi-Fused Heatmap Output for Motion Prediction (MF-Heat): This invention proposes a multimodal vehicle trajectory prediction method based on target-driven and distributed heatmap output.

[0122] Table 3 compares the results on the Argoverse test set (K=1). MR is expressed as a percentage, with bold indicating the best result.

[0123]

[0124]

[0125] Table 4 compares the results of the Argoverse test set (K=6), '-' indicates that no relevant results were found.

[0126] method minADE minFDE p-minADE p-minFDE MR Argo-CV 3.39 7.57 5.18 9.36 81.68 Argo-NN+map 2.08 4.03 3.87 5.82 58.21 Argo-LSTM+map 2.34 5.44 4.14 7.23 69.16 TNT 0.94 1.54 2.73 3.33 13.30 uulm-mrm 0.94 1.55 － － 21.79 MotionCNN 1.00 1.60 2.67 3.26 21.75 MF-Heat(FDE) 0.90 1.36 2.76 3.22 13.15 MF-Heat(MR) 0.93 1.48 2.47 3.02 11.32

[0127] The experimental results are shown in Tables 3 and 4. As can be seen from the tables, the MF-Heat method performs exceptionally well when K=1, outperforming similar methods and even surpassing the TNT method, which is also target-driven. When K=6, MF-Heat using the MR sampling algorithm achieves the best performance in the probability-considered p-minADE and p-minFDE metrics, indicating that the high-probability trajectories predicted by MF-Heat generally have smaller displacement errors. Furthermore, it also outperforms similar methods in minADE. MR is an important metric for evaluating prediction performance, reflecting the trajectory coverage; a lower MR means that the predicted trajectory better covers the real scene. The MF-Heat method using MR sampling performs exceptionally well in the MR metric, significantly outperforming the other comparative methods.

[0128] The MF-Heat method for FDE sampling sacrifices MR for a lower minFDE. In the experiment, the iteration parameter L=4 was set. The results showed that the method achieved the best performance in minFDE, which significantly outperformed similar methods. It increased MR by about 2%, but still outperformed the other comparison methods, indicating that the sacrifice of scene coverage was small.

[0129] 2. Ablation experiment

[0130] To analyze the impact of different modules and hyperparameter settings on MF-Heat, ablation experiments were conducted, and the results were analyzed on the validation set. To analyze the influence of different components in the network architecture, MF-Heat was decomposed into modules, and each module was evaluated. Furthermore, the impact of the backbone network extracting raster map features on the results was explored, with comparisons made by replacing the Swin Transformer with ResNet50. In the loss calculation module, both the hyperparameter β and the auxiliary loss affect model training, and their impacts were evaluated separately.

[0131] 1) Module ablation

[0132] Table 5 shows the ablation experimental results for different modules. '-' indicates that the model did not converge.

[0133]

[0134] Ablation was performed on different modules, and the heatmap was sampled using the MR algorithm. The results are shown in Table 5. Without accessing the rasterized high-precision map, neither the bidirectional LSTM nor the U-LSTM model converged, indicating that the target distribution heatmap strongly depends on the rasterized map. When encoding the high-precision map and motion history separately, the U-LSTM method performed better than the bidirectional LSTM method, demonstrating the effectiveness of the U-LSTM module in extracting motion history features. Adding the vehicle-to-vehicle attention module A2A to U-LSTM and Map did not significantly change the model performance, indicating that the prediction results are not strongly dependent on vehicle-to-vehicle interactions and are related to the scene distribution of the dataset. Adding the attention module M2A significantly improved the model performance, indicating that the proxy features were enhanced through the map, proving the necessity of map-proxy information interaction. Among all modules, the absence of the M2A or A2A module significantly reduced the model performance. Stacking attention mechanism modules can enhance the model's representational ability, but they must coordinate with each other to achieve the best results.

[0135] 2) Backbone Network

[0136] Table 6. Experimental results under different backbone networks

[0137]

[0138] The performance of different backbone networks in target point prediction is compared, and the results are shown in Table 6. ResNet50 still lags significantly behind Swin Transformer in target point prediction, further illustrating the superiority of Swin Transformer in raster map feature extraction.

[0139] 3) The effect of β in the loss function

[0140] For the true location of the target point (i) target ,j target When calculating the loss, a coefficient β needs to be multiplied. To study the impact of β value on the results, three value methods were set for comparison: I) β = 1000; II) β = 1; III) β = 1000 in the early stage of training and 1 in the later stage of training. The experimental results are shown in Table 7.

[0141] Table 7 Experimental results for different β values

[0142]

[0143] The results show that when β is 1000, the model mainly focuses on the location of the true target point, resulting in concentrated but inaccurate predictions, with excessive penalty applied to the true location. When β is 1, the model's performance improves somewhat, but it is still not as good as the result of initially setting β to 1000 and then to 1. Setting β to 1000 initially is equivalent to imposing a priori assumption about the target point's location on the model, focusing on the region of the true target point rather than individual points in the early stages of training. In the later stages of training, due to the excessive penalty of β, the high-probability region becomes larger, so β ​​needs to be adjusted to reduce the probability of certain locations, allowing the sampling algorithm to sample the ideal results.

[0144] 4) Impact of auxiliary losses

[0145] Table 8 Results of Auxiliary Loss Ablation Experiment

[0146]

[0147] To evaluate the effect of the auxiliary loss, ablation experiments were conducted, and Table 8 shows a comparison of the results with and without the auxiliary loss. As can be seen from the table, all metrics improved after adding the auxiliary loss. This indicates that the auxiliary loss can effectively shorten the distance between features in each layer of the decoder, thereby improving the overall performance of the model.

[0148] 3. Qualitative Analysis of Test Set Results

[0149] A visualization analysis was performed on the results of the Argoverse test set. Since real trajectories were unavailable in the test set, only a qualitative rationalization analysis of the predicted trajectories was conducted. The results are as follows: Figure 10 As shown in a and b.

[0150] As shown in the figure, the multimodal predicted trajectories on the test set generally conform to human common sense and satisfy the constraints of vehicle dynamics and road rules. Whether on a straight road or at a curved intersection, the method proposed in this invention effectively covers reasonable driving behaviors. For example, when the predicted target is at an intersection, depending on the intention, it may exhibit either a straight-going or turning behavior. For similar multimodal behaviors, MF-Heat produced reasonable predictions and assigned different confidence levels.

[0151] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-modal vehicle trajectory prediction method that targets driving and distributing heatmaps output, characterized in that, The method comprises the following steps: (1) Select a vehicle around the autonomous vehicle as the prediction subject and collect the trajectory data of the prediction subject. Collect the driving trajectory of the surrounding vehicle through the target tracking module and record the type of each traffic participant. Divide the collected trajectory into segments of length T according to the timestamp. obs The historical trajectory and length are T pred The future trajectory of the subject being predicted is used as the true label for the prediction task. (2) data cleaning is performed on the collected trajectory, a corresponding prediction scenario is mined, and the collected trajectory is divided into a training set, a verification set and a test set; (3) pre-processing the driving trajectory and high-precision map data: the information of the prediction subject and surrounding vehicles is represented into a motion vector, and the high-precision map information is represented into a specific size grid map as road information known by the trajectory prediction task; (4) the trajectory prediction task is divided into two stages of target point prediction and trajectory regression to correspond to the motivation uncertainty and motion uncertainty in the real driving process; (5) a target point prediction network is built to predict the end point of the future trajectory, and a loss function is defined to guide the training of the target point prediction network; Building the target point prediction network and training includes the following steps: 1) assuming that the target point distribution obeys a two-dimensional discrete distribution, fitting is performed by using a deep neural network, and the network output result is called a distribution heat map; a target point prediction network is built based on an encoder-decoder architecture: a. motion feature encoder based on U-LSTM: the trajectory history data is first fused through a one-dimensional convolution block, and then fused again through a U-LSTM to output a feature vector of a specific dimension; b. map feature encoder based on Swin Transformer: the rasterized map features are extracted by using a Swin Transformer to obtain a larger receptive field; c. feature fusion module based on a multi-stage attention mechanism: the historical motion and high-precision map features are fused through an attention mechanism, and relative distance information is explicitly added as a supplement to enhance the feature expression capability; d. decoder based on transposed convolution: the feature map is up-sampled through a transposed convolution block, and after up-sampling to the original rasterized map size, the original high-precision map is spliced to the feature map to compensate for the information lost in the down-sampling process, and finally the channel number is reduced to 1 through a convolution block, and each pixel output is constrained in the range of [0, 1] through a Sigmoid function; 2) Network training by pixel-by-pixel focal loss, the real distribution Y is a two-dimensional Gaussian distribution centered on the real target point, and the value of the (i,j) position on the output heat map is The value of the (i,j) position on the real distribution is Y ij The loss function is represented as follows: wherein, is a loss function for the real distribution Y and the output heat map distribution is the value of the i-th row and j-th column of the output heat map; Y ij is the value of the i-th row and j-th column on the real distribution; H, W are the height and width of the heat map, respectively; β is a weight coefficient.​ 3) the target point prediction network is trained using the data processed in step (3), the hyperparameters and training strategies are set, the best model network with the best training effect is selected through the verification set, and the performance is evaluated in the test set; (6) after the training of the target point prediction network is completed, different strategies are used to obtain dissimilar multi-modal target points from the target point prediction network; (7) based on the target point and context information, a trajectory regression network is built to complete trajectory regression.

2. The multi-modal vehicle trajectory prediction method of claim 1, wherein, For each prediction scenario, the position of all trajectory data is converted into a coordinate system with the prediction subject as the origin and the orientation of the last observation time of the prediction subject as the x-axis.

3. The multi-modal vehicle trajectory prediction method of claim 1, wherein, The motion vector comprises a type of traffic participant, historical trajectory coordinates and a time interval; the time interval is represented by a difference between a current time and a previous time; the rasterized map comprises lane center line data, lane boundaries and drivable area information, wherein the lane center line data occupies three channels and is represented by HSV coding, H is a hue, the value range of which is 0°-360°; S is a saturation, the value range of which is 0-1; and V is a lightness; the lane boundary and drivable area data occupy one channel, if there is data at a certain position, the pixel value of the corresponding position is set to 255, otherwise, it is 0.

4. The multi-modal vehicle trajectory prediction method of claim 1, wherein, On the output of each transposed convolution block of the decoder, the original image size is restored through an auxiliary transposed convolution block, the channel number is converted to 1, then the Sigmoid function is used to output the heat map distribution, and the total loss function will contain the output loss after the multi-layer transposed convolution block and the auxiliary loss in the transposed convolution block, as shown in the following formula: wherein, is the output of the l-th transpose convolutional block, and a is a hyperparameter used to balance the two losses, which is used to control the coefficient before the auxiliary loss.

5. The multi-modal vehicle trajectory prediction method driven by objectives and distribution heatmaps output of claim 1, wherein, The U-LSTM-based motion feature encoder is an improved long short-term memory network, which first performs a reverse LSTM process to obtain future information, and then transmits the output to a forward LSTM.

6. The multi-modal vehicle trajectory prediction method of claim 1, wherein, The Swin Transformer-based map feature encoder comprises three stages, each stage performing one down-sampling on an image, and each stage comprising two Swin Transformer blocks.

7. The multi-modal vehicle trajectory prediction method of claim 1, wherein, An MR sampling algorithm and an FDE sampling algorithm are used to obtain dissimilar multi-modal target points from the target point prediction network; The MR sampling algorithm comprises the following steps: Regions with maximum probability integral values are iteratively selected, after selecting the corresponding regions, the probability values of all positions in the regions are set to 0, and the center of the region is taken as a predicted target point, and the process is repeated K times, so that K target points, i.e., prediction modes, are obtained; The FDE sampling algorithm uses the K target points obtained by the MR sampling algorithm as initial target points, and iteratively optimizes to obtain multi-modal target points, mainly comprising the following steps: Compute the position x on the heat map i and the distance to the kth center point where subscript i denotes the ith position;​ Assume position x i The nearest center point is Calculate the distance m i of each point x from the nearest center point i ; For the k-th center point Filter out neighborhood points within a meter range, perform a weighted average to obtain the new center point coordinates, and proceed to the next iteration. The weights of the weighted average are: and The product of p, where p i For position x i Probability on a heatmap; The above process is repeated L times, so that more accurate sampling target points are obtained.

8. The multi-modal vehicle trajectory prediction method driven by objectives and distribution heatmaps output of any one of claims 1-7, wherein, The trajectory regression network comprises the following modules: The fusion information pooling layer: the intermediate layer features of the target point prediction network are flattened into one-dimensional vectors by global average pooling, and are spliced with target point features, wherein the intermediate layer features of the target point prediction network are output feature maps after the scene input passes through the U-LSTM, Swin Transformer and multi-stage attention mechanism, and contain rich road structure information; The linear residual block: comprising a fully connected layer, a layer normalization layer and a ReLU activation function, which performs multi-level transformation on the input dimension.

Images