A trajectory target point prediction method based on knowledge distillation

Abstract

The application discloses a trajectory target point prediction method based on knowledge distillation, comprising the following steps: constructing a trajectory data set containing high-precision map data; building a teacher model with high prediction accuracy; pre-training the teacher model by using the trajectory data set; freezing the parameters of the trained teacher model; building a student model with similar structure but smaller parameter quantity based on the teacher model; designing the supervision loss of the student model to make the output close to the real target point; using the teacher model to perform knowledge distillation on the student model, mainly including: smoothing output distillation, attention map distillation and fusion feature map channel distribution distillation; setting appropriate hyperparameters and training strategies to train the student model. The trajectory target point prediction method based on knowledge distillation is applied to vehicle trajectory prediction for the first time, effectively reduces the model parameter quantity and improves the inference speed without greatly reducing the accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent driving perception, and relates to a trajectory prediction method, in particular to a trajectory target point prediction method based on knowledge distillation. BACKGROUND

[0002] The automatic driving technology has great significance for improving traffic safety and efficiency, promoting the innovation and upgrading of the automobile industry, and meeting the travel needs of consumers. In order to drive efficiently and safely on the road, the automatic driving car needs to actively predict the future state of the surrounding traffic participants to make further decisions. The prediction module, as a post-processing link of the environment perception module, plays an important role in the safe decision planning and motion control of the automatic driving car.

[0003] The predicted trajectory is essentially multi-modal. The car driving to the intersection may contain multiple behaviors such as straight driving, turning, lane changing, etc. due to the influence of environment, traffic participant type, motion state, intention, etc. The trajectory prediction problem aims to generate possible multi-modal trajectories in the future according to historical motion and high-precision map information. In order to effectively model information interaction to cover future multi-modal, the prediction network usually has a large number of parameters, which is challenging for real-time application and commercial deployment. In addition, the inference speed is particularly important for automatic driving, because it is related to the safety of the car, therefore, it is necessary to study network compression and knowledge distillation and other network lightweight technologies.

[0004] Currently, knowledge distillation is less applied in trajectory prediction, and most existing methods do not consider model size and inference speed, which can significantly affect system response, so advanced lightweight technology is needed to reduce the size without losing too much accuracy, making it easier to deploy and improve security. Model lightweight is the development trend of future deep learning. From a commercial point of view, many models are currently limited by their size and cannot be deployed on mobile devices, making the use of deep learning have certain threshold. Research on lightweight prediction models has great practical significance for the landing application of autonomous vehicles. On the one hand, it is beneficial to improve driving safety and ensure that downstream modules respond quickly to the scene; on the other hand, it is beneficial to reduce the system's computing power requirements and reduce production costs. Monti et al. (How many Observations are Enough?Knowledge Distillation for Trajectory Forecasting) distill the input data, which is motivated by the fact that the results of perception algorithms are not perfect during the collection of input trajectories. Incorrect detection results can accumulate in crowded scenes, leading to tracking drift, so consider transferring knowledge from the teacher network to the student network with shorter observation sequences. Su et al. (Narrowing the Coordinate-frame Gap in Behavior Prediction Models: Distillation for Efficient and Accurate Scene-centric Motion Forecasting) apply knowledge distillation technology to narrow the performance gap between agent-centric and scene-centric models, which helps to infer the motion of all agents at once without the need for individual scene normalization for each agent. SUMMARY

[0005] To solve the above technical problems, the present application provides a trajectory target point prediction method based on knowledge distillation, which can solve the problems of large model deployment difficulty and long inference time in the field of trajectory prediction, and improve the prediction performance of the student model. In addition, the proposed knowledge distillation method is universal and can be applied to the distillation of other large-scale trajectory prediction models, and has strong practical application value.

[0006] The present application is realized at least by one of the following technical solutions.

[0007] A trajectory target point prediction method based on knowledge distillation, comprising the following steps:

[0008] (1) Collect trajectory data and high-precision map data, construct a trajectory prediction dataset, and divide the dataset into training, validation and test sets;

[0009] (2) building a target point prediction teacher model, pre-training the target point prediction teacher model by using the trajectory dataset;

[0010] (3) adjusting the structure of the teacher model from the model architecture and network layer parameter configuration, taking the adjusted teacher model as the student model to be distilled;

[0011] (4) defining the supervision loss of the student model according to the real label, so that the model prediction is close to the real label;

[0012] (5) using the teacher model to perform knowledge distillation on the student model, including:

[0013] a. Smoothed output distillation: using the output of the target point prediction teacher model to guide the learning of the student model;

[0014] b. Attention map distillation: using the attention map of the target point prediction teacher model to guide the learning of the student model, the degree of interaction between features is represented by the attention map, and the attention map is calculated by the dot product between vectors;

[0015] c. Fusion feature map channel distribution distillation: adjusting and aligning the feature maps of the target point prediction teacher model and the student model in the channel dimension;

[0016] (6) combining the supervision loss and the distillation loss to define the total loss for training the student model;

[0017] (7) setting a training strategy, training the student model with smaller parameter quantity by means of the pre-trained target point prediction teacher model, so that the target point prediction accuracy of the student model reaches a similar effect to that of the target point prediction teacher model, while improving the inference speed;

[0018] (8) sampling the target points of the trained student model, taking the sampled multiple target points as the target points of multiple predicted trajectories in trajectory prediction, to obtain a multi-modal prediction result.

[0019] Further, the high-precision map data is represented in a rasterized form, and the lane centerline is encoded into multiple image channels, and the drivable area and the car-to-boundary line are respectively encoded into one or more image channels.

[0020] Further, the trajectory prediction dataset includes trajectory data of multiple traffic entities, and according to the constructed trajectory prediction dataset, the prediction task is defined as: for a certain entity in an autonomous driving scene, using a historical trajectory with a length of T obs to predict the length of T predThe future trajectory is divided into two stages: target point prediction and trajectory regression. The target point prediction stage first predicts the target points of multiple future trajectories, and the trajectory regression stage completes the trajectory based on the predicted target points.

[0021] Furthermore, the target point prediction teacher model includes a feature encoder, a multi-source fusion module, and a feature decoder. The feature encoder extracts features from historical trajectory data and high-precision map data, and projects the original vectors into the feature space. The multi-source fusion module fuses the features of traffic entities in the scene and the high-precision map through an attention mechanism. The feature decoder decodes the fused features output by the multi-source fusion module to obtain a two-dimensional distribution of target points for multiple predicted trajectories of the predicted entities. The two-dimensional distribution is represented by a raster image with a size of H×W, and the value of each pixel represents the probability that the target point is located at that position.

[0022] Furthermore, the methods for adjusting the target point prediction teacher model to a student model in step (3) include: reducing the feature dimension, reducing the network layer parameters, and replacing the network structure.

[0023] Furthermore, in the smoothing output distillation of step (5), the output y of the teacher model for predicting the target point is... T Smoothing is performed as shown in the following formula:

[0024]

[0025] Among them, y T The distribution heatmap of the teacher model output for predicting target points; y GT For a true distribution heatmap, assuming the actual target point is located in row i* and column j*, then y GT It is a two-dimensional Gaussian distribution centered at (i*,j*). This is the smoothed heatmap of the distribution.

[0026] Assign the output of the target point prediction teacher model a value of 1, that is, set the value of the two-dimensional Gaussian distribution position (i*,j*) to 1. Let y be 1. S This represents the heatmap of the student model output, obtained by optimizing the distribution y. S With distribution The KL divergence is used to realize knowledge transfer from the teacher model to the student model for target point prediction, as shown in the following formula:

[0027]

[0028] in, for and KL divergence; and are the distribution probability values of the i-th position in the distribution heat map of the smooth distribution and the student model output respectively; H and W represent the height and width of the output heat map;

[0029] Optimizing the KL divergence is equivalent to optimizing the cross-entropy, and the smoothing output distillation loss is expressed as:

[0030]

[0031] wherein H and W represent the height and width of the output heat map respectively.

[0032] Further, the attention map distillation of step (5) adopts a mean square error loss as the distillation target:

[0033]

[0034] wherein, is the attention map distillation loss; H' and W' represent the height and width of the attention map respectively; and are the values of the h-th row and w-th column in the attention map of the target point prediction teacher model and student model respectively, obtained by calculating the dot product of the h-th feature and the w-th feature, and then applying a softmax function in the w-th dimension, the features being features obtained by a feature encoder from high-precision map data or features obtained by a feature encoder from traffic entity historical trajectories.

[0035] Further, in the fusion feature map channel distribution distillation of step (5), the historical trajectory data and the high-precision map data will output a feature map with a size of H" x W" and a channel number of C after passing through the feature encoders and the multi-source fusion module of the teacher / student model. First, the activation values of the channels are converted into probability distributions in order to measure the difference between the distributions using a probability distance metric, and a softmax function with temperature is used to convert them into probability distributions, as follows:

[0036]

[0037] wherein, is the softmax function with temperature; e c,i is the value of the i-th position in the c-th channel of the fusion feature map; τ e is a temperature hyperparameter, and let and represent the feature maps of the c-th channel of the fusion feature map of the target point prediction teacher model and student model respectively, and the distillation loss of the fusion feature map is represented by the KL divergence between the teacher channel distribution and the student channel distribution , as follows:

[0038]

[0039] wherein, denotes and the fusion feature map channel distribution distillation loss.

[0040] Further, in step (6), the total loss of the student model is represented as the sum of the supervision loss and the three-part distillation loss, as follows:

[0041]

[0042] wherein, is the supervision loss; denotes the smoothed output distillation loss, denotes the attention map distillation loss, denotes the fusion feature map channel distribution distillation loss, and the coefficients λ, η and γ in the distillation loss are hyperparameters for balancing the loss.

[0043] Further, in step (8), target point sampling is performed by an MR sampling algorithm.

[0044] The trajectory target point prediction method based on knowledge distillation has the following beneficial effects compared with the prior art:

[0045] 1. The trajectory target point prediction method based on knowledge distillation is the first time to apply knowledge distillation technology to vehicle trajectory prediction, mainly aiming at the target point prediction network based on the encoder-decoder structure, effectively realizing the reduction of model parameter quantity and the improvement of inference speed without significantly reducing the accuracy.

[0046] 2. The trajectory target point prediction method based on knowledge distillation distills the knowledge of the pre-trained teacher model to the student model with smaller parameter quantity, to improve the prediction accuracy of the student model, realize the commercial deployment of small models replacing large models, reduce the calculation cost, and provide protection for the landing application of trajectory prediction models. In addition, the proposed knowledge distillation method is universal and can be extended to the distillation of other large-scale trajectory prediction models, and has strong practical application value.

[0047] 3. The smoothed output distillation method proposed in the present application uses the output of the teacher model to guide the learning of the student model, because the output of the teacher model not only contains the real target point distribution, but also contains the multi-modal distribution, which is more conducive to the learning of the student model. At the same time, the teacher model output is smoothed in combination with the real distribution, which improves the performance of multi-modal prediction and makes the model obtain improvement in all indicators.

[0048] 4. The attention map distillation method proposed in this invention learns the relational features between vector pairs through the MSE loss between attention maps. Attention maps are calculated from the dot product of vectors and represent the degree of correlation between features. Adding attention map distillation loss helps the teacher model convey vector correlation information and improves model performance.

[0049] 5. The channel distribution distillation method for fused feature maps proposed in this invention first converts the activation values ​​of the channels into probability distributions so that the differences in distributions can be measured using a probabilistic distance metric. Motion history information and high-precision map information are often embedded in the encoded fused features. To better transfer this knowledge to the student model, the feature maps of the teacher and student models are adjusted and aligned along the channel dimension. Considering the impact of intermediate layer distillation, the fused feature map has a greater influence on the results, bringing more gains to the model.

[0050] 6. The trajectory target point prediction method based on knowledge distillation proposed in this invention has a strong correlation between the output and the intermediate layer features. Therefore, the impact of distillation loss needs to be considered holistically; considering only a part of the loss would cause the model to lose its original capabilities. This invention achieves complementary advantages of distillation loss, thus maximizing its effectiveness. Attached Figure Description

[0051] Figure 1 This is an architecture diagram of a trajectory target point prediction method based on knowledge distillation according to the present invention;

[0052] Figure 2 This is a diagram of the teacher network architecture according to an embodiment of the present invention.

[0053] Figure 3 This is a schematic diagram of the output distillation of a trajectory target point prediction method based on knowledge distillation according to the present invention;

[0054] Figure 4 This is a schematic diagram of attention-map distillation in the method of the present invention;

[0055] Figure 5 This is a schematic diagram of distillation, representing a feature of the method of the present invention.

[0056] Figure 6 The temperature τ in this embodiment of the invention e A schematic diagram illustrating the impact of loss weight γ on the prediction results;

[0057] Figure 7 This is a comparison chart of the student model before and after distillation in an embodiment of the present invention. Detailed Implementation

[0058] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and specific implementation steps. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0059] As shown in a trajectory target point prediction method based on knowledge distillation, the specific implementation steps include the following: Figure 1

[0060] (1) Collect trajectory data containing high-precision map information, construct a trajectory prediction dataset, and divide the dataset into training, validation and test sets. According to the constructed trajectory dataset, the prediction task is defined as: for a certain entity in an autonomous driving scene, using a historical trajectory with a length of T obs , the position of the entity after T pred in the future is predicted.

[0061] As an embodiment, the high-precision map information data is represented in a rasterized form, the rasterized map size is 288x288, the resolution is 0.5 meters / pixel, the lane center line is encoded into 3 image channels, the drivable area and the car-to-boundary line are encoded into 1 image channel, and there are a total of 5 channels. The trajectory prediction dataset includes trajectory data of various traffic entities (such as pedestrians, vehicles, etc.), according to the constructed trajectory prediction dataset, the prediction task is defined as: for a certain entity in an autonomous driving scene, using a historical trajectory with a length of T obs , the future trajectory with a length of T pred of the entity is predicted, due to the diversity of the future, multiple future trajectories and their probabilities need to be predicted. Further, the trajectory prediction task can be divided into two stages of target point prediction and trajectory regression, the target point prediction stage first predicts the target points of multiple future trajectories, and the trajectory regression stage completes trajectory completion based on the predicted target points. The target point refers to the last point in the future trajectory with a length of T pred , the present application only performs knowledge distillation on the target point prediction stage to obtain a student model with smaller parameter quantity, and improves the reasoning speed.

[0062] The implementation process of the present application can also be described directly using the trajectory data of a large public trajectory prediction dataset Argoverse as an example. The Argoverse dataset is mainly collected from Pittsburgh and Miami, and contains trajectory and rich semantic map data of more than 300,000 target agents, the semantic map covers up to 290 kilometers, and finally 324,557 5-second sequences are collected, and the sampling frequency is fixed at 10Hz. For the collected 5-second sequences, the trajectory of the history 2 seconds and the trajectory of the future 3 seconds are divided, that is, T obs = 2, T pred ​= 3. To evaluate the performance of the multi-modal prediction, a single scene will consider K output trajectories, and the present application takes K = 6 as an example, which also corresponds to the standard of Argoverse. For 324,557 sequences, 205,942 are divided into the training set, 39,472 are divided into the validation set, and 78,143 are divided into the test set. In order to ensure the diversity of the data, the training, validation and test sets are taken from non-intersecting parts of the city, that is, about 1 / 8 and 1 / 4 of each city are taken as validation and test data respectively.

[0063] (2) Build a target point prediction teacher model and pre-train it using the trajectory data set; the target point prediction teacher model includes a feature encoder, a multi-source fusion module and a feature decoder;

[0064] As an embodiment, a feasible target point prediction teacher model is shown in Figure 2 , which is called MF-Heat. First, the historical motion data with a length of T obs is obtained after passing through the motion feature encoder, and the motion feature encoder is composed of a double-layer long short-term memory network (LSTM). The rasterized map is obtained after passing through the convolutional neural network (CNN), and the feature map has a channel number of 512 and a size of 18x18. The CNN used is a residual network ResNet50. The multi-source fusion module is composed of two-stage attention mechanism. The first stage takes all entity features as queries, rasterized map features as keys and values, and calculates the attention mechanism. The second stage takes all entities as queries, keys and values, and calculates the attention mechanism, which is equivalent to a self-attention mechanism. Finally, the target point distribution heat map is obtained through the decoder structure based on the convolution kernel size of 4x4 transpose convolution block. The distribution is a two-dimensional distribution, and the size is 288x288. The value of each pixel represents the probability of the target point being located at the current position. The teacher model is pre-trained by pixel-wise (PW) supervision loss, which is calculated by the following formula:

[0065]

[0066]

[0067] wherein, is the pixel-wise supervision loss between the real distribution heat map and the predicted distribution heat map Y; H and W are the height and width of the heat map respectively; Y is a two-dimensional Gaussian distribution centered on the real target point; Y ij is the value of the real distribution at position (i,j); to output the value of the (i, j) position on the heat map; β is the positive sample weight coefficient, The function symbol is represented.

[0068] After the teacher model is trained, the parameters are frozen and are not updated during the student model training stage.

[0069] (3) Adjust the structure of the teacher model from the aspects of network architecture and network layer parameter configuration, etc., to reduce the number of model parameters, such as reducing the dimension of feature extraction from 128 to 64; reducing the number of network layers, such as reducing the number of convolution layers; reducing the network layer parameters, such as reducing the size of the convolution kernel in the convolution layer; replacing the network structure, such as replacing the Long Short-Term Memory (LSTM) with the Gate Recurrent Unit (GRU) which has fewer parameters. In actual application, the above strategies can be adjusted according to specific needs. The adjusted teacher model is used as the student model to be distilled, named MF_tiny-Heat.

[0070] The specific adjustment method is: replacing the map encoder with a more lightweight residual network ResNet34; in the motion encoder part, replacing the bidirectional LSTM with a unidirectional LSTM to achieve faster inference; in the decoder part, reducing the volume of the decoder, and reducing the convolution kernel used in the transpose convolution from 4x4 to 2x2 to achieve the equivalent effect of twice upsampling.

[0071] (4) Define the supervision loss of the student model, which is the pixel-wise supervision loss mentioned above, and the positive sample coefficient β is set to 1000. The student model is supervised and trained using the real label, and the supervision loss is defined to make the model prediction close to the real label.

[0072] (5) Use the teacher model to distill the knowledge of the student model, mainly from the following aspects:

[0073] a. Smoothed output distillation: use the output of the teacher model to guide the learning of the student model, because the teacher model not only contains the real target point distribution, but also contains the multi-modal distribution (such as in a crossroad, the distribution may cover straight and turning). Effective use of the output of the teacher model can give the student model stronger priori and improve its effect.

[0074] In the smoothed output distillation, the teacher model output is y T , obtained by a temperature Sigmoid function, the student model output distribution heat map is y S , and the real distribution heat map is y GT , and the output distillation process is as follows Figure 3The teacher model (network) outputs a teacher distribution heatmap y T Meanwhile, the gradient is canceled back, and the parameters of the teacher model are not updated. In order to let the student model learn the multi-modal nature of the scene, the distance between the teacher distribution y S and the student distribution y T needs to be narrowed, but the influence of the real distribution y GT also needs to be considered, so the output y T of the teacher model is smoothed as follows:

[0075]

[0076] Assuming that the real target point is located at the i*row and j*column, then y GT is a two-dimensional Gaussian distribution centered at (i*, j*), and the variance of the distribution is 4 pixels; is the smoothed distribution heatmap;

[0077] The output of the teacher model after training, although the relative size of the probability between pixels is close to the real distribution, the probability value is generally small, in order to highlight the position of the real target point (i*, j*), the output of the corresponding position of the teacher model is assigned a value of 1, that is, let In order to make the student distribution y S close to the smoothed distribution , the KL divergence between y S and y can be optimized, and the loss is as follows:

[0078]

[0079] Since is a known distribution, its entropy is a constant, so optimizing the KL divergence is equivalent to optimizing the cross-entropy, and the loss can be expressed as:

[0080]

[0081] where H and W represent the height and width of the output heatmap, is the KL divergence between y and y ; and are the distribution probability values of the i-th position in the smoothed distribution heatmap and the distribution heatmap output by the student model, respectively;

[0082] b.Attention map distillation: The degree of interaction between features can be represented by an attention map, which is calculated by the dot product of feature vectors. The feature vectors can be the vectors obtained after feature extraction of historical trajectories or the vectors of each position in the feature map obtained after CNN processing of high-precision map data. The attention map output by the intermediate layer of the teacher model contains useful information that can guide the student model to learn feature correlation information. The attention map distillation process is shown in FIG. 8. Figure 4 The teacher / student encoder is a feature encoder of the teacher / student model and has a feature extraction function. The feature vectors obtained by the encoder are input into the multi-source fusion module, which is composed of a two-stage attention mechanism. The attention mechanism obtains an attention map through dot product and softmax function. In attention map distillation, in order to measure the similarity of the attention maps between the teacher model and the student model, the mean square error (MSE) loss is used as the distillation target:

[0083]

[0084] wherein, represents the attention map distillation loss; H' and W' represent the height and width of the attention map, respectively; and are the values of the hth row and wth column in the attention map of the target point prediction teacher model and student model, respectively. The values are obtained by calculating the dot product of the hth feature and the wth feature, and then applying the softmax function in the wth dimension. The features are the features obtained after the high-precision map data is processed by the feature encoder, or the features obtained after the historical trajectories of traffic entities are processed by the feature encoder.

[0085] c.Fusion feature map channel distribution distillation: Motion history information and high-precision map information are often contained in the encoded fusion features. In order to better transfer this part of knowledge to the student model, the feature maps of the teacher model and the student model can be adjusted and aligned in the channel dimension, and the distillation architecture is shown in FIG. 9. Figure 5

[0086] In the fusion feature map channel distribution distillation, the activation values of the channels are first converted into probability distributions, so that the probability distance metric (such as KL divergence) can be used to measure the difference between the distributions. The probability distribution is converted by a softmax function with temperature, as shown below:

[0087]

[0088] wherein, is the softmax function with temperature, e c,i ​is the feature vector of the fused feature map; H” and W” are the height and width of the feature map respectively; c = 1, 2, …, is the index of the channel; τ e is a temperature hyperparameter, a larger temperature will make the distribution more stable, in the embodiment, τ e = 3.

[0089] By applying the softmax function for channel normalization, the influence of the magnitude between the teacher network and the student network is eliminated. Let and respectively represent the feature map of the c-th channel of the fused feature map of the target point prediction teacher model and the student model, and the KL divergence of the teacher channel distribution and the student channel distribution is used to represent the distillation loss of the fused feature map, as shown in the following formula:

[0090]

[0091] The KL divergence is an asymmetric quantity, that is, D KL (p||q) ≠ D KL (q||p). From the above formula, if is large, should be as large as to minimize the KL divergence. However, if is very small, the KL divergence does not pay much attention to the influence of . Therefore, the student model tends to learn the areas with larger activation values in the teacher model, and the areas with smaller activation values have less influence on the student model, that is, in trajectory prediction, the above formula helps the student model to learn the factors that have a greater impact on the predicted target, and the asymmetry of the KL divergence is beneficial to the distillation learning of the dense prediction task.

[0092] (6) Combine the supervision loss and the distillation loss to define the total loss of the model. The total loss of the model can be represented as the sum of the supervision loss and the three parts of the distillation loss, as shown below:

[0093]

[0094] wherein the supervision loss is a pixel-wise focal loss; the coefficients λ, η and γ in the distillation loss are respectively hyperparameters for balancing the loss, and they are usually set to specific values to make the loss in the same order of magnitude. In the present application, λ = 15, η = 30, and γ = 3.

[0095] (7) Set appropriate hyperparameters and training strategies for model training, and use the validation set to select the best-performing model as the final application model.

[0096] As a preferred embodiment, the model training batch size is set to 32, trained by the Adam optimizer, the initial learning rate is 0.001, the learning rate is adjusted to 0.0001 at the 5th epoch, and a total of 10 epochs are trained.

[0097] (8) After the model training is completed, target point sampling is performed as the target point of multiple prediction trajectories in trajectory prediction to obtain a multimodal prediction result.

[0098] As a preferred embodiment, target point sampling is performed by an MR sampling algorithm, as shown in Table 1 below.

[0099] Table 1 MR sampling algorithm flow

[0100]

[0101] The present embodiment also provides experimental data to illustrate / support a knowledge distillation-based trajectory target point prediction method of the present application. The experiment is completed on the Argoverse dataset, and the model is verified based on the MR sampling algorithm. The widely used evaluation indicators of Argoverse are used for model evaluation, mainly including the following indicators:

[0102] Minimum final displacement error (Minimum Final Displacement Error, minFDE): the L2 distance between the endpoint of the best prediction trajectory and the endpoint of the true trajectory. In the case of multimodal prediction, the best trajectory refers to the trajectory with the smallest endpoint error.

[0103] Miss rate (Miss Rate, MR): according to the endpoint error, the number of scenarios in which the predicted trajectory is not within 2 meters of the true trajectory.

[0104] p-minFDE: similar to minFDE, the difference is that the probability is considered, and min(-log(p),-log(0.05)) is added on the basis of minFDE, where p corresponds to the probability of the best prediction trajectory.

[0105] 1. Comparison of results before and after distillation

[0106] The teacher model is set to the MF-Heat pre-trained model, the student model is distilled, and the model without distillation is compared, and the results are shown in Table 2. The results include all multimodal trajectory points, and the target points are obtained on the trained target point prediction network through the MR sampling algorithm.

[0107] Table 2 Comparison of prediction results of teacher model and student model

[0108]

[0109]

[0110] As can be seen from the table, after distillation, the student model improves in both single-modal and multi-modal performance, with more significant improvement in multi-modal performance, achieving similar results to the teacher model on minFDE, with a gap of only 1 cm, which is 11 cm lower than the student model without distillation. On MR, it is reduced by about 2%, and on p-minFDE, it is reduced by about 0.3, indicating that the distilled model has advantages in both prediction accuracy and diversity, effectively proving that the distillation loss has a positive effect on the knowledge transfer of the teacher model, thereby achieving better trajectory prediction performance.

[0111] The purpose of knowledge distillation is to effectively reduce the number of model parameters and improve inference speed without significantly reducing accuracy. Comparing the parameter quantity and floating point operations (FLOPs) of the teacher model and the student model, the results are shown in Table 3, which shows that the distilled model reduces the number of parameters and floating point operations by about 30%, which helps to improve the inference speed.

[0112] Table 3 Comparison of parameter quantity of teacher model and student model

[0113]

[0114] 2. Distillation loss ablation experiment

[0115] The distilled model is affected by three parts of the distillation loss, in addition, output smoothing also affects the distilled model. To analyze the effect of each part, an ablation experiment is conducted, and the results are shown in Table 4.

[0116] Table 4 Distillation loss ablation results

[0117]

[0118]

[0119] From the ablation results, we can see that increasing the output distillation loss alone, which makes the student model simulate the output of the teacher model, has a negative effect and destroys the generalization ability of the original student model. On the basis of the output distillation loss, adding attention map distillation or fusion feature map distillation will bring benefits and surpass the un-distilled model in effect. This shows that the output and the intermediate layer features are strongly correlated, and the influence of the distillation loss needs to be considered as a whole. Considering a part of the loss alone will make the model lose its original ability, therefore, the distillation loss needs to be complementary to each other to achieve the best effect. Considering the influence of the intermediate layer distillation, the fusion feature map has a greater impact on the results and brings more gains to the model. In addition, the output smoothing processing also promotes the multi-modal results, and after application, all indicators are improved.

[0120] 3. Temperature τ in fusion feature map distillation e and the influence of loss weight γ

[0121] In fusion feature map distillation, changing the temperature τ e will change the probability distribution of each channel in the feature map, so under different loss weights γ, different temperatures τ e are used for experiments, where the weight coefficient γ is γ = 1, 2, 3, and the temperature is τ e ∈ [1, 4], and the results are shown in Figure 6 .

[0122] As can be seen from the figure, increasing the temperature τ e can reduce the displacement error to a certain extent, including single-modal FDE and multi-modal FDE, which shows that a more smooth feature map distribution is helpful for the knowledge transfer of the teacher model, but too high temperature τ e will adversely affect the model and increase the displacement error. For the loss rate MR, the temperature also has a similar effect. The loss weight γ also has a significant impact on the performance of the student model. From the FDE graph, we can see that FDE decreases as γ increases, which shows that increasing γ will make the student model pay more attention to the learning of the intermediate layer feature map, thereby improving the overall performance. In addition, as shown in Figure 6 e), the MR(K = 6) curve shows a trend of first decreasing and then rising as γ increases, which shows that too high γ will reduce the scene coverage of the model. For the student model, γ and τ e are important hyperparameters, and they need to be adjusted according to real needs to produce the greatest benefit.

[0123] 4. Qualitative analysis of predicted trajectory

[0124] After obtaining the target point using the MR sampling algorithm, the predicted trajectory can be completed using a trajectory regression network for visualization analysis. The output of the distilled student model MF_tiny-Heat (Distill) is visualized and compared with the undistilled model MF_tiny-Heat. The results are as follows. Figure 7 As shown.

[0125] Figure 7 a~ Figure 7 f contains the results of three sets of scenarios, in the first set of scenarios ( Figure 7 In scenarios a) and b)), the undistilled student model fails to effectively cover the modes of the U-shaped bend, while after knowledge distillation, the model enhances scene understanding, and the target point distribution is concentrated near the real target point. In the second set of scenarios ( Figure 7 In scenarios c) and d), the undistilled student model completely missed the right-turn mode, while after knowledge distillation, the target point distribution successfully covered the right-turn lane. In the third scenario ( ​ In e) and f)), the undistilled student model, although it covers the straight-line mode, has a large longitudinal error, indicating that the model lacks accurate inference of vehicle speed. After knowledge distillation, the model effectively covers the area where the real target point is located, proving that the teacher model enhances the student model's modeling of longitudinal motion.

[0126] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A trajectory target point prediction method based on knowledge distillation, characterized in that, The method comprises the following steps: (1) collecting trajectory data and high-precision map data, constructing a trajectory prediction dataset, and dividing the dataset into training, validation and test sets; (2) building a target point prediction teacher model, and pre-training the target point prediction teacher model using the trajectory dataset; (3) adjusting the structure of the teacher model from the model architecture and network layer parameter configuration, and taking the adjusted teacher model as a student model to be distilled; (4) defining the supervision loss of the student model according to the true label, so that the model prediction approaches the true label; (5) using the teacher model to perform knowledge distillation on the student model, including: a. Smoothed output distillation: using the output of the target point prediction teacher model to guide the learning of the student model; b. Attention map distillation: using the attention map of the target point prediction teacher model to guide the learning of the student model, the degree of interaction between features is represented by the attention map, and the attention map is calculated by the dot product between vectors; c. Fusion feature map channel distribution distillation: adjusting and aligning the feature maps of the target point prediction teacher model and the student model in the channel dimension; (6) combining the supervision loss and the distillation loss to define the total loss for training the student model; (7) setting a training strategy, using the pre-trained target point prediction teacher model to train a student model with smaller parameter quantity, so that the target point prediction accuracy of the student model approaches the effect of the target point prediction teacher model, while improving the inference speed; (8) sampling target points from the trained student model, and taking the sampled multiple target points as target points of multiple predicted trajectories in trajectory prediction to obtain a multi-modal prediction result.

2. The trajectory target point prediction method based on knowledge distillation according to claim 1, characterized in that, The high-precision map data is represented in a rasterized form, and the lane centerline is encoded into multiple image channels, and the drivable area and the car-to-boundary line are respectively encoded into one or more image channels. 3.The trajectory target point prediction method based on knowledge distillation according to claim 1, wherein, The trajectory prediction dataset includes trajectory data of multiple traffic entities, and according to the constructed trajectory prediction dataset, the prediction task is defined as: for a certain entity in an automatic driving scene, the future trajectory of the entity with a length of T obs is predicted by using the historical trajectory with a length of T pred , the trajectory prediction task is divided into two stages of target point prediction and trajectory regression, the target point prediction stage firstly predicts the target points of multiple future trajectories, and the trajectory regression stage completes trajectory completion based on the predicted target points.

4. The trajectory target point prediction method based on knowledge distillation according to claim 1, characterized in that, The target point prediction teacher model comprises a feature encoder, a multi-source fusion module and a feature decoder; the feature encoder extracts features from the historical trajectory data and the high-precision map data, and projects the original vector to a feature space; The multi-source fusion module fuses the features of the traffic entities and the high-precision map in the scene through an attention mechanism; The feature decoder decodes the fusion features output by the multi-source fusion module to obtain a two-dimensional distribution of target points of multiple predicted trajectories of the predicted entity, wherein the two-dimensional distribution is represented by a grid map, and the size is HxW, and the value of each pixel represents the probability of the target point being located at the position.

5. The trajectory target point prediction method based on knowledge distillation according to claim 1, characterized in that, In step (3), the target point prediction teacher model is adjusted to the student model in the following ways: reducing the feature dimension, reducing the network layer parameter, and replacing the network structure. 6.The trajectory target point prediction method based on knowledge distillation according to claim 1, wherein, The output of the smoothing in step (5) distills the output y of the target point prediction teacher model T The smoothing is performed as follows: wherein y T is a distribution heat map output by the target point prediction teacher model; y GT is a real distribution heat map, assuming that the position of the real target point is the i*row and j*column, then y GT is a two-dimensional Gaussian distribution centered at (i*, j*), is a smoothed distribution heat map; The output of the target point prediction teacher model is assigned a value of 1, i.e. the value of the two-dimensional Gaussian distribution position (i*,j*) is set to 1 y S is a distributional heatmap representing the output of the student model, and the knowledge transfer from the target point prediction teacher model to the student model is achieved by optimizing the distribution y S with respect to the KL divergence of the distribution , as shown in the following equation: wherein, is with KL divergence; and are the distribution probability values of the i-th position in the smoothed distribution heat map and the distribution heat map of the student model output, respectively; H and W represent the height and width of the output heat map. Optimizing the KL divergence is equivalent to optimizing the cross-entropy, which smoothens the output distillation loss is represented as: Wherein, H and W represent the height and width of the output heat map respectively.

7. The trajectory target point prediction method based on knowledge distillation according to claim 1, characterized in that, The attention map distillation of step (5) uses mean square error loss as the distillation target: wherein, represents the attention map distillation loss; H' and W' represent the height and width of the attention map, respectively; and are the values of the hth row and wth column in the attention map of the target point prediction teacher model and student model, respectively, obtained by calculating the dot product of the hth feature and the wth feature, and then applying a softmax function in the dimension where w is located. The features are obtained by passing high-precision map data through a feature encoder or passing traffic entity historical trajectories through a feature encoder. 8.The trajectory target point prediction method based on knowledge distillation according to claim 1, wherein, In the fusion feature map channel distribution distillation of step (5), the historical trajectory data and the high-precision map data will output a feature map with a size of H''xW'' and a channel number of C after passing through the feature encoder and the multi-source fusion module of the teacher / student model. First, the activation values of the channels are converted into probability distributions so as to measure the difference between the distributions by using the probability distance metric, and the probability distribution is converted into a probability distribution by using a softmax function with temperature, as shown below: where, is a softmax function with temperature; e c,i is the value of the i-th position in the c-th channel of the fused feature map; τ e is a temperature hyper-parameter, let and denote the feature maps of the c-th channel of the fused feature maps of the teacher and student models for keypoint prediction, respectively, using the teacher channel distribution and the student channel distribution The distillation loss of the fused feature maps is expressed as the KL divergence between the teacher and student channel distributions, as follows: wherein, represents and the fusion feature map channel distribution distillation loss.

9. The trajectory target point prediction method based on knowledge distillation according to any one of claims 1-8, characterized in that, In step (6), the total loss of the student model is expressed as the sum of the supervision loss and the three-part distillation loss, as shown below: wherein, is a supervision loss; denotes a smoothed output distillation loss, denotes an attention map distillation loss, denotes a fused feature map channel distribution distillation loss, and coefficients λ, η and γ in the distillation losses are hyperparameters for balancing the losses, respectively.

10. The trajectory target point prediction method based on knowledge distillation according to claim 9, characterized in that, In step (8), target point sampling is performed by using the MR sampling algorithm.

Images