Training method of multi-modal trajectory prediction model and electronic device

Abstract

The application discloses a kind of training method of multimodal trajectory prediction model and electronic equipment.Therein, the method relates to automatic driving, model training field, the method includes: using multimodal trajectory prediction model to the trajectory prediction of agent perception information in training data, obtain the prediction result of multiple modes in future time period;Based on the deviation of the prediction trajectory of each mode and the true value trajectory in training data in multiple reference directions, the prediction trajectory of each mode is geometrically matched with the true value trajectory in training data, and the geometric matching result of each mode is obtained;Based on the geometric matching result of multiple modes, determine the first target mode from multiple modes;Based on the prediction result and true value trajectory corresponding to the first target mode, construct target loss value;Based on target loss value, update multimodal trajectory prediction model.The application solves the technical problem that the training effect of multimodal trajectory prediction model is poor in related technology.

Description

Technical Field

[0001] This application relates to the fields of autonomous driving and model training, and more specifically, to a training method and electronic device for a multimodal trajectory prediction model. Background Technology

[0002] In autonomous driving perception and prediction tasks, to accurately characterize the uncertainty of future behavior of surrounding intelligent agents, multimodal trajectory prediction models are typically used to generate multiple possible future trajectories. During the training phase of these models, selecting appropriate modalities from multiple prediction modalities, aligning them with the ground truth labels, and applying supervisory signals are crucial for determining the convergence stability and multimodal diversity of the multimodal trajectory prediction model. However, current model training strategies are inefficient for multimodal trajectory prediction models, easily leading to problems such as multimodal collapse or unstable model learning performance during training.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a training method and electronic device for a multimodal trajectory prediction model, so as to at least solve the technical problem of poor training effect of multimodal trajectory prediction models in related technologies.

[0005] According to one aspect of the embodiments of this application, a training method for a multimodal trajectory prediction model is provided, comprising: using the multimodal trajectory prediction model to predict the trajectory of agent perception information in training data to obtain prediction results of multiple modalities within a future time period, wherein the prediction results include at least: predicted trajectories; based on the deviations of the predicted trajectories of each modality from the ground truth trajectories in the training data in multiple reference directions, performing geometric matching between the predicted trajectories of each modality and the ground truth trajectories in the training data to obtain geometric matching results of each modality; determining a first target modality from the multiple modalities based on the geometric matching results of the multiple modalities; constructing a target loss value based on the prediction results and ground truth trajectories corresponding to the first target modality; and updating the multimodal trajectory prediction model based on the target loss value.

[0006] According to another aspect of the embodiments of this application, a training apparatus for a multimodal trajectory prediction model is also provided, comprising: a prediction module, configured to use the multimodal trajectory prediction model to predict the trajectory of an agent's perception information in training data, thereby obtaining prediction results for multiple modalities within a future time period, wherein the prediction results include at least: predicted trajectories; a matching module, configured to perform geometric matching between the predicted trajectories of each modality and the ground truth trajectories in the training data based on the deviations of the predicted trajectories of each modality and the ground truth trajectories in the training data in multiple reference directions, thereby obtaining geometric matching results for each modality; a first determination module, configured to determine a first target modality from the multiple modalities based on the geometric matching results of the multiple modalities; a construction module, configured to construct a target loss value based on the prediction results and ground truth trajectories corresponding to the first target modality; and an update module, configured to update the multimodal trajectory prediction model based on the target loss value.

[0007] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0008] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0009] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0010] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0011] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0012] In this embodiment, a multimodal trajectory prediction model is used to predict the trajectory of the agent's perception information in the training data, obtaining prediction results for multiple modalities within a future time period. Based on the deviations of the predicted trajectories of each modality from the ground truth trajectories in the training data in multiple reference directions, the predicted trajectories of each modality are geometrically matched with the ground truth trajectories in the training data to obtain geometric matching results for each modality. Based on the geometric matching results of multiple modalities, a first target modality is determined from the multiple modalities. Based on the prediction results and ground truth trajectories corresponding to the first target modality, a target loss value is constructed. The multimodal trajectory prediction model is updated based on the target loss value. By using the multimodal trajectory prediction model to predict the agent's perception information to obtain multimodal results, and performing geometric matching based on the deviations of the predicted trajectories from the ground truth trajectories in multiple reference directions, modalities can be selected based on geometric consistency rather than a single numerical distance, effectively avoiding the problem of inconsistency between the supervision signal and the actual interaction semantics during training. Subsequently, the first target mode is determined based on the geometric matching results, and the target loss value is constructed using the prediction results and ground truth trajectories corresponding to the first target mode to update the model. This ensures that the direction of the training gradient conforms to the physical interaction logic, reduces the reinforcement of erroneous modes, and improves the training stability and prediction accuracy of the multimodal trajectory prediction model. This solves the technical problem of poor training effect of multimodal trajectory prediction models in related technologies. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0014] Figure 1 This is a flowchart of a training method for a multimodal trajectory prediction model according to an embodiment of this application;

[0015] Figure 2 This is a schematic diagram of an optional geometric matching result determination process according to an embodiment of this application;

[0016] Figure 3 This is a detailed flowchart of an optional multimodal trajectory prediction model training method according to an embodiment of this application;

[0017] Figure 4 This is a schematic diagram of an optional supervisory modality application process according to an embodiment of this application;

[0018] Figure 5 This is a schematic diagram of an optional geometric matching determination principle according to an embodiment of this application;

[0019] Figure 6This is a schematic diagram of a training device for a multimodal trajectory prediction model according to an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] According to an embodiment of this application, an embodiment of a training method for a multimodal trajectory prediction model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0023] Figure 1 This is a flowchart of a training method for a multimodal trajectory prediction model according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0024] Step S102: Use a multimodal trajectory prediction model to predict the trajectory of the agent's perception information in the training data to obtain prediction results for multiple modalities in the future time period. The prediction results include at least the predicted trajectory.

[0025] The aforementioned multimodal trajectory prediction model can refer to a deep learning-based artificial intelligence algorithm model. The input to this model can be historical observation data and scene context information, and the output can be a trajectory prediction distribution for multiple different modalities within a future time period. This multimodal trajectory prediction model can include an encoder for feature extraction and a decoder for generating multimodal prediction results with uncertainty. It can characterize various potential possibilities of the agent's future movement through probability distributions or discrete modality sets.

[0026] The aforementioned training data refers to a dataset used for supervised learning or parameter tuning of a multimodal trajectory prediction model. This training data can consist of historical sensor data (such as camera and radar point clouds), high-precision map information, and ground truth values ​​of agent trajectories recorded within historical time periods. During training, this training data can provide a mapping relationship between actual inputs and expected outputs. By determining the difference between the predicted results and the ground truth values, the parameter weights of the multimodal trajectory prediction model can be updated, thereby improving the accuracy and robustness of the multimodal trajectory prediction model in predicting future trajectories.

[0027] The aforementioned agent perception information refers to the state data of other dynamic objects in the surrounding environment (such as automatically or manually driven vehicles, pedestrians, and cyclists, collectively referred to as agents in this embodiment) acquired by the agent through sensors. This agent perception information may include the agent's historical position sequence, speed, heading angle, type classification, bounding box, and semantic features, etc., and can serve as the basic input information for a multimodal trajectory prediction model to understand the surrounding traffic scene and infer the agent's future movement intentions.

[0028] The aforementioned future time period can refer to a preset time window relative to the current observation time (i.e., the input time of the training data), which can be used to define the target range for trajectory prediction. Within this future time period, the multimodal trajectory prediction model can output a sequence of position coordinates of the agent starting from the current time, at regular time steps, thereby describing the agent's future motion evolution.

[0029] The aforementioned prediction results refer to the complete set of predictions output by the multimodal trajectory prediction model for each detected agent after receiving input information. These prediction results can include not only specific motion trajectory data, but also the probability confidence level corresponding to each modality, uncertainty estimates of the motion state, and interaction intent classification labels, thus accurately reflecting the various scenarios the agent will face in future motion and the relative probability of these scenarios occurring.

[0030] The predicted trajectory mentioned above can refer to the sequence data in the prediction result that specifically describes the changes in the agent's spatial position over a future time period. This predicted trajectory can be represented as a series of discrete time steps corresponding to two-dimensional or three-dimensional coordinate points. In a multimodal scenario, this predicted trajectory can contain multiple independent trajectory branches (i.e., multiple modalities), each branch representing a possible motion path. The multimodal trajectory prediction model can cover various driving behaviors or maneuvers that the agent might take by outputting these predicted trajectories.

[0031] In one alternative embodiment, the multimodal trajectory prediction model is considered to be able to output multiple possible future motion scenarios simultaneously, so as to fully characterize the uncertainties in the intelligent agent traffic scenario.

[0032] Based on this, the training system for the multimodal trajectory prediction model (hereinafter referred to as the training system) can receive agent perception information from the training data as input, and use an encoded feature extraction module to process the agent's position, velocity, heading, and environmental context information in the agent perception information. Subsequently, the training system can input the processed features into the multimodal decoding module, generating multiple independent prediction branches in parallel during the decoding process. Each prediction branch can correspond to a motion mode hypothesis. On this basis, the training system can output a set of predicted trajectories containing two-dimensional displacement sequences of multiple time steps within a future time period, thereby providing basic data support for subsequent mode selection and loss determination based on geometric matching and error metrics.

[0033] For example, the training system can input the agent's historical perception frame sequence (i.e., the aforementioned agent perception information) into a multimodal trajectory prediction model. Specifically, this historical perception frame sequence can be input into the multimodal trajectory prediction model, using a Transformer-based encoding network. This encoding network can extract spatiotemporal features from the input through a self-attention mechanism, and these spatiotemporal features can be input into the multimodal decoding network within the multimodal trajectory prediction model. This decoding network can contain multiple parallel prediction branches, each corresponding to a modality. Each prediction branch can output a sequence of two-dimensional plane coordinates for multiple time steps within a future time period to form the predicted trajectory. Furthermore, these prediction branches can output the uncertainty scale parameter of the coordinate axis corresponding to each time step and the logistic value of each modality. These logistic values ​​can be normalized using a softmax function (normalization function) to obtain the prediction probability of each modality. Finally, the multimodal trajectory prediction model can combine the predicted trajectories, uncertainty scale parameters, and prediction probabilities of each modality into a multimodal prediction result based on the training data.

[0034] In another optional embodiment, the training system can input the agent's perception information into a multimodal trajectory prediction model, specifically a scene modeling module built on a graph neural network. This scene modeling module can aggregate the spatial relationship features of surrounding obstacles through graph convolution operations, and then concatenate the aggregated scene features with the agent's own state features before inputting them into the trajectory generation network within the multimodal trajectory prediction model. This trajectory generation network can gradually recover the trajectory sequence for future time periods from Gaussian noise through a multi-step denoising process. In each denoising step, the network simultaneously predicts the mean and variance of the trajectory, ultimately outputting multiple predicted trajectories generated under different random seeds or different initial noise conditions, serving as the prediction results for multiple modalities.

[0035] In another alternative embodiment, the training system can input agent perception information into a multimodal trajectory prediction model based on a sequence encoding module of a Long Short-Term Memory (LSTM) network. This sequence encoding module can extract hidden state vectors and input them into multiple expert networks within the multimodal trajectory prediction model. Each expert network can correspond to a fixed driving behavior modality, and can output predicted trajectory coordinates for multiple time steps within a future time period after weighted fusion of input features. Furthermore, each expert network can output a confidence score corresponding to these predicted trajectory coordinates, and normalize this confidence score as the modality prediction probability. Subsequently, the training system can use the predicted trajectories output by each expert network and their corresponding standard deviations as uncertainty scale parameters, thereby obtaining a multimodal prediction result containing multiple modality predicted trajectories, uncertainty scale parameters, and modality prediction probabilities.

[0036] Step S104: Based on the deviation between the predicted trajectory of each modality and the true trajectory in the training data in multiple reference directions, perform geometric matching between the predicted trajectory of each modality and the true trajectory in the training data to obtain the geometric matching result of each modality.

[0037] The aforementioned ground truth trajectory refers to the actual motion path of the agent recorded in the training data and observed by the sensors. This ground truth trajectory can serve as a standard answer or target reference in supervised learning, used to evaluate and correct the prediction results of the multimodal trajectory prediction model.

[0038] The aforementioned reference direction can refer to the geometric dimension or coordinate axis direction used to quantify the spatial difference between the predicted trajectory and the true trajectory. For example, the reference direction can include longitudinal (along the direction of motion) and lateral (perpendicular to the direction of motion), or different angular directions defined based on a local coordinate system.

[0039] The aforementioned geometric matching result can refer to the logical judgment state obtained by comparing the deviations of the predicted trajectories of each mode from the true trajectory in multiple reference directions and determining whether the deviations meet the preset geometric threshold conditions. This geometric matching result can be used to identify whether the predicted trajectory of any mode is considered to have sufficient consistency with the true trajectory in terms of geometric shape, thereby determining whether the mode should participate in subsequent primary or secondary supervision.

[0040] In one alternative embodiment, considering the use of a geometric matching mechanism based on reference direction deviation, spatial constraints with clear physical meaning can be introduced, enabling the multimodal trajectory prediction model to not only focus on minimizing numerical errors during training, but also to ensure that the predicted trajectory conforms to the actual road geometric constraints in terms of heading consistency and lateral and longitudinal deviations during critical periods, thereby improving the interpretability and rationality of the prediction results at the interactive semantic level.

[0041] Based on this, the training system can transform the predicted trajectories and ground truth trajectories of each modality to a unified relative coordinate system, and can determine the positional differences between adjacent frames of the ground truth trajectory to approximate the agent's velocity and guide its direction. The training system can also determine the longitudinal and lateral deviations between the predicted and ground truth trajectories at multiple sampling times, and compare these deviations with preset thresholds scaled based on the agent's velocity. If the deviation at any sampling time satisfies the preset threshold, the training system can determine that the modality has achieved a successful geometric match. Based on this, the training system can output a set of Boolean results indicating whether each modality satisfies the geometric matching condition, as the aforementioned geometric matching result.

[0042] For example, the training system can determine the heading angle of the ground truth trajectory at multiple sampling times and dynamically scale the preset longitudinal and lateral deviation thresholds based on the agent's historical velocity. Subsequently, the training system can iterate through the predicted trajectory of each modality, determining the longitudinal and lateral deviations between the predicted trajectory points and the ground truth trajectory points at multiple sampling times, and comparing the deviation values ​​at each time point with the corresponding scaling thresholds. If the deviation at any given time point meets the threshold condition, the training system can determine that the modality is a matching modality, thus obtaining the geometric matching results for each modality.

[0043] In another alternative embodiment, the training system can construct a local coordinate system centered on the ground truth trajectory and project the predicted trajectories of each modality into this local coordinate system. This allows the system to determine the integral absolute error between the predicted trajectories of each modality and the ground truth trajectory in both the longitudinal and lateral dimensions. The training system can then compare this integral absolute error with a dynamic error tolerance determined based on agent kinematic constraints. If the integral absolute error is less than or equal to the dynamic error tolerance, the training system can determine that the modality has passed geometric matching, thus obtaining the geometric matching results for each modality. Modalities that have passed matching are marked as matched states, while those that have not are marked as unmatched states.

[0044] In another alternative embodiment, the training system can employ a superior transmission distance algorithm to determine the geometric distance between the predicted trajectory and the true trajectory of each modality, and can predict the motion envelope within a future time period based on the agent's current velocity and acceleration. The training system can use the motion envelope as the acceptable boundary of the geometric matching region. If the superior transmission distance between the predicted trajectory and the true trajectory of each modality is less than or equal to a threshold defined by the motion envelope, the training system can determine that the modality conforms to the geometric constraints, thereby obtaining the geometric matching results of each modality. Modalities that conform to the constraints are marked as matched modes, and modalities that do not conform to the constraints are marked as unmatched modes.

[0045] Step S106: Based on the geometric matching results of multiple modes, determine the first target mode from the multiple modes.

[0046] The aforementioned first target mode can refer to the mode selected from all candidate modes based on the geometric matching results of multiple modes, according to predetermined rules, such as selecting matching modes with smaller numbers or preset priorities. This first target mode can serve as the unique mode index or representative prediction result for applying the master regression supervision signal, thus playing an important role in guiding the main learning direction of the multimodal trajectory prediction model under the logic of prioritizing geometric matching.

[0047] In one alternative embodiment, considering the geometric matching results based on multiple modalities, determining the first target modality can prioritize ensuring that the supervision signal points to the predicted trajectory that conforms to physical geometric constraints and has reasonable motion semantics. Thus, in the presence of an effective geometric matching modality, the multimodal trajectory prediction model can enhance its learning of feasible scenarios, and in the absence of a matching, the stability and continuity of the training process can be guaranteed through a backoff mechanism.

[0048] Based on this, the training system can check the geometric matching results of all modalities. If there is a true geometric matching result for at least one modality, then select the modality with the higher ranking from the successfully matched modalities as the first target modality in a predetermined order, such as the order of modality index from smallest to largest.

[0049] If the geometric matching results of all modes are false, that is, there is no matching mode, the training system can start the backoff strategy, determine the temporal L2 error weighted sum corresponding to each mode, and select the mode with the smaller weighted sum as the first target mode.

[0050] Furthermore, if the weighted sum of multiple modalities is relatively small, the training system can select the modality with the smaller index as the first target modality in a predetermined order, thereby completing the determination of the first target modality.

[0051] For example, the training system can iterate through the geometric matching results of all modalities and filter out the set of modalities marked as matching states. If the set is not empty, the training system can select the modality with the smaller modality index value as the first target modality. If the set is empty, the training system can determine the weighted sum of the temporal L2 (L2 norm) errors of all modalities at multiple future time points and select the modality with the smaller weighted sum as the first target modality. Furthermore, if there are multiple modalities with equally small weighted sums, the training system can select the modality with the smaller modality index value as the first target modality.

[0052] In another optional embodiment, the training system can determine the heading consistency score and position deviation score between the predicted trajectory and the ground truth trajectory for each mode at key time points, and set priority thresholds. If the position deviation score for any mode is lower than the position threshold, and the heading consistency score is higher than the heading threshold, the training system can determine that mode as a geometrically matched mode. If multiple geometrically matched modes exist, the training system can select the mode with the lower position deviation score as the first target mode. If no geometrically matched mode exists, the training system can select the mode with the lowest global position deviation score as the first target mode.

[0053] In another optional embodiment, the training system can construct a multidimensional geometric matching vector including longitudinal deviation, lateral deviation, and heading deviation. For the geometric matching result of each mode, a preset weighted scoring function is applied to determine the geometric matching score. Subsequently, the training system can select the mode with a higher geometric matching score that is greater than a preset positive threshold as the first target mode. If the geometric matching scores of all modes are not greater than the preset positive threshold, the training system can select the mode with the smaller temporal L2 error weighted sum as the first target mode. Furthermore, if there are multiple modes with smaller temporal L2 error weighted sums, the training system can select the mode with the smaller mode index value as the first target mode.

[0054] Step S108: Construct the target loss value based on the prediction result and ground truth trajectory corresponding to the first target mode.

[0055] The aforementioned target loss value can refer to a scalar value derived from the difference between the predicted result corresponding to the first target mode and the true trajectory. This target loss value can be used to quantify the fitting error of the predicted trajectory at key time points through time-weighted regression loss, and can also serve as a master supervision signal to guide the updating of parameters of the multimodal trajectory prediction model.

[0056] In an optional embodiment, considering the construction of a target loss value based on the prediction result and the true trajectory corresponding to the first target mode, a clear gradient adjustment direction can be established, so that the parameter update of the multimodal trajectory prediction model closely revolves around the better prediction mode selected by geometric matching, thereby ensuring that the multimodal trajectory prediction model continuously strengthens its ability to fit the prediction results that meet geometric constraints and have small numerical errors during the training process.

[0057] Based on this, the training system can determine the negative log-likelihood values ​​for multiple future time steps according to the predicted trajectory and the true trajectory of the first target modality. Then, the training system can use preset temporal weights to perform a weighted summation of the negative log-likelihood values ​​at each time step, obtaining a probability error scalar that characterizes the degree of fit to the probability space of the modality. Subsequently, the training system can directly use this probability error scalar as a regression loss term, and combine it with the classification loss value constructed based on the index of the first target modality to form the target loss value used for backpropagation.

[0058] For example, the training system can obtain the predicted trajectory corresponding to the first target mode and the uncertainty scale parameter of the predicted trajectory. Based on the deviation between the predicted trajectory and the ground truth trajectory at multiple future time steps, and the uncertainty scale parameter, the training system can determine the negative log-likelihood value at each time step using a mixture Laplace distribution. Next, the training system can use preset temporal weights to perform a weighted summation of the negative log-likelihood values ​​at each time step to obtain the probability error scalar corresponding to the first target mode. Subsequently, the training system can use the identifier of the first target mode as the class label and apply the cross-entropy function to the mode prediction probabilities of multiple modes to obtain the classification loss value. Finally, the training system can add the probability error scalar to the classification loss value to obtain the initial loss, and then normalize this initial loss by combining the agent weights, sample weights, and an effective time mask to obtain the final target loss value.

[0059] In another optional embodiment, the training system can obtain the predicted trajectory corresponding to the first target mode and directly determine the mean squared error between the predicted trajectory and the ground truth trajectory at multiple future time steps. Subsequently, the training system can use preset temporal weights to perform a weighted summation of the mean squared errors to obtain the regression error scalar corresponding to the first target mode. Next, the training system can determine the Kullback-Leibler divergence (CLED) between the predicted probabilities of multiple modes and the identifier of the first target mode as a classification regularization term, and can add the regression error scalar to the classification regularization term to obtain the combined loss. Finally, the training system can use an effective time mask to perform a mask-weighted average of the combined loss to obtain the target loss value.

[0060] In another optional embodiment, the training system can acquire the predicted trajectory and the ground truth trajectory corresponding to the first target mode, and construct a local coordinate system centered on the ground truth trajectory. The training system can project the predicted trajectory onto this local coordinate system to determine the cumulative bias in the longitudinal and lateral directions. The training system can use the Softplus function (smoothing step function) to perform a nonlinear mapping on the cumulative bias to obtain a smooth regression loss component, and determine the log probability of the first target mode in the mode probability distribution as the confidence loss component. Subsequently, the training system can linearly combine the regression loss component and the confidence loss component according to preset coefficients, and normalize by dividing by the number of effective time steps to obtain the target loss value.

[0061] Step S110: Update the multimodal trajectory prediction model based on the target loss value.

[0062] In one alternative embodiment, considering that the multimodal trajectory prediction model is updated based on the target loss value, the gradient signal can be transmitted from the output layer to each layer of the network through the backpropagation algorithm, thereby effectively adjusting the parameter weights inside the multimodal trajectory prediction model, so as to minimize the comprehensive error between the predicted trajectory and the true trajectory, and improve the trajectory prediction accuracy and reliability of the multimodal output of the multimodal trajectory prediction model in complex traffic scenarios.

[0063] Based on this, the training system can utilize an automatic differentiation mechanism to determine the partial derivative gradient of the target loss value with respect to all trainable parameters of the multimodal trajectory prediction model. The training system can then input the obtained partial derivative gradient into an optimizer such as stochastic gradient descent or Adam (Adaptive Moment Estimation), and iteratively update the weights and bias parameters of the multimodal trajectory prediction model according to preset learning rate and momentum parameters.

[0064] During this process, the training system can also repeatedly perform forward propagation to determine the target loss value and backpropagation to update the parameters until the performance index of the multimodal trajectory prediction model on the validation set meets the convergence condition or reaches the preset training rounds, thereby completing the training of the multimodal trajectory prediction model.

[0065] For example, the training system can determine the gradients of each parameter in the multimodal trajectory prediction model relative to the target loss value using the backpropagation algorithm. Subsequently, the training system can use a stochastic gradient descent optimizer or an adaptive moment estimation optimizer to update the model parameters based on the obtained gradients. Specifically, the training system can subtract the product of the learning rate and the gradient from the current parameter values ​​to complete one training iteration. The training system can repeat this process until the multimodal trajectory prediction model converges or reaches a preset number of training epochs.

[0066] In another alternative embodiment, the training system can construct the state variables of an AdamW (Adam with Weight Decay) optimizer based on the target loss value. Subsequently, the training system can obtain the gradients of the model parameters through backpropagation and use the AdamW algorithm to perform momentum correction and weight decay on the gradients, generating updated parameter values. The training system can then apply these updated values ​​to the model parameters to achieve parameter updates. After each training step, a cosine annealing scheduling strategy can be applied to the learning rate to adjust its magnitude, thereby completing the update of the multimodal trajectory prediction model.

[0067] In another alternative embodiment, the training system can use gradient accumulation techniques based on the target loss value to perform forward and backward propagation on multiple micro-batches to accumulate gradients. When the accumulated gradients reach a preset number of gradient accumulation steps, the training system can perform a parameter update operation. Using a LAMB (Layer-wise Adaptive Moments optimizer for Batch training) or RAdam (Rectified Adam) optimizer, the accumulated gradients are normalized and scaled to determine the update step size for the model parameters. The training system can then apply this update step size to the model parameters, thereby completing the parameter update on the multimodal trajectory prediction model. Afterward, the training system can reset the gradient accumulator to prepare for the next training cycle.

[0068] In this embodiment, a multimodal trajectory prediction model is used to predict the trajectory of the agent's perception information in the training data, obtaining prediction results for multiple modalities within a future time period. Based on the deviations of the predicted trajectories of each modality from the ground truth trajectories in the training data in multiple reference directions, the predicted trajectories of each modality are geometrically matched with the ground truth trajectories in the training data to obtain geometric matching results for each modality. Based on the geometric matching results of multiple modalities, a first target modality is determined from the multiple modalities. Based on the prediction results and ground truth trajectories corresponding to the first target modality, a target loss value is constructed. The multimodal trajectory prediction model is updated based on the target loss value. By using the multimodal trajectory prediction model to predict the agent's perception information to obtain multimodal results, and performing geometric matching based on the deviations of the predicted trajectories from the ground truth trajectories in multiple reference directions, modalities can be selected based on geometric consistency rather than a single numerical distance, effectively avoiding the problem of inconsistency between the supervision signal and the actual interaction semantics during training. Subsequently, the first target mode is determined based on the geometric matching results, and the target loss value is constructed using the prediction results and ground truth trajectories corresponding to the first target mode to update the model. This ensures that the direction of the training gradient conforms to the physical interaction logic, reduces the reinforcement of erroneous modes, and improves the training stability and prediction accuracy of the multimodal trajectory prediction model. This solves the technical problem of poor training effect of multimodal trajectory prediction models in related technologies.

[0069] In the above embodiments of this application, based on the deviations of the predicted trajectories of each modality from the ground truth trajectories in the training data in multiple reference directions, the predicted trajectories of each modality are geometrically matched with the ground truth trajectories in the training data to obtain the geometric matching results of each modality. This includes: obtaining predicted trajectory points at multiple sampling times from the predicted trajectories of each modality based on multiple sampling times in a future time period, and obtaining ground truth trajectory points at multiple sampling times from the ground truth trajectories; matching the predicted trajectory points at each sampling time with the ground truth trajectory points based on the deviations of the predicted trajectory points at each sampling time from the ground truth trajectories in multiple reference directions to obtain the matching results corresponding to each sampling time; and determining the geometric matching results of each modality based on the matching results corresponding to multiple sampling times.

[0070] The aforementioned sampling time can refer to multiple discrete time points distributed within the predicted future time period. It can be used to extract specific location coordinates from the continuous predicted trajectory and the true trajectory, so as to compare the predicted trajectory with the actual observed trajectory point by point and determine the geometric deviation at important time nodes.

[0071] The predicted trajectory point mentioned above can refer to the spatial coordinates of the agent at each sampling time in any modality, generated by a multimodal trajectory prediction model. This predicted trajectory point can represent the estimate of the agent's possible location at that moment by the multimodal trajectory prediction model.

[0072] The aforementioned ground truth trajectory points can refer to the actual spatial coordinates of the agent recorded in the training data, corresponding to the actual observed location of the agent at the same sampling time. These ground truth trajectory points can serve as an objective standard reference, used to compare with the predicted trajectory points to evaluate the accuracy of the prediction results.

[0073] The matching result mentioned above can refer to a binary judgment state obtained by comparing the deviations of the predicted trajectory point and the true trajectory point in multiple reference directions at any sampling time, and determining whether they meet a preset threshold condition. This matching result can be used to identify whether the predicted trajectory at that sampling time is considered to be geometrically consistent with the true trajectory.

[0074] In one alternative embodiment, considering the deviations of the predicted trajectories based on each modality from the ground truth trajectories in the training data in multiple reference directions, geometric matching can be performed to introduce time-series constraints with clear physical meaning. By verifying in time periods whether the deviations of the predicted trajectories in the longitudinal and lateral directions meet the dynamic thresholds based on agent velocity scaling, it can be ensured that the selected supervisory modality not only has low numerical error, but also has geometric rationality and interpretability in kinematic characteristics.

[0075] Based on this, the training system can determine multiple discrete sampling times within a future time period and extract the two-dimensional position coordinates of the corresponding sampling times from the predicted trajectories output by each modality, as predicted trajectory points. Similarly, the training system can extract the two-dimensional position coordinates of the same sampling times from the ground truth trajectory sequence of the training data, as ground truth trajectory points.

[0076] Next, for each sampling moment, the training system can determine the longitudinal and lateral deviations between the predicted trajectory point and the true trajectory point in the local coordinate system, and compare these two deviation values ​​with preset thresholds that dynamically scale with the agent's current velocity. If the longitudinal or lateral deviation at any sampling moment meets the threshold, the training system can determine that the sampling moment is a successful match; otherwise, the training system can determine that the sampling moment is a failed match.

[0077] If any sampling moment is determined to be a successful match, the training system can determine that the geometric matching result of that mode is true. If all sampling moments fail to match, the training system can determine that the geometric matching result of that mode is false, thus obtaining the geometric matching result of each mode.

[0078] For ease of understanding, Figure 2 This is a schematic diagram of an optional geometric matching result determination process according to an embodiment of this application, such as... Figure 2As shown, the training system can obtain predicted trajectory points for multiple sampling times from the predicted trajectories of each modality within a future time period, and obtain ground truth trajectory points for multiple sampling times from the ground truth trajectories. Subsequently, the training system can match the predicted trajectory points and ground truth trajectory points at each sampling time with the ground truth trajectory points based on the deviations of the predicted trajectory points and ground truth trajectory points in multiple reference directions, obtaining the matching results corresponding to each sampling time. Finally, the training system can determine the geometric matching results for each modality based on the matching results corresponding to multiple sampling times.

[0079] In the above embodiments of this application, based on the deviations between the predicted trajectory points and the true trajectory points at each sampling time in multiple reference directions, the predicted trajectory points and the true trajectory points at each sampling time are matched to obtain the matching results corresponding to each sampling time. This includes: constructing matching thresholds corresponding to multiple sampling times, wherein the matching thresholds include thresholds corresponding to multiple reference directions; matching the deviations between the predicted trajectory points and the true trajectory points at each sampling time in multiple reference directions with the matching thresholds corresponding to each sampling time to obtain the matching results corresponding to each sampling time; wherein, constructing matching thresholds corresponding to multiple sampling times in a future time period includes: determining the speed of the agent; adjusting the initial thresholds corresponding to multiple sampling times based on the speed to obtain the matching thresholds corresponding to multiple sampling times.

[0080] The aforementioned matching threshold can refer to a dynamic error limit used to determine whether the predicted trajectory point and the true trajectory point are geometrically consistent at any sampling time. This matching threshold can include upper limits of allowable deviation in different reference directions such as longitudinal and lateral, and can be adaptively adjusted according to the agent's real-time speed. For example, the higher the agent's speed, the larger the allowable range of deviation for the training system, thereby ensuring that the evaluation of trajectory geometric similarity conforms to physical laws when the agent is in different motion states.

[0081] The aforementioned initial threshold can refer to a pre-set standard error limit used to measure the deviation between the predicted trajectory and the true trajectory in multiple reference directions, before considering the influence of the agent's velocity. This initial threshold can serve as a basic benchmark and needs to be scaled or corrected in conjunction with the agent's velocity information before it can be transformed into a matching threshold suitable for a specific scenario.

[0082] In one alternative embodiment, considering the construction of a matching threshold based on the agent's speed dynamic adjustment, it is possible to adapt to the kinematic characteristics of the agent at different moving speeds, thereby enabling the geometric matching criterion to have a larger fault tolerance range in high-speed scenarios and maintain strict positioning accuracy in low-speed scenarios, thereby improving the fitting degree of the geometric matching results to the real traffic flow motion pattern.

[0083] Based on this, the training system can determine the agent's average or instantaneous velocity up to the current moment from the historical trajectory of the training data. The training system can obtain a preset initial threshold as a benchmark parameter, and then use this velocity value to linearly or non-linearly scale and adjust the initial threshold to generate a matching threshold that varies with velocity.

[0084] Subsequently, for each sampling moment within a future time period, the training system can determine the deviations of the predicted trajectory point and the ground truth trajectory point in the longitudinal reference direction and the lateral reference direction, respectively. The training system can then compare the obtained longitudinal deviation with the matching threshold corresponding to the longitudinal reference direction to determine whether the longitudinal deviation is less than or equal to the matching threshold.

[0085] Similarly, the training system can compare the lateral deviation with the dynamic matching threshold corresponding to the lateral reference direction to determine whether the lateral deviation is less than or equal to the matching threshold. If both the longitudinal and lateral deviations meet their respective matching thresholds, the training system can determine that the matching result at that sampling time is true. If either the longitudinal or lateral deviation does not meet its corresponding matching threshold, the training system can determine that the matching result at that sampling time is false, thus obtaining the matching results for each sampling time.

[0086] For example, the training system can extract the agent's historical position sequence at the current moment and determine the instantaneous velocity through difference. After obtaining this velocity value, the training system can map the instantaneous velocity to a longitudinal deviation threshold and a lateral deviation threshold according to a preset velocity-threshold mapping function, thereby constructing a matching threshold set containing multiple reference direction thresholds. Subsequently, the training system can traverse multiple sampling moments within a future time period, determining the longitudinal and lateral coordinate differences between the predicted trajectory point and the true trajectory point at each sampling moment, as the deviation. The training system can compare the longitudinal and lateral deviations at each sampling moment with the corresponding longitudinal and lateral deviation thresholds, respectively. If the deviation is less than or equal to the corresponding matching threshold, the training system can determine that the matching result at that sampling moment is a match; if the deviation is greater than the corresponding matching threshold, the training system can determine that the matching result at that sampling moment is a mismatch, thus obtaining the matching result corresponding to each sampling moment.

[0087] In another optional embodiment, the training system can fit a kinematic model based on the agent's historical trajectory data to predict the agent's expected velocity distribution over future time periods. The training system can take the mean of the expected velocity distribution as the agent's representative velocity and convert this representative velocity into a dynamic adjustment factor using a linear scaling factor. Subsequently, the training system can multiply this dynamic adjustment factor by preset base longitudinal and lateral thresholds to obtain longitudinal and lateral matching thresholds corresponding to multiple sampling times. Next, the training system can determine the longitudinal and lateral error components of the predicted trajectory point at each sampling time relative to the true trajectory point, and compare the longitudinal error component at each sampling time with the longitudinal matching threshold, and the lateral error component with the lateral matching threshold. If the error components in both directions satisfy the condition of being less than or equal to the corresponding matching threshold, the training system can generate a matching success flag for that sampling time; otherwise, it generates a matching failure flag, thereby constructing a sequence of matching results for each sampling time.

[0088] In the above embodiments of this application, the deviations between the predicted trajectory points and the true trajectory points at each sampling time in multiple reference directions are matched with the matching thresholds corresponding to each sampling time to obtain the matching results for each sampling time. This includes: if the deviations in each reference direction are all less than or equal to the thresholds corresponding to each reference direction, the matching result for that sampling time is determined to be a successful match between the predicted trajectory point and the true trajectory point at that sampling time; if the deviation in any reference direction is greater than the threshold corresponding to that reference direction, the matching result for that sampling time is determined to be a successful match between the predicted trajectory point and the true trajectory point at that sampling time. The corresponding ground truth trajectory point matching failed; wherein, based on the matching results corresponding to multiple sampling times, the geometric matching result of each mode is determined, including: if the matching result corresponding to any sampling time is that the predicted trajectory point corresponding to that sampling time and the ground truth trajectory point corresponding to that sampling time are successfully matched, the geometric matching result of each mode is determined to indicate that the predicted trajectory of each mode and the ground truth trajectory are successfully matched; if the matching results corresponding to multiple sampling times are that the predicted trajectory points corresponding to multiple sampling times and the ground truth trajectory points corresponding to multiple sampling times are all unmatched, the geometric matching result of each mode is determined to indicate that the predicted trajectory of each mode and the ground truth trajectory are unmatched.

[0089] In one alternative embodiment, considering the use of point-by-point matching logic based on reference direction deviation and dynamic threshold, the geometric consistency between the predicted trajectory and the true trajectory in the kinematic dimension can be accurately quantified. Furthermore, by employing a success-based decision strategy for all time periods or single points, the robustness and coverage of the matching results can be ensured, thereby providing a reliable geometric semantic basis for subsequent modality selection.

[0090] Based on this, the training system can obtain the longitudinal bias, lateral bias, and matching thresholds corresponding to the current sampling time, respectively. The training system can determine whether the longitudinal bias is less than or equal to the longitudinal threshold, and similarly, it can determine whether the lateral bias is less than or equal to the lateral threshold.

[0091] If the longitudinal deviation is less than or equal to the longitudinal threshold, and the lateral deviation is less than or equal to the lateral threshold, the training system can determine that the matching result at that sampling time is a successful match. If either the longitudinal deviation is greater than the longitudinal threshold, or the lateral deviation is greater than the lateral threshold, the training system can determine that the matching result at that sampling time is a failed match.

[0092] The training system can aggregate the matching results of all sampling times within a future time period. If at least one predicted trajectory point at a sampling time successfully matches the ground truth trajectory point at that sampling time, the training system can determine that the geometric matching result for each mode is a successful match between the predicted trajectory and the ground truth trajectory. If none of the predicted trajectory points at any sampling time match the ground truth trajectory points at that sampling time, the training system can determine that the geometric matching result for each mode is a failed match between the predicted trajectory and the ground truth trajectory.

[0093] For example, for each sampling moment within a future time period, the training system can determine the coordinate differences between the predicted trajectory point and the ground truth trajectory point in both the longitudinal and lateral reference directions. Then, the training system can compare the longitudinal difference with a longitudinal threshold and the lateral difference with a lateral threshold. If the absolute value of both the longitudinal and lateral differences is less than or equal to the longitudinal and lateral thresholds, the training system can determine that the matching result for that sampling moment is successful. If either the longitudinal or lateral difference is greater than the corresponding threshold, the training system can determine that the matching result for that sampling moment is unsuccessful. Subsequently, the training system can iterate through the matching results of all sampling moments. If at least one sampling moment shows a successful matching result, the training system can determine that the geometric matching result for that mode is successful; if all sampling moments show unsuccessful matching results, the training system can determine that the geometric matching result for that mode is unsuccessful.

[0094] In another optional embodiment, for each sampling moment, the training system can extract the longitudinal and lateral displacement deviations of the predicted trajectory points and the ground truth trajectory points in the current vehicle coordinate system. The training system can numerically compare the longitudinal displacement deviation with a longitudinal threshold dynamically adjusted based on speed, and the lateral displacement deviation with a lateral threshold dynamically adjusted based on speed. If the longitudinal displacement deviation is less than or equal to the longitudinal threshold, and the lateral displacement deviation is less than or equal to the lateral threshold, the training system can mark that sampling moment as a successful match; otherwise, it can mark it as a failed match. Next, the training system can summarize the state markers of the mode across all sampling moments. If at least one sampling moment is in a successful match state, the overall geometric matching of the mode is considered successful; only if all sampling moments are in a failed match state is the overall geometric matching of the mode considered failed.

[0095] In the above embodiments of this application, the method further includes: determining the heading angle based on adjacent displacements in the true trajectory; determining the position deviation at each sampling time based on the displacement between the predicted trajectory point and the true trajectory point at each sampling time; and decomposing the position deviation at each sampling time in multiple reference directions based on the heading angle to obtain the deviation between the predicted trajectory point and the true trajectory point at each sampling time in multiple reference directions.

[0096] The heading angle mentioned above can refer to the angle of motion direction determined based on the position displacement vectors of the agent at two adjacent moments in the true trajectory. This heading angle can reflect the actual driving orientation of the agent at the sampling moment, and thus can be used to establish a local coordinate system, so as to decompose the position deviation in the global coordinate system into a longitudinal component along the motion direction and a lateral component perpendicular to the motion direction.

[0097] The aforementioned positional deviation refers to the Euclidean distance vector between the predicted trajectory point and the true trajectory point in the two-dimensional plane at each sampling time. This positional deviation can include the combined differences between the predicted and true positions in the lateral and longitudinal directions, thus serving as the basis for subsequent coordinate rotation and direction decomposition combined with the heading angle to obtain different reference direction deviations.

[0098] In an alternative embodiment, considering that the position deviation can be decomposed into geometric components along and perpendicular to the direction of motion by using the heading angle of the true trajectory, the influence of the agent's orientation change on the deviation calculation can be eliminated, thereby enabling the deviation in different reference directions to more accurately reflect the degree of deviation of the predicted trajectory from the true trajectory in a kinematic sense.

[0099] Based on this, the training system can extract the position coordinates of two adjacent time steps in the true trajectory and determine the vector difference between these two position coordinates. The training system can then use the arctangent function to determine the heading angle for the current time period based on the horizontal and vertical components of this vector difference. Next, the training system can determine the two-dimensional position coordinate difference between the predicted trajectory point and the true trajectory point at each sampling time to obtain the original position deviation vector. The training system can construct a local rotation matrix based on the heading angle and transform the two-dimensional position coordinate difference under the action of this rotation matrix, so that the transformed coordinate axes are parallel to and perpendicular to the heading angle direction, respectively. Finally, the training system can extract the first component of the transformed coordinate vector as the vertical deviation and the second component as the horizontal deviation, thereby obtaining the deviation between the predicted trajectory point and the true trajectory point at each sampling time in each reference direction.

[0100] For example, the training system can extract the coordinates of two adjacent time steps in the true trajectory and determine the angle between the direction vector of the line connecting these two coordinate points and the X-axis of the global coordinate system, which serves as the heading angle for that time period. Furthermore, the training system can determine the coordinate difference between the predicted trajectory points and the true trajectory points, obtaining a position deviation vector. This position deviation vector can then be projected onto the longitudinal axis (direction of vehicle heading) and the lateral axis (perpendicular to the direction of vehicle heading), determined by the heading angle. Based on this, the training system can obtain the longitudinal and lateral deviations through dot product operations, thus decomposing the position deviation into components in multiple reference directions.

[0101] In another alternative embodiment, the training system can smooth the true trajectory to eliminate noise, determine the displacement vector between adjacent sampling points and normalize it to obtain a unit tangent vector, the angle of which is the heading angle. Subsequently, the training system can determine the Euclidean distance vector between the predicted point and the true point at each sampling time, and perform a dot product operation between this Euclidean distance vector and the unit tangent vector, as well as the perpendicular vector of the unit tangent vector, to obtain scalar values ​​for the longitudinal deviation and the lateral deviation, respectively, thereby completing the decomposition of the position deviation in the heading coordinate system.

[0102] In the above embodiments of this application, determining a first target mode from multiple modes based on the geometric matching results of multiple modes includes: determining the first target mode from the modes corresponding to at least one predicted trajectory when the geometric matching results of multiple modes indicate that at least one predicted trajectory successfully matches the true trajectory, wherein the identifier value of the first target mode is less than the identifier value of other modes, and the other modes are modes other than the first target mode among the modes corresponding to at least one predicted trajectory.

[0103] The aforementioned identifier value can refer to a discrete numerical index used for sorting and uniqueness differentiation among multiple candidate modalities. When a geometric match is successful and multiple candidate modalities exist, the training system can determine the first target modality from the multiple successfully matched modalities by comparing the identifier values ​​of each modality according to predetermined rules, thus ensuring the determinism and consistency of the modality selection logic.

[0104] The aforementioned other modes can refer to the set of all candidate modes that, apart from the selected first target mode, also satisfy the geometric matching success condition at the same geometric matching determination time.

[0105] In one alternative embodiment, considering that the first target mode is determined based on the geometric matching results of multiple modes, it is possible to prioritize ensuring that the supervision signal points to the predicted trajectory that conforms to the physical geometric constraints and has reasonable motion semantics. This can enhance the learning of feasible scenarios by the multimodal trajectory prediction model when there are effective geometric matching modes.

[0106] Based on this, the training system can check the geometric matching result set of all modes. If the geometric matching results of multiple modes indicate that at least one predicted trajectory matches the true trajectory, the training system can select the mode with the higher order from the modes corresponding to at least one predicted trajectory that has been successfully matched as the first target mode according to a preset order, such as the order of the corresponding identifier values ​​of the modes from small to large.

[0107] For example, the training system can iterate through the list of geometric matching results for all modalities and filter out the set of modality indices marked as successfully matched. If this set is not empty, the training system can directly select the index value with the smaller value (i.e., the aforementioned identifier value) as the identifier of the first target modality. If the set is empty, the training system can determine the weighted sum of the temporal L2 errors of all modalities and select the modality with the smaller weighted sum as the first target modality. If multiple modality weighted sums are all small, the training system can select the modality with the smaller index value as the first target modality.

[0108] In another optional embodiment, the training system can initialize the first target modality identifier to an undefined state and sequentially check the geometric matching flag of each modality. Once it detects that the matching flag of any modality is true, the training system can immediately assign the identifier value of that modality to the first target modality identifier and terminate the traversal. If the first target modality identifier is still undefined after traversing all modalities, the training system can backtrack to selecting the modality based on the criterion of smaller weighted sum of temporal L2 errors, thereby selecting the modality with the smaller weighted sum and smaller identifier value as the first target modality.

[0109] In the above embodiments of this application, the method further includes: when the geometric matching results of multiple modes indicate that the predicted trajectories and true trajectories of multiple modes fail to match, determining weight values ​​corresponding to multiple future moments within a future time period based on the true trajectories, wherein the weight values ​​corresponding to the multiple future moments satisfy a monotonically non-decreasing relationship; determining a geometric error scalar corresponding to each mode based on the predicted trajectories, true trajectories, and weight values ​​corresponding to the multiple future moments, wherein the geometric error scalar is used to characterize the deviation between the predicted trajectories of each mode and the true trajectories in geometric space; determining the minimum value among the geometric error scalars corresponding to the multiple modes, and determining the mode corresponding to the minimum value as the first target mode.

[0110] The aforementioned weight values ​​can refer to scalar coefficients assigned to each discrete future moment within a future time period. These weight values ​​can satisfy a monotonically non-decreasing relationship over time, which is used to give greater importance to trajectory deviations at more distant moments when the geometric error is certain. This strengthens the multimodal trajectory prediction model's ability to fit the predicted trajectory's end position and long-term trend, thereby ensuring that the model selection strategy focuses more on the consistency of the endpoint in a backoff scenario where all geometric matching fails.

[0111] The aforementioned geometric error scalar can be a single value obtained by multiplying the spatial deviations between the predicted trajectories of each mode at multiple future time points and the true trajectory using a weighted summation method, multiplying the deviations by their corresponding weights, and aggregating the results. This geometric error scalar comprehensively reflects the overall degree of deviation of each mode's predicted trajectory from the true trajectory in geometric space throughout the entire prediction period. The smaller the value of this geometric error scalar, the closer the predicted trajectory of that mode is to the true trajectory.

[0112] In an alternative embodiment, considering the failure of all geometric matching, the introduction of a geometric error scalar based on monotonically non-decreasing weights as a backoff criterion can enhance the fitting ability of the multimodal trajectory prediction model to the trajectory end position. Thus, modes with smaller numerical errors can be used as supervision targets to maintain the continuity and stability of the training process.

[0113] Based on this, the training system can determine multiple future moments within a future time period and assign corresponding weight values ​​to each future moment according to a preset monotonically non-decreasing relationship, wherein the weight value of a subsequent moment can be greater than or equal to the weight value of a preceding moment.

[0114] For each mode, the training system can determine the Euclidean distance between the trajectory points at each future time on the predicted trajectory of that mode and the corresponding trajectory points at each future time on the true trajectory. It can also multiply the Euclidean distances at each future time by their respective weight values ​​and sum them up to obtain the geometric error scalar for that mode.

[0115] Subsequently, the training system can iterate through all modes and compare the geometric error scalars corresponding to these modes to select the mode with the smaller geometric error scalar. If multiple modes have equally small geometric error scalars, the training system can select the mode with the smaller label value as the first target mode according to a preset predetermined order.

[0116] For example, the training system can construct a vector with a length equal to the number of future time steps, where each element corresponds to a future time point, and the vector element values ​​exhibit a monotonically non-decreasing distribution as the time index increases, thus emphasizing the importance of the final time point. Next, the training system can determine the L2 distance between the predicted trajectory point and the ground truth trajectory point for each modality at each time point, and multiply the L2 distance at each time point by the corresponding weight value to obtain the weighted error. The training system can then sum the weighted errors over all time points to obtain the geometric error scalar for that modality. Finally, the training system can iterate through the geometric error scalars of all modalities and select the modality corresponding to the scalar with the smallest value as the first target modality.

[0117] In another optional embodiment, the training system can generate a weight sequence according to a preset time decay strategy. During this process, it can ensure that the weight at a later time step is not less than the weight at the previous time step. The training system can determine the squared position deviation between the predicted trajectory and the true trajectory at each time step for each mode, and can multiply the squared position deviation by the corresponding weight value and accumulate the results to obtain the weighted squared error scalar for that mode. The training system can compare the weighted squared error scalars of all modes, find the smaller value, and identify the mode corresponding to the smaller value as the first target mode.

[0118] In the above embodiments of this application, determining the weight values ​​corresponding to multiple future moments within a future time period based on the true trajectory includes: transforming the true trajectory to the coordinate system corresponding to multiple modes to obtain the transformed trajectory; generating a time-valid mask based on the transformed trajectory, wherein the time-valid mask is used to characterize whether the true trajectory points corresponding to multiple future moments in the true trajectory are valid data; determining the weight values ​​corresponding to multiple future moments based on the time-valid mask, the weight values ​​corresponding to the agent, and the weight values ​​corresponding to the training data; wherein determining the geometric error scalar corresponding to each mode based on the predicted trajectory, the true trajectory, and the weight values ​​corresponding to multiple future moments includes: determining the Euclidean distance corresponding to multiple future moments based on the predicted trajectory and the true trajectory of each mode; and weighting the Euclidean distance corresponding to multiple future moments based on the weight values ​​corresponding to multiple future moments to obtain the geometric error scalar corresponding to each mode.

[0119] The aforementioned time validity mask can refer to a sequence of binary or Boolean tags used to identify the data validity status at each discrete moment within a future prediction period of the true trajectory. This validity time mask can guide the loss calculation or error aggregation process to ignore invalid moments by distinguishing between valid and invalid data points, thereby ensuring that the determination of the geometric error scalar is based solely on reliable true observation data.

[0120] In one alternative embodiment, considering the generation of a time-effective mask based on the true trajectory and the construction of a multi-scale weighting mechanism by combining the agent and sample weights, the interference of invalid data on geometric error calculation can be eliminated, and the contribution differences of different samples and time periods can be reflected, thereby ensuring that the geometric error scalar can accurately reflect the true geometric fit goodness of each modality prediction trajectory within the effective time span.

[0121] Based on this, the training system can transform the true trajectory from the global coordinate system to the local coordinate system corresponding to each mode, and generate a time-valid mask based on the data integrity of each future time in the true trajectory. The mask value corresponding to the valid time can be 1, and the mask value corresponding to the invalid time can be 0.

[0122] The training system can extract the weight values ​​corresponding to the agent at a given future time (hereinafter referred to as agent weight values) and the weight values ​​corresponding to the training data (hereinafter referred to as sample weight values). Multiplying the effective temporal mask value, agent weight values, and sample weight values ​​yields the base weights for that future time. The training system can also determine the Euclidean distances between the predicted trajectories and ground truth trajectories for each modality at multiple future time points. Multiplying the Euclidean distances at each future time point by the corresponding base weights and summing them yields the geometric error scalar for each modality.

[0123] For example, the training system can transform the ground truth trajectory from a global coordinate system to a relative coordinate system with the agent's current pose as the origin, and generate a time-valid mask to identify valid data moments. Based on this, the training system can combine agent-level weights and sample-level weights, multiplying the weights that are valid and non-zero in the mask, and dividing by the sum of the valid weights to obtain the normalized temporal point weights. Next, the training system can determine the Euclidean distance between the predicted trajectory and the ground truth trajectory at each future moment, multiply the Euclidean distance at each moment by the corresponding temporal point weight, and sum them to obtain the geometric error scalar for each mode.

[0124] For example, the training system can transform the ground truth trajectory to a relative coordinate system with the agent's current pose as the origin, and generate a time-validity mask to identify valid data moments. It can also combine this with an obstacle vehicle type mask to exclude specific categories that are not subject to supervision. Based on this, the training system can construct initial temporal point weights according to the time-validity mask, type mask, and obstacle-level and sample-level weights, where the weights for invalid moments or types to be excluded can be directly set to zero. Subsequently, the training system can multiply the non-zero valid weights at each moment to obtain the cumulative weights, and then normalize them by dividing by the sum of the valid weights, thus obtaining the final temporal point weights used for weighted calculation. Then, the training system can determine the Euclidean distance between each modality's predicted trajectory and the ground truth trajectory at valid moments, and multiply the Euclidean distance at each moment by the corresponding normalized temporal point weights and sum them to obtain the geometric error scalar for each modality. This geometric error scalar reflects the overall geometric deviation of the modality after considering time validity, type filtering, and weight normalization.

[0125] In another optional embodiment, the training system can construct a Boolean-type time-valid mask based on the distribution of the true trajectory within the effective time period. For moments when the mask is true, the training system can multiply the agent weights, sample weights, and coefficients adjusted based on time monotonicity to obtain initial weights. The training system can normalize the initial weights to obtain the final temporal point weights, and then determine the spatial distance between the predicted trajectory point and the true trajectory point at each time step. The distance values ​​at each time step can be multiplied by the corresponding temporal point weights and accumulated to generate a scalar value characterizing the degree of geometric deviation of each mode, which serves as the aforementioned geometric error scalar.

[0126] In the above embodiments of this application, the prediction result further includes: an uncertainty scale parameter and a mode prediction probability; based on the prediction result and the true trajectory corresponding to the first target mode, a target loss value is constructed, including: based on the prediction trajectory of each mode, the uncertainty scale parameter and the mode prediction probability, and the true trajectory, determining the negative log-likelihood values ​​corresponding to multiple future time moments; based on the weight values ​​corresponding to multiple future time moments, weighting the negative log-likelihood values ​​corresponding to multiple future time moments to obtain the probability error scalar corresponding to each mode, wherein the probability error scalar is used to characterize the degree of fit between the prediction trajectory of each mode and the true trajectory in the probability space; and based on the probability error scalar corresponding to the first target mode, a target loss value is constructed.

[0127] The aforementioned uncertainty scale parameter can refer to a scalar value that is additionally predicted by the multimodal trajectory prediction model when outputting the predicted position at each future time, and is used to characterize the reliability or dispersion of the prediction result.

[0128] The aforementioned modality prediction probability can refer to the numerical distribution representing the probability that each predicted modality is a real scene, output by the classification head of the multimodal trajectory prediction model. This modality prediction probability can exist in the form of log odds, and during training, it can be aligned with the selected first target modality index through cross-entropy loss, thereby ensuring that the modality priority output by the multimodal trajectory prediction model is consistent with the master supervised modality determined based on geometric or error criteria.

[0129] The aforementioned negative log-likelihood value refers to a scalar loss value derived from the predicted trajectory, uncertainty scale parameter, and true trajectory of each modality within a probability distribution model. It measures the statistical difference between the predicted distribution and the actual observed values. This negative log-likelihood value reflects the quality of the prediction result's fit to the true value in the probability space. The smaller the negative log-likelihood value, the higher the probability that the predicted distribution covers the true value or the higher the prediction accuracy.

[0130] The aforementioned probability error scalar can refer to a single value obtained by weighting and aggregating the negative log-likelihood values ​​of multiple future times with their corresponding time weights. This probability error scalar can comprehensively characterize the overall degree of fit between the predicted trajectories of each modality and the ground truth trajectories at the probability distribution level throughout the entire prediction period. Therefore, it can serve as a core supervisory signal for constructing the main regression loss term to guide the parameter updates of the multimodal trajectory prediction model.

[0131] In one alternative embodiment, by utilizing an uncertainty scaling parameter to transform the positional deviation of the predicted trajectory into a negative log-likelihood at the probability distribution level, both prediction accuracy and the calibrability of uncertainty estimation can be improved. Furthermore, by introducing temporal weights to weight and aggregate the probability errors, the monitoring signal for key time periods can be strengthened, thereby constructing a target loss value that balances geometric plausibility and probability fit.

[0132] Based on this, the training system can obtain the predicted trajectory, uncertainty scale parameter, and mode prediction probability of each modality output. For each modality, the training system can determine the negative log-likelihood value at a future time based on the deviation between the predicted trajectory and the true trajectory at multiple future time points, as well as the corresponding uncertainty scale parameter.

[0133] The training system can use the weights corresponding to multiple future time points determined above to perform a weighted summation of the negative log-likelihood values ​​at each future time point, obtaining the probability error scalar corresponding to that mode. The training system can select the probability error scalar corresponding to the first target mode as the regression loss term, and combine it with the mode prediction probability and the classification loss term determined by the index of the first target mode to construct the final target loss value.

[0134] For example, the training system can utilize a mixture Laplace distribution model to determine the negative log-likelihood value at each time step based on the mean of the predicted trajectory output by each modality, the uncertainty scale parameter, and the mixture weights, combined with the position of the ground truth trajectory at multiple future time steps. Subsequently, the training system can multiply and sum the negative log-likelihood values ​​at each time step with the corresponding temporal weights to obtain the probability error scalar for each modality. Based on this, the training system can select the probability error scalar corresponding to the first target modality as the regression loss term, and add this regression loss term to the cross-entropy classification loss labeled with the first target modality identifier, as well as a smoothing diversity regularization term, to obtain the final target loss value.

[0135] In another optional embodiment, the training system can, based on the Gaussian distribution assumption, use the mean of the predicted trajectories output by each modality and the uncertainty scale parameter in the covariance matrix to determine the log-likelihood of the true trajectory at multiple future time points, and then take the negative of this log-likelihood value to obtain the negative log-likelihood value. Subsequently, the training system can multiply the negative log-likelihood values ​​at each time point by a preset temporal weight and accumulate them to obtain the probability error scalar for each modality. The training system can extract the probability error scalar corresponding to the first target modality, combine it with the cross-entropy loss with the first target modality as the class label, and construct the target loss value through a linear weighted summation.

[0136] In the above embodiments of this application, determining the negative log-likelihood values ​​corresponding to multiple future time points based on the predicted trajectories of each modality, uncertainty scale parameters, modality prediction probabilities, and true trajectories includes: determining the positions of multiple components based on the predicted trajectories of each modality; performing non-negative mapping processing on the uncertainty scale parameters of each modality to obtain scale parameters of multiple components; normalizing the modality prediction probabilities of each modality to obtain mixed distribution weights of multiple components; and determining the negative log-likelihood values ​​corresponding to multiple future time points based on the positions of multiple components, scale parameters, mixed distribution weights, and true trajectories. Specifically, determining the negative log-likelihood values ​​corresponding to multiple future time points based on the positions of multiple components, scale parameters, mixed distribution weights, and true trajectories includes: determining the log-likelihood values ​​of each component on different coordinate axes based on the positions of each component, scale parameters, mixed distribution weights, and true trajectories; synthesizing the log-likelihood values ​​of each component on different coordinate axes to obtain the log-likelihood values ​​of each component; and obtaining the negative log-likelihood values ​​corresponding to multiple future time points based on the log-likelihood values ​​of multiple components.

[0137] The aforementioned scaling parameter can refer to a positive value obtained after nonnegative mapping of the original uncertainty scaling parameter. This scaling parameter can be used to characterize the dispersion or variance of each component in the probability distribution model, and can serve as a scaling factor in the probability density function, directly determining the sharpness or smoothness of the probability distribution around the predicted trajectory point.

[0138] The aforementioned mixed distribution weights refer to the normalized values ​​of the modal prediction probabilities, which can be used to characterize the contribution ratio of each component to the final prediction distribution in a multi-component mixed probability model. These mixed distribution weights can be combined with the distribution functions of each component to construct a complex mixed probability density function that can simultaneously characterize multiple possible motion scenarios through weighted summation.

[0139] The aforementioned log-likelihood value refers to the log probability value generated by the current probability distribution for a given data point, based on the location, scale parameters, mixture distribution weights, and probability distribution model of each component, given true trajectory data. A larger log-likelihood value indicates a higher degree of fit between the predicted distribution and the actual observations.

[0140] In an alternative embodiment, considering the use of a mixed Laplace distribution to model the uncertainty of trajectory prediction, it is possible to more flexibly adapt to data with different distribution patterns and improve robustness to outliers. At the same time, through nonnegative mapping and normalization, the mathematical validity of the probability distribution can be ensured, thereby constructing a negative log-likelihood value that can accurately quantify the probability difference between the prediction results of each modality and the true trajectory.

[0141] Based on this, the training system can determine the position parameters of multiple components in spatial coordinates according to the predicted trajectories of each modality. The training system can apply non-negative mapping processing, such as a soft plus function or a logarithmic scale truncation exponential mapping, to the uncertain scale parameters output by each modality to obtain multi-component scale parameters that are guaranteed to be positive.

[0142] The training system can normalize the modal prediction probabilities of each modality output using the softmax function to obtain a multi-component mixed distribution weight that satisfies the sum-of-ones constraint. The training system can determine the log-likelihood value of each component on the horizontal and vertical axes separately, and can use the log-sum-exp algorithm to synthesize the log-likelihood values ​​of the same component on different axes to obtain the total log-likelihood value of that component.

[0143] The training system can sum the total log-likelihood values ​​of each component according to the mixed distribution weights to obtain the negative log-likelihood value corresponding to the future time, and then obtain a sequence of negative log-likelihood values ​​corresponding to multiple future time.

[0144] For example, the training system can decompose the predicted trajectories of each modality output into multiple Gaussian or Laplace components, applying the Softplus function to the uncertainty scale parameter to ensure its positive value. Furthermore, the training system can normalize the modality prediction probabilities using the Log-sum-exp (Logarithm of the Sum of Exponentials) operation, obtaining the mixed distribution weights of each component. For each component, the training system can calculate its log-likelihood value based on the true trajectory along the X and Y axes, and sum the log-likelihood values ​​corresponding to each coordinate axis to obtain the total log-likelihood value for that component. Finally, the training system can perform a weighted summation of the log-likelihood values ​​of each component and take the negative value to obtain the negative log-likelihood value at that moment.

[0145] In the above embodiments of this application, constructing a target loss value based on the probability error scalar corresponding to the first target mode includes: determining a second target mode from multiple modes based on the geometric error scalars corresponding to multiple modes; weighting the probability error scalars corresponding to the first target mode and the probability error scalars corresponding to the second target mode to obtain a current loss value; normalizing the current loss value based on the weight values ​​corresponding to the agent, the weight values ​​corresponding to the training data, and the weight values ​​of the effective time mask to obtain a target loss value; wherein, determining the second target mode from multiple modes based on the geometric error scalars corresponding to multiple modes includes: determining the minimum value among the geometric error scalars corresponding to multiple modes; and determining the mode corresponding to the minimum value as the second target mode.

[0146] The aforementioned second target mode can refer to the mode whose corresponding geometric error scalar reaches a smaller value among all candidate prediction modes. This second target mode can represent a prediction result that is generally closer to the true trajectory in the numerical space, and thus can serve as a complementary or alternative supervision object to the main supervision mode (i.e., the aforementioned first target mode), so as to provide additional gradient signals when geometric matching fails or when it is necessary to enhance numerical superiority.

[0147] The aforementioned current loss value can refer to the intermediate loss value obtained by linearly weighting and summing the probability error scalars corresponding to the first target mode and the second target mode according to preset weights. This current loss value combines the main supervision signal based on geometric semantic matching and the auxiliary supervision signal based on numerical superiority, thereby balancing the geometric rationality of trajectory prediction with statistical fitting accuracy, and can thus serve as the basis for sample-level normalization to obtain the final target loss value.

[0148] In an alternative embodiment, considering the introduction of a second target mode determined based on a geometric error scalar, a complementary supervision mechanism can be constructed, thereby retaining the regression gradient of the mode with better numerical error in addition to the main supervision path that prioritizes geometric matching, thus alleviating the target conflict under a single supervision criterion.

[0149] Based on this, the training system can determine the geometric error scalar corresponding to each mode, traverse all modes and compare the geometric error scalars of these modes, and select the mode with the smaller geometric error scalar as the second target mode. If there are multiple modes with equally small geometric error scalars, the training system can select the mode with the smaller label value as the second target mode according to a preset predetermined order.

[0150] The training system can obtain the probability error scalar corresponding to the first target mode and the probability error scalar corresponding to the second target mode, and use the preset first weight and second weight to weight these two probability error scalars respectively, and add the weighted results to obtain the current loss value.

[0151] The training system can extract the agent weight value corresponding to that moment, the sample weight value corresponding to the training data, and the weight value of the effective time mask. Based on the product of these weight values, the current loss value is normalized to obtain the target loss value.

[0152] For example, the training system can iterate through the geometric error scalars of all modes and select the mode corresponding to the smaller geometric error scalar as the second target mode. Based on this, the training system can extract the probability error scalars corresponding to the first and second target modes, and multiply them by preset primary and secondary weight coefficients respectively, then sum them to obtain the current loss value. Subsequently, the training system can use the sample weights, agent weights, and the sum of weights corresponding to valid moments in the time-effective mask to perform division normalization on the current loss value, obtaining the final target loss value.

[0153] In another alternative embodiment, the training system can determine the smaller value of the geometric error scalar for each modality and identify the modality that produces the smaller value as the second target modality. Subsequently, the training system can obtain the probability error scalars for the first and second target modalities respectively and perform a weighted linear combination operation to generate the current loss value. Then, the training system can filter out the weights of valid time points based on the time validity mask, combine the agent weights and sample weights, divide the current loss value by the cumulative weights of all valid time points, and complete the normalization to obtain the target loss value.

[0154] In the above embodiments of this application, the target loss value is obtained by normalizing the current loss value based on the weight value corresponding to the agent, the weight value corresponding to the training data, and the weight value of the effective time mask. This includes: normalizing the current loss value based on the weight value corresponding to the agent, the weight value corresponding to the training data, and the weight value of the effective time mask to obtain a regression loss value; constructing a classification loss value based on the modality prediction probabilities of multiple modalities, using the first target modality as the category label; and obtaining the target loss value based on the regression loss value and the classification loss value.

[0155] The aforementioned regression loss value can be a scalar value obtained by multiplying the current loss value by the agent weights, training data weights, and effective time mask weights, then dividing by the sum of these weights and normalizing. This regression loss value can be specifically used to measure the geometric fitting error between the predicted trajectory and the true trajectory in the location or probability distribution space, and thus can serve as an important component in constructing the target loss value.

[0156] The aforementioned classification loss value can refer to the cross-entropy loss between the predicted modality probability distribution and the true class label, using the index of the first target modality as the true class label. This classification loss value can be used to constrain the modality probability distribution output by the multimodal trajectory prediction model, thereby ensuring that the predicted higher probability modes are consistent with the master supervised modes determined based on geometric matching or error criteria.

[0157] In an alternative embodiment, considering that multi-scale weight normalization can eliminate the impact of data quality differences between different samples on gradient updates, and in addition, combining modality classification loss can make the probability distribution output by the multimodal trajectory prediction model consistent with the modality selection decision, thereby improving the interpretability and decision reliability of the prediction results in downstream interaction and planning tasks.

[0158] Based on this, the training system can obtain the agent weight value corresponding to the current sample, the sample weight value corresponding to the training data, and the weight value of the effective time mask. The training system can multiply the current loss value by the product of these three weight values, and then divide by the sum of the effective time mask, agent weight value, and sample weight value over all effective time steps to obtain the normalized regression loss value.

[0159] The training system can use the first target modality as the class label, apply the cross-entropy function to the modality prediction probabilities of multiple modalities to determine the classification loss value, and then add the normalized regression loss value to the classification loss value to obtain the final target loss value.

[0160] In the above embodiments of this application, obtaining a target loss value based on regression loss value and classification loss value includes: determining the acceleration sequence of multiple modalities based on the predicted trajectories of multiple modalities, and constructing a smoothing loss value based on the acceleration sequence of multiple modalities; and / or determining the trajectory distance between any two modalities based on the predicted trajectories of any two modalities, and constructing a diversity loss value based on the trajectory distance between any two modalities; obtaining the target loss value based on regression loss value, classification loss value, smoothing loss value and / or diversity loss value.

[0161] The aforementioned acceleration sequence can refer to a numerical sequence generated by performing second-order difference operations on the predicted trajectories of multiple modes in the time dimension, representing the instantaneous acceleration changes of each predicted mode at each moment within the prediction time period. This acceleration sequence can reflect the smoothness of the predicted trajectory's motion, and thus can be used to detect whether the prediction results exhibit violent fluctuations that violate physical properties.

[0162] The aforementioned smoothing loss value can refer to a scalar value obtained by weighting or summing the norms of the acceleration values ​​at each time step based on the acceleration sequence generated from the predicted trajectories of each modality. This smoothing loss value can be used to penalize unreasonable and drastic acceleration changes in the predicted trajectory, forcing the multimodal trajectory prediction model to learn a smooth motion trajectory that conforms to physical laws, thereby improving the physical feasibility and stability of the prediction results.

[0163] The aforementioned trajectory distance can refer to a scalar value obtained by determining the spatial distance between the predicted positions of any two different prediction modalities of the same agent at multiple future time steps and aggregating them along the time axis. This trajectory distance can be used to quantify the degree of difference between prediction results of different modalities, thus serving as a key basis for evaluating the diversity of multimodal predictions.

[0164] The aforementioned diversity loss value can refer to the loss function value constructed by introducing boundary constraints or penalty terms based on the trajectory distance between any two modalities. This diversity loss value can be used to prevent different modalities from converging to similar prediction results during training (i.e., modal collapse), thereby ensuring that the multimodal trajectory prediction model can retain multiple possible future scenario assumptions to maintain the richness and coverage of the multimodal output.

[0165] In one alternative embodiment, considering that introducing a smoothing loss value can constrain the continuity of the second derivative of the predicted trajectory, thereby suppressing high-frequency noise and improving the physical plausibility of the trajectory, while introducing a diversity loss value can prevent multimodal prediction collapse and enhance scene coverage by penalizing intermodal overlap, thus constructing a comprehensive optimization objective that takes into account kinematic smoothness, modal discriminability, and regression classification accuracy.

[0166] Based on this, the training system can determine the first-order difference of adjacent time step positions to obtain the velocity sequence for the predicted trajectory sequence of each mode, and then determine the first-order difference of adjacent time steps of the velocity sequence to obtain the acceleration sequence. The training system can perform a weighted average of the norms of each element of the acceleration sequence to obtain the smoothing loss value of that mode, and can sum the smoothing loss values ​​of all modes to obtain the overall smoothing loss value.

[0167] In another optional embodiment, for any two different modalities, the training system can take the average or larger value of the Euclidean distances of the predicted trajectories of the two different modalities at each time step as the trajectory distance of the modality pair. The training system can construct a boundary term based on a preset distance threshold to determine the sum or average of the trajectory distances of all modality pairs, thereby obtaining the diversity loss value.

[0168] In another alternative embodiment, when both smoothing loss and diversity loss are enabled, the training system can add the regression loss, classification loss, smoothing loss, and diversity loss by a preset coefficient to obtain the target loss value.

[0169] When only smoothing loss is enabled, the training system can add the regression loss, classification loss, and smoothing loss by a preset coefficient to obtain the target loss value.

[0170] When only the diversity loss value is enabled, the training system can add the regression loss value, classification loss value, and diversity loss value together by a preset coefficient to obtain the target loss value.

[0171] For ease of understanding, Figure 3 This is a detailed flowchart of a training method for an optional multimodal trajectory prediction model according to an embodiment of this application, as follows: Figure 3As shown, after a batch of sample input, the encoder can convert the samples into query features for the obstacle agent. Next, the multimodal decoder can combine the trajectories, scales, and modal probabilities of each modality to generate a multimodal representation. Then, the ground truth can be converted to relative coordinates and subjected to time-effective masking. Afterward, temporal point weights can be generated for subsequent weighting. After determining the temporal point weights, the loss can be calculated by weighting A(m) with NLL (Negative Log-Likelihood) and B(m) with L2 error, where A(m) represents the temporal weighted average negative log-likelihood scalar, and B(m) represents the temporal weighted L2 error summation scalar of the m-th modality. Subsequently, geometric matching and backtracking can be performed to generate the primary supervisory modality (i.e., the first target modality mentioned above) and the secondary supervisory modality (i.e., the second target modality mentioned above). Next, the regression losses of the primary supervisory mode and the secondary supervisory mode can be weighted and summed to perform intra-batch weighted normalization.

[0172] For example, the process of determining the batch regression loss during intra-batch weighted normalization can be illustrated by the following formula:

[0173] ;

[0174] In the formula, This represents the batch regression loss term, which can serve as the main regression loss component for training the model. It is not a single-modal loss, but a composite loss value obtained by weighted fusion of the losses from two different modality selection strategies (primary supervisory modality and auxiliary supervisory modality). This represents the main regression loss term. This represents the auxiliary regression loss term. It signifies the importance of sovereignty. This indicates the secondary weight.

[0175] For example, the process of determining the main regression loss term can be shown in the following formula:

[0176] ;

[0177] In the formula, Indicates the future time step number. This represents the set of valid time steps involved in the loss calculation, determined by the time validity mask. Represents the weights of time series points. This represents the negative log-likelihood. Indicates the first The modality at time... The predicted trajectory coordinates Represents the true locus at time [time]. The true coordinates, Indicates the first The modality at time... Uncertainty scale parameter, This represents a constant used to prevent the denominator from being zero and to ensure numerical stability.

[0178] The process of determining the weights of time series points can be shown in the following formula:

[0179] ;

[0180] In the formula, Represents the weights of time series points. Indicates the current time step. This represents the maximum time index.

[0181] Specifically, if the negative log-likelihood uses a mixed Laplace expression, the calculation process for the negative log-likelihood can be shown in the following formula:

[0182] ;

[0183] In the formula, Indicates the first Each component at time... Mixed weights, Represents the Laplace distribution density function. Indicates the first Each component at time... The mean (predicted trajectory).

[0184] Finally, the total loss (i.e. the target loss value mentioned above) can be determined by the main supervisor mode-to-mode cross-entropy, smoothing loss, and diversity loss.

[0185] The process of determining the smoothing loss can be shown in the following formula:

[0186] ;

[0187] In the formula, Indicates smoothing loss. express The velocity vector at time t, express The velocity vector at time t.

[0188] The process for determining diversity loss can be illustrated by the following formula:

[0189] .

[0190] In the formula, Indicating a loss of diversity, This represents the modified linear unit function. Indicates the penalty threshold. This represents the actual distance between modalities.

[0191] Figure 4 This is a schematic diagram of an optional supervisory modality application process according to an embodiment of this application, such as... Figure 4 As shown, the training system can determine whether a matching mode exists based on each input mode B(m). If a matching mode exists, the training system can determine the main supervised mode as the mode corresponding to the smaller identifier value. If no matching mode exists, the training system can determine the main supervised mode as the non-matching mode with a smaller B(m). Then, the training system can determine the main supervised mode and the auxiliary supervised mode separately, and determine the main regression loss corresponding to the main supervised mode and the auxiliary regression loss corresponding to the auxiliary supervised mode. The training system can perform intra-batch weighted normalization based on the main regression loss and the auxiliary regression loss. When the weight value of the auxiliary regression loss is 0, the training system can disable the auxiliary supervised mode.

[0192] Figure 5 This is a schematic diagram of an optional geometric matching determination principle according to an embodiment of this application, such as... Figure 5 As shown, the principle mainly includes geometric matching, single-moment error decomposition, checkpoint setting on the future time axis, three-level threshold configuration, threshold scaling with the agent's movement speed, and single-modal matching judgment.

[0193] Geometric matching involves comparing the predicted trajectory with the true trajectory at multiple sampling times. For example, the sampling times can be 3T / 8, 5T / 8, and the end of the future time length T. Each sampling time can be judged independently. If the matching condition is met at any time, the modal match can be determined.

[0194] Single-time error decomposition breaks down the error between the predicted trajectory and the true trajectory at a single time step into longitudinal and lateral deviations, so that threshold judgments can be made separately.

[0195] The checkpoint setting on the future timeline involves selecting several key discrete time points in the predicted future time series as observation benchmarks for evaluating trajectory deviations.

[0196] The three-level threshold configuration is a set of three different deviation tolerance thresholds preset for different scenarios or accuracy requirements, so as to flexibly adapt to matching judgment in complex road environments.

[0197] The threshold scaling with the agent's movement speed dynamically adjusts the strictness of the matching threshold based on the real-time driving speed of the obstacle agent, ensuring that the geometric matching standard in high-speed and low-speed scenarios conforms to physical intuition.

[0198] The single-mode matching judgment is performed independently for each independent predicted mode, based on the above geometric matching rules, and outputs a binary result indicating whether the mode is considered a successful match.

[0199] The process of speed scaling can be represented by the following formula:

[0200] ;

[0201] In the formula, Indicates the speed scaling factor. This represents the truncation function. Indicates the speed of the current obstacle. Indicates the lower limit threshold of speed. This indicates the upper limit threshold for speed.

[0202] According to an embodiment of this application, a training apparatus for a multimodal trajectory prediction model is provided. It should be noted that this apparatus can be used to execute the training method for the multimodal trajectory prediction model described above. The specific implementation process and application scenarios are the same as those in the above embodiment, and will not be repeated here. Figure 6 This is a schematic diagram of a training device for a multimodal trajectory prediction model according to an embodiment of this application, as shown below. Figure 6 As shown, the device includes:

[0203] The prediction module 602 is used to predict the trajectory of the agent's perception information in the training data using a multimodal trajectory prediction model, and to obtain prediction results of multiple modalities in the future time period. The prediction results include at least the predicted trajectory.

[0204] The matching module 604 is used to perform geometric matching between the predicted trajectory of each modality and the true trajectory in the training data based on the deviation of the predicted trajectory of each modality in multiple reference directions, so as to obtain the geometric matching result of each modality.

[0205] The first determining module 606 is used to determine the first target mode from multiple modes based on the geometric matching results of multiple modes.

[0206] Module 608 is used to construct the target loss value based on the prediction results and true value trajectory corresponding to the first target mode.

[0207] The update module 610 is used to update the multimodal trajectory prediction model based on the target loss value.

[0208] Furthermore, the matching module is also used to: obtain predicted trajectory points for multiple sampling times from the predicted trajectories of each mode based on multiple sampling times within a future time period, and obtain true trajectory points for multiple sampling times from the true trajectory; match the predicted trajectory points and true trajectory points for each sampling time with the true trajectory points based on the deviations of the predicted trajectory points and true trajectory points for each sampling time in multiple reference directions to obtain the matching results corresponding to each sampling time; and determine the geometric matching results for each mode based on the matching results corresponding to multiple sampling times.

[0209] Furthermore, the matching module is also used to: construct matching thresholds corresponding to multiple sampling times, wherein the matching thresholds include: thresholds corresponding to multiple reference directions; match the deviations of the predicted trajectory points and true trajectory points at each sampling time in multiple reference directions with the matching thresholds corresponding to each sampling time to obtain the matching results corresponding to each sampling time; wherein, constructing matching thresholds corresponding to multiple sampling times in a future time period includes: determining the speed of the agent; adjusting the initial thresholds corresponding to multiple sampling times based on the speed to obtain the matching thresholds corresponding to multiple sampling times.

[0210] Furthermore, the matching module is also used to: determine that the matching result at the sampling time is a successful match between the predicted trajectory point and the ground truth trajectory point at the sampling time when the deviation in each reference direction is less than or equal to the threshold corresponding to each reference direction; and determine that the matching result at the sampling time is a failed match between the predicted trajectory point and the ground truth trajectory point at the sampling time when the deviation in any reference direction is greater than the threshold corresponding to that reference direction. Specifically, determining the geometric matching result for each mode based on the matching results at multiple sampling times includes: determining that the geometric matching result for each mode represents a successful match between the predicted trajectory point and the ground truth trajectory point at any sampling time when the matching result at any sampling time is a successful match between the predicted trajectory point and the ground truth trajectory point at the sampling time; and determining that the geometric matching result for each mode represents a failed match between the predicted trajectory point and the ground truth trajectory point at any of the multiple sampling times.

[0211] Furthermore, the device also includes: a second determining module, used to determine the heading angle based on adjacent displacements in the true trajectory; to determine the position deviation at each sampling time based on the displacement between the predicted trajectory point and the true trajectory point at each sampling time; and to decompose the position deviation at each sampling time in multiple reference directions based on the heading angle to obtain the deviation between the predicted trajectory point and the true trajectory point at each sampling time in multiple reference directions.

[0212] Furthermore, the first determining module is also used to: determine a first target mode from the modes corresponding to at least one predicted trajectory when the geometric matching results of multiple modes indicate that at least one predicted trajectory matches the true trajectory successfully, wherein the identifier value of the first target mode is less than the identifier value of other modes, and the other modes are modes other than the first target mode among the modes corresponding to at least one predicted trajectory.

[0213] Furthermore, the device also includes: a third determining module, used to determine, based on the true trajectory, multiple future time points within a future time period when the geometric matching results of multiple modes indicate that the predicted trajectories and true trajectories of multiple modes fail to match; wherein the weight values ​​corresponding to the multiple future time points satisfy a monotonically non-decreasing relationship; determine the geometric error scalar corresponding to each mode based on the predicted trajectories, true trajectories, and weight values ​​corresponding to the multiple future time points; wherein the geometric error scalar is used to characterize the deviation between the predicted trajectories of each mode and the true trajectory in geometric space; determine the minimum value among the geometric error scalars corresponding to multiple modes, and determine the mode corresponding to the minimum value as the first target mode.

[0214] Furthermore, the third determining module is also used to: transform the true trajectory to the coordinate system corresponding to multiple modes to obtain the transformed trajectory; generate a time-valid mask based on the transformed trajectory, wherein the time-valid mask is used to characterize whether the true trajectory points corresponding to multiple future time moments in the true trajectory are valid data; determine the weight values ​​corresponding to multiple future time moments based on the time-valid mask, the weight values ​​corresponding to the agent, and the weight values ​​corresponding to the training data; wherein, based on the predicted trajectory, the true trajectory, and the weight values ​​corresponding to multiple future time moments of each mode, determine the geometric error scalar corresponding to each mode, including: determining the Euclidean distance corresponding to multiple future time moments based on the predicted trajectory and the true trajectory of each mode; and weighting the Euclidean distance corresponding to multiple future time moments based on the weight values ​​corresponding to multiple future time moments to obtain the geometric error scalar corresponding to each mode.

[0215] Furthermore, the prediction results also include: uncertainty scale parameters and modality prediction probabilities; the construction module is also used to: determine the negative log-likelihood values ​​corresponding to multiple future time points based on the predicted trajectories of each modality, uncertainty scale parameters, modality prediction probabilities, and true trajectories; weight the negative log-likelihood values ​​corresponding to multiple future time points based on the weight values ​​corresponding to multiple future time points to obtain the probability error scalar corresponding to each modality, wherein the probability error scalar is used to characterize the degree of fit between the predicted trajectory of each modality and the true trajectory in the probability space; and construct the target loss value based on the probability error scalar corresponding to the first target modality.

[0216] Furthermore, the construction module is also used to: determine the positions of multiple components based on the predicted trajectories of each modality; perform non-negative mapping on the uncertainty scale parameters of each modality to obtain the scale parameters of multiple components; normalize the modality prediction probabilities of each modality to obtain the mixed distribution weights of multiple components; and determine the negative log-likelihood values ​​corresponding to multiple future time points based on the positions, scale parameters, mixed distribution weights, and true trajectories of multiple components. Specifically, determining the negative log-likelihood values ​​corresponding to multiple future time points based on the positions, scale parameters, mixed distribution weights, and true trajectories of multiple components includes: determining the log-likelihood values ​​of each component on different coordinate axes based on the positions, scale parameters, mixed distribution weights, and true trajectories of each component; synthesizing the log-likelihood values ​​of each component on different coordinate axes to obtain the log-likelihood values ​​of each component; and obtaining the negative log-likelihood values ​​corresponding to multiple future time points based on the log-likelihood values ​​of multiple components.

[0217] Furthermore, the construction module is also used to: determine a second target mode from multiple modes based on the geometric error scalars corresponding to multiple modes; perform weighted processing on the probability error scalars corresponding to the first target mode and the probability error scalars corresponding to the second target mode to obtain the current loss value; and normalize the current loss value based on the weight value corresponding to the agent, the weight value corresponding to the training data, and the weight value of the effective time mask to obtain the target loss value; wherein, determining the second target mode from multiple modes based on the geometric error scalars corresponding to multiple modes includes: determining the minimum value among the geometric error scalars corresponding to multiple modes; and determining the mode corresponding to the minimum value as the second target mode.

[0218] Furthermore, the construction module is also used to: normalize the current loss value based on the weight value corresponding to the agent, the weight value corresponding to the training data, and the weight value of the effective time mask to obtain the regression loss value; construct the classification loss value based on the modality prediction probability of multiple modalities with the first target modality as the category label; and obtain the target loss value based on the regression loss value and the classification loss value.

[0219] Furthermore, the construction module is also used to: determine the acceleration sequence of multiple modalities based on the predicted trajectories of multiple modalities, and construct a smoothing loss value based on the acceleration sequence of multiple modalities; and / or, determine the trajectory distance between any two modalities based on the predicted trajectories of any two modalities, and construct a diversity loss value based on the trajectory distance between any two modalities; and obtain the target loss value based on the regression loss value and the classification loss value, as well as the smoothing loss value and / or the diversity loss value.

[0220] Embodiments of this application also provide an electronic device, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0221] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0222] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0223] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0224] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0225] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0226] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0227] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0228] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0229] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0230] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A training method for a multimodal trajectory prediction model, characterized in that, include: A multimodal trajectory prediction model is used to predict the trajectory of an agent's perception information in training data to obtain prediction results for multiple modalities within a future time period. The prediction results include at least the predicted trajectory. Based on the deviations of the predicted trajectories of each modality from the ground truth trajectories in the training data in multiple reference directions, the predicted trajectories of each modality are geometrically matched with the ground truth trajectories in the training data to obtain the geometric matching results of each modality. Based on the geometric matching results of the multiple modes, a first target mode is determined from the multiple modes; Based on the prediction result corresponding to the first target mode and the true value trajectory, a target loss value is constructed; The multimodal trajectory prediction model is updated based on the target loss value.

2. The method according to claim 1, characterized in that, Based on the deviations of the predicted trajectories of each modality from the ground truth trajectories in the training data in multiple reference directions, the predicted trajectories of each modality are geometrically matched with the ground truth trajectories in the training data to obtain the geometric matching results for each modality, including: Based on multiple sampling times within the future time period, the predicted trajectory points of the multiple sampling times are obtained from the predicted trajectories of each modality, and the true trajectory points of the multiple sampling times are obtained from the true trajectory. Based on the deviation between the predicted trajectory points and the true trajectory points at each sampling time in the multiple reference directions, the predicted trajectory points and the true trajectory points at each sampling time are matched to obtain the matching results corresponding to each sampling time. Based on the matching results corresponding to the multiple sampling times, the geometric matching results of each mode are determined.

3. The method according to claim 2, characterized in that, Based on the deviations between the predicted trajectory points and the true trajectory points at each sampling time along the multiple reference directions, the predicted trajectory points and the true trajectory points at each sampling time are matched to obtain the matching results corresponding to each sampling time, including: Construct matching thresholds corresponding to the plurality of sampling times, wherein the matching thresholds include: thresholds corresponding to the plurality of reference directions; The deviations between the predicted trajectory points and the true trajectory points at each sampling time in the multiple reference directions are matched with the matching thresholds corresponding to each sampling time to obtain the matching results corresponding to each sampling time. The process of constructing matching thresholds corresponding to multiple sampling times within the future time period includes: Determine the speed of the agent; The initial thresholds corresponding to the plurality of sampling times are adjusted based on the speed to obtain the matching thresholds corresponding to the plurality of sampling times.

4. The method according to claim 2, characterized in that, The deviations between the predicted trajectory points and the true trajectory points at each sampling time in the multiple reference directions are matched with the matching thresholds corresponding to each sampling time to obtain the matching results for each sampling time, including: If the deviation in each reference direction is less than or equal to the threshold corresponding to each reference direction, the matching result at that sampling time is determined to be a successful match between the predicted trajectory point at that sampling time and the true trajectory point at that sampling time. If the deviation in any reference direction is greater than the threshold corresponding to that reference direction, the matching result at that sampling time is determined to be that the predicted trajectory point at that sampling time and the true trajectory point at that sampling time have failed to match. Specifically, the geometric matching result of each mode is determined based on the matching results corresponding to the multiple sampling times, including: If the matching result at any sampling time is that the predicted trajectory point at that sampling time and the true trajectory point at that sampling time are successfully matched, the geometric matching result of each mode is determined to indicate that the predicted trajectory of each mode is successfully matched with the true trajectory. If the matching result at the multiple sampling times is that the predicted trajectory points at the multiple sampling times fail to match the true trajectory points at the multiple sampling times, the geometric matching result of each mode is determined to indicate that the predicted trajectory of each mode fails to match the true trajectory.

5. The method according to claim 2, characterized in that, The method further includes: The heading angle is determined based on adjacent displacements in the true trajectory; Based on the displacement between the predicted trajectory point and the true trajectory point at each sampling time, the position deviation at each sampling time is determined; Based on the heading angle, the position deviation at each sampling time is decomposed in the multiple reference directions to obtain the deviation between the predicted trajectory point and the true trajectory point at each sampling time in the multiple reference directions.

6. The method according to any one of claims 1 to 5, characterized in that, Based on the geometric matching results of the multiple modes, a first target mode is determined from the multiple modes, including: If the geometric matching results of the plurality of modes indicate that at least one predicted trajectory successfully matches the ground truth trajectory, the first target mode is determined from the modes corresponding to the at least one predicted trajectory, wherein the identifier value of the first target mode is less than the identifier value of other modes, and the other modes are modes other than the first target mode among the modes corresponding to the at least one predicted trajectory.

7. The method of claim 6, characterized in that, The method further includes: When the geometric matching results of the multiple modalities indicate that the predicted trajectories of the multiple modalities fail to match the true trajectories, the weight values ​​corresponding to multiple future moments within the future time period are determined based on the true trajectories, wherein the weight values ​​corresponding to the multiple future moments satisfy a monotonically non-decreasing relationship. Based on the predicted trajectory of each mode, the true trajectory, and the weight values ​​corresponding to the multiple future times, a geometric error scalar corresponding to each mode is determined, wherein the geometric error scalar is used to characterize the deviation between the predicted trajectory of each mode and the true trajectory in geometric space. The minimum value among the geometric error scalars corresponding to the plurality of modes is determined, and the mode corresponding to the minimum value is determined as the first target mode.

8. The method according to claim 7, characterized in that, Based on the true value trajectory, determine the weight values ​​corresponding to multiple future moments within the future time period, including: The true trajectory is transformed to the coordinate system corresponding to the multiple modes to obtain the transformed trajectory; Based on the transformed trajectory, a time-valid mask is generated, wherein the time-valid mask is used to characterize whether the ground truth trajectory points corresponding to the plurality of future times in the ground truth trajectory are valid data; Based on the time-valid mask, the weight value corresponding to the agent, and the weight value corresponding to the training data, the weight values ​​corresponding to the multiple future time moments are determined. Specifically, based on the predicted trajectory of each mode, the true trajectory, and the weight values ​​corresponding to the multiple future time moments, the geometric error scalar corresponding to each mode is determined, including: Based on the predicted trajectory of each modality and the true trajectory, the Euclidean distance corresponding to the multiple future moments is determined; Based on the weight values ​​corresponding to the multiple future time points, the Euclidean distances corresponding to the multiple future time points are weighted to obtain the geometric error scalar for each mode.

9. The method according to any one of claims 1 to 5, characterized in that, The prediction results also include: uncertainty scale parameters and mode prediction probabilities; based on the prediction results corresponding to the first target mode and the true value trajectory, a target loss value is constructed, including: Based on the predicted trajectory of each mode, the uncertainty scale parameter and the mode prediction probability, the true trajectory is used to determine the negative log-likelihood values ​​corresponding to multiple future moments. Based on the weight values ​​corresponding to the multiple future time points, the negative log-likelihood values ​​corresponding to the multiple future time points are weighted to obtain the probability error scalar corresponding to each mode. The probability error scalar is used to characterize the degree of fit between the predicted trajectory of each mode and the true trajectory in the probability space. The target loss value is constructed based on the probability error scalar corresponding to the first target mode.

10. The method according to claim 9, characterized in that, Based on the predicted trajectories of each modality, the uncertainty scale parameter, and the modality prediction probability, the true trajectory is used to determine the negative log-likelihood values ​​corresponding to the multiple future time points, including: Based on the predicted trajectories of each mode, the positions of multiple components are determined; The scale parameters of the multiple components are obtained by performing nonnegative mapping on the uncertainty scale parameters of each mode. The modal prediction probabilities of each modality are normalized to obtain the mixed distribution weights of the multiple components; Based on the location, scale parameter, and mixed distribution weight of the multiple components, as well as the true value trajectory, the negative log-likelihood value corresponding to the multiple future time moments is determined; Specifically, based on the positions, scale parameters, and mixed distribution weights of the multiple components, and the true trajectory, the negative log-likelihood values ​​corresponding to the multiple future time moments are determined, including: Based on the location, scale parameter, and mixed distribution weight of each component, as well as the true value trajectory, the log-likelihood value of each component on different coordinate axes is determined. The log-likelihood values ​​of each component on different coordinate axes are combined to obtain the log-likelihood value of each component; Based on the log-likelihood values ​​of the multiple components, the negative log-likelihood values ​​corresponding to the multiple future times are obtained.

11. The method according to claim 9, characterized in that, Based on the probability error scalar corresponding to the first target mode, a target loss value is constructed, including: Based on the geometric error scalars corresponding to the multiple modes, a second target mode is determined from the multiple modes; The probability error scalar corresponding to the first target mode and the probability error scalar corresponding to the second target mode are weighted to obtain the current loss value; Based on the weight values ​​corresponding to the agent, the weight values ​​corresponding to the training data, and the weight values ​​of the effective time mask, the current loss value is normalized to obtain the target loss value. The determination of the second target mode from the plurality of modes based on the geometric error scalars corresponding to the plurality of modes includes: Determine the minimum value among the geometric error scalars corresponding to the multiple modes; The mode corresponding to the minimum value is determined as the second target mode.

12. The method according to claim 11, characterized in that, Based on the weight values ​​corresponding to the agent, the weight values ​​corresponding to the training data, and the weight values ​​of the effective time mask, the current loss value is normalized to obtain the target loss value, including: Based on the weight values ​​corresponding to the agent, the weight values ​​corresponding to the training data, and the weight values ​​of the effective time mask, the current loss value is normalized to obtain the regression loss value. Using the first target modality as the category label, a classification loss value is constructed based on the modality prediction probabilities of the multiple modalities; The target loss value is obtained based on the regression loss value and the classification loss value.

13. The method according to claim 12, characterized in that, Based on the regression loss value and the classification loss value, the target loss value is obtained, including: Based on the predicted trajectories of the multiple modes, the acceleration sequences of the multiple modes are determined, and a smoothing loss value is constructed based on the acceleration sequences of the multiple modes; and / or, Based on the predicted trajectories of any two modalities, determine the trajectory distance between any two modalities, and construct a diversity loss value based on the trajectory distance between any two modalities; The target loss value is obtained based on the regression loss value, the classification loss value, the smoothing loss value, and / or the diversity loss value.

14. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 13.

Images