Training method and device of trajectory prediction model and medium

Abstract

The invention discloses a training method and device of a trajectory prediction model and a medium. Comprising the following steps: predicting a training sample according to an initial trajectory prediction model to obtain multiple types of trajectory prediction results; target guiding loss, collision avoidance loss and trajectory classification loss are determined according to the trajectory prediction results of the multiple categories; determining target loss according to the target guiding loss, the collision avoidance loss and the track classification loss; and performing iterative parameter adjustment on the initial trajectory prediction model according to the target loss corresponding to the training sample in the sample set to obtain a trained target trajectory prediction model. According to the method, the target direction of the terminal point is used as a constraint condition of trajectory planning through the target guiding loss; the passable area except the obstacle is used as the constraint condition of trajectory planning through the collision avoidance loss, and the classification truth value trajectory is used as the constraint condition of trajectory planning through the trajectory classification loss, so that the accuracy of trajectory prediction according to the trajectory prediction model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles (UAVs), and in particular to a training method, device, and medium for a trajectory prediction model. Background Art

[0002] Drones are widely used in agriculture, exploration, disaster relief, public safety, entertainment, future warfare and other fields. In many complex scenarios, drones are required to complete flight missions autonomously and efficiently. In different flight missions, there are also different requirements for drone flight strategies. For example, in disaster relief missions, the requirement for flight strategy is mainly based on time limit, while in entertainment missions, the requirement for flight strategy is mainly based on safety.

[0003] In the UAV autonomous navigation system, the trajectory prediction model is mainly relied on to predict the current flight trajectory. Therefore, in order to ensure the accuracy of the UAV's flight trajectory under different flight strategies, it is necessary to improve the training effect of the trajectory prediction model. Summary of the Invention

[0004] The present invention provides a method, device and medium for training a trajectory prediction model to improve the training effect of the trajectory prediction model.

[0005] According to one aspect of the present invention, a method for training a trajectory prediction model is provided, comprising:

[0006] Predict the training samples according to the initial trajectory prediction model to obtain trajectory prediction results of multiple categories;

[0007] Determining a target guidance loss, a collision avoidance loss, and a trajectory classification loss based on the trajectory prediction results of the multiple categories; wherein the target guidance loss is determined based on the trajectory direction of the trajectory prediction result and the target direction of the end point, the collision avoidance loss is determined based on the distance between the trajectory point position of the trajectory prediction result and the obstacle position, and the trajectory classification loss is determined based on the trajectory prediction results of different categories and the corresponding true category labels;

[0008] determining a target loss based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss;

[0009] The initial trajectory prediction model is iteratively parameterized according to the target loss corresponding to the training sample in the sample set to obtain a trained target trajectory prediction model.

[0010] According to another aspect of the present invention, a training device for a trajectory prediction model is provided, comprising:

[0011] The sample trajectory prediction module is used to predict the training samples based on the initial trajectory prediction model and obtain trajectory prediction results of multiple categories;

[0012] a multi-loss determination module, configured to determine a target guidance loss, a collision avoidance loss, and a trajectory classification loss based on the trajectory prediction results of the multiple categories; wherein the target guidance loss is determined based on the trajectory direction of the trajectory prediction result and the target direction of the end point, the collision avoidance loss is determined based on the distance between the trajectory point position of the trajectory prediction result and the obstacle position, and the trajectory classification loss is determined based on the trajectory prediction results of different categories and the corresponding true category labels;

[0013] a target loss determination module, configured to determine a target loss based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss;

[0014] The model parameter iterative adjustment module is used to iteratively adjust the parameters of the initial trajectory prediction model according to the target loss corresponding to the training sample in the sample set to obtain a trained target trajectory prediction model.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] At least one processor; and a memory in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the trajectory prediction model training method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the training method of the trajectory prediction model described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention uses the target direction of the end point as a constraint condition for trajectory planning through goal guidance loss; uses the traversable area excluding obstacles as a constraint condition for trajectory planning through collision avoidance loss; and uses the classified true value trajectory as a constraint condition for trajectory planning through trajectory classification loss, thereby improving the accuracy of trajectory prediction model training and thus improving the accuracy of trajectory prediction based on the trajectory prediction model.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a trajectory prediction model training method provided according to an embodiment of the present invention;

[0022] Figure 2 is a flowchart of another trajectory prediction model training method provided by an embodiment of the present invention;

[0023] Figure 3 2 is a schematic structural diagram of a trajectory prediction model training device provided according to an embodiment of the present invention;

[0024] Figure 4 3 is a schematic diagram of the structure of an electronic device for implementing the training method of the trajectory prediction model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "candidate", "target", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices.

[0027] Figure 1A flowchart of a trajectory prediction model training method is provided for an embodiment of the present invention. This embodiment is applicable to the case of optimizing and training a trajectory prediction model for guiding UAV flight. The method can be executed by a trajectory prediction model training device. The trajectory prediction model training device can be implemented in the form of hardware and / or software. The trajectory prediction model training device can be configured in a server with computing capabilities. Figure 1 As shown, the method includes:

[0028] S110 , predicting the training samples according to the initial trajectory prediction model to obtain trajectory prediction results of multiple categories.

[0029] The initial trajectory prediction model can be an untrained or pre-trained network architecture. The network architecture can be a classification network, and the present invention does not limit the network type of the classification network. The training samples are trajectory paths and scene images obtained by controlling the flight of the drone in different scenarios.

[0030] Specifically, scene images of different scenes in the training samples are input into the initial trajectory prediction model, and the output of the initial trajectory prediction model is a plurality of different categories of trajectory prediction results, where different categories of trajectory prediction results represent different trajectory direction angles, and the number of output categories of trajectory prediction results can be adjusted according to actual training needs, which is not limited here. For example, the number of categories is 20. Exemplarily, scene images of each scene in the training sample are respectively input into the initial trajectory prediction model, and the initial trajectory prediction model gives trajectory prediction results for obstacle avoidance flight at multiple directions and angles, each trajectory prediction result including multiple trajectory point position information or multiple trajectory velocity vectors.

[0031] S120 : Determine a target guidance loss, a collision avoidance loss, and a trajectory classification loss according to trajectory prediction results of multiple categories.

[0032] Because the model training in this embodiment is supervised, the training samples include pre-determined drone trajectory paths for different scenarios. These drone trajectory paths serve as the true trajectory values, guiding the trajectory prediction results output by the initial trajectory prediction model toward the true trajectory values. Therefore, different types of losses are determined based on the trajectory prediction results for multiple categories of scenarios obtained by the initial trajectory prediction model, as well as the true trajectory values ​​for each scenario. These different types of losses are coordinated from multiple dimensions to simultaneously optimize the accuracy, safety, and goal-oriented nature of the training trajectory predictions.

[0033] Different types of losses include goal-oriented loss, collision avoidance loss, and trajectory classification loss. The goal-oriented loss is determined based on the trajectory direction of the trajectory prediction result and the target direction of the end point, wherein the trajectory direction of the trajectory prediction result represents the predicted flight direction of the UAV, which can be a direction vector from the trajectory start point to the end point in the trajectory prediction result, or multiple direction vectors from the trajectory start point to the end point in different time periods in the trajectory prediction result. The target direction of the end point refers to the flight destination direction of the UAV in different scenarios. The flight destination direction can be pre-marked in different scenarios, or determined based on the end point direction of the UAV trajectory path in each scenario in the sample. Exemplarily, the goal-oriented loss can be determined based on the cumulative deviation between the trajectory direction of each type of trajectory prediction result corresponding to each scenario and the target direction of the end point. The trajectory prediction result is guided by the goal-oriented loss to move toward the desired target direction, thereby optimizing the goal orientation of the trajectory prediction.

[0034] The collision avoidance loss is determined based on the distance between the trajectory point position of the trajectory prediction result and the obstacle position, wherein the trajectory point position of the trajectory prediction result is used to represent the predicted flight position of the drone, and the obstacle position is determined based on the scene images of different scenes. For example, the obstacle position is determined based on the obstacle recognition result of the scene image. Exemplarily, the collision trajectory loss can be determined based on the cumulative distance between each trajectory point position and the obstacle position of each type of trajectory prediction result corresponding to each scene. The guidance of the collision avoidance loss enables the trajectory prediction result to safely avoid obstacles, thereby optimizing the safety of the trajectory prediction.

[0035] The trajectory classification loss is determined based on the trajectory prediction results for different categories and the corresponding true class labels. The drone trajectory paths for each scenario in the training samples correspond to different modalities. Different modalities are used to represent different flight strategies, such as an aggressive strategy (which minimizes flight time); a safety strategy (which maintains a safe distance from obstacles); an intermediate strategy (which combines short flight time and safety), and so on. Other flight strategies can be determined based on these strategies, and flight strategies can be adjusted based on actual scenarios. There is no limit on the number of modalities. Exemplarily, the drone trajectory paths corresponding to different modalities for each scenario are pre-determined as the true class labels for each modality. Trajectory prediction results for different categories are associated with different modalities. Exemplarily, when the trajectory prediction model outputs trajectory prediction results for different categories, the corresponding output information includes information about the modality to which they belong. For example, the trajectory prediction model outputs trajectory prediction results for different flight directions and angles under different modalities, meaning that each modality corresponds to multiple different types of trajectory prediction results. Exemplarily, the trajectory classification loss is determined based on the cumulative deviation between the trajectory prediction results for each modality and the corresponding true class labels for each scenario. Through the guidance of trajectory classification loss, the trajectory prediction results can optimize the deviation between the prediction results of discrete trajectory categories and the true value, thereby optimizing the accuracy of trajectory prediction.

[0036] S130: Determine target loss according to target guidance loss, collision avoidance loss, and trajectory classification loss.

[0037] The target loss is determined based on the combined results of the target-oriented loss, collision avoidance loss, and trajectory classification loss. Specifically, the target loss is determined based on the weighted sum of the target-oriented loss, collision avoidance loss, and trajectory classification loss. For example, L_total = α·L_target + β·L_collision + γ·L_classification, where L_total represents the target-oriented loss, α represents the weight of the target-oriented loss, L_target represents the collision avoidance loss, β represents the weight of the collision avoidance loss, L_classification represents the trajectory classification loss, and γ represents the weight of the trajectory classification loss. The weight values ​​can be determined based on the actual scenario requirements to balance the importance of losses in different dimensions.

[0038] S140 , iteratively adjust the parameters of the initial trajectory prediction model according to the target loss corresponding to the training samples in the sample set to obtain a trained target trajectory prediction model.

[0039] The parameters of the initial trajectory prediction model are continuously adjusted according to the target loss obtained from the training samples in the sample set. For example, all training samples in the sample set are divided into multiple batch samples (batches), and the parameters in the initial trajectory prediction model are adjusted once according to the target loss obtained from the training samples in each batch. Then, the parameter adjustment direction is determined according to the changing trend of the target loss obtained from the training samples in the next batch. By continuously iteratively training all training samples in the sample set until the loss converges or the preset number of iterations is reached, a trained target trajectory prediction model is obtained.

[0040] Optionally, after obtaining the trained target trajectory prediction model, the method further includes:

[0041] The target trajectory prediction model is used in the UAV's autonomous navigation system to determine trajectory prediction results for different modes. This allows the model to be used to perform obstacle avoidance flights using different flight strategies. Through loss training in the three dimensions mentioned above, flight trajectories that better fit the actual environment are generated.

[0042] The technical solution of this embodiment uses the target direction of the end point as a constraint condition for trajectory planning through goal-oriented loss; uses the traversable area excluding obstacles as a constraint condition for trajectory planning through collision avoidance loss; and uses the classified true value trajectory as a constraint condition for trajectory planning through trajectory classification loss, thereby improving the accuracy of trajectory prediction model training and thus improving the accuracy of trajectory prediction based on the trajectory prediction model.

[0043] Figure 2 This is a flow chart of another method for training a trajectory prediction model provided by an embodiment of the present invention. This embodiment further refines the multi-loss determination process in the above embodiment. Figure 2 As shown, the method includes:

[0044] S210 , predicting the training samples according to the initial trajectory prediction model to obtain trajectory prediction results of multiple categories.

[0045] S220 , determining the trajectory direction according to the trajectory starting point and trajectory end point in the trajectory prediction results of each category; and determining the target guidance loss according to the angular deviation between the trajectory direction and the target direction corresponding to the training sample.

[0046] The trajectory direction of each category is determined based on the trajectory starting point and trajectory end point in the trajectory prediction results of each category in each sample scene. The target direction of the corresponding end point of each sample scene is then determined. The cumulative angular deviation of the sample scene is determined based on the sum of the angular deviations between the trajectory direction and the target direction of all categories in each sample scene. The target guidance loss is determined based on the cumulative angular deviations of all sample scenes.

[0047] Exemplarily, the initial trajectory prediction model outputs trajectory prediction results for multiple categories corresponding to multiple modalities, and the trajectory prediction results corresponding to each modality with the highest prediction probability value are determined to be the target trajectory prediction result corresponding to the modality. Since the output result of the trained trajectory prediction model is a trajectory prediction result corresponding to each modality, in order to improve the accuracy of the target-oriented loss, the target trajectory prediction result corresponding to each modality is used in the calculation during the determination of the target-oriented loss. For example, the target trajectory prediction result with the highest prediction probability value corresponding to each modality obtained for each sample scene is determined, and the sum of the angular deviations between the trajectory direction and the target direction of the target trajectory prediction results of all modalities of each sample scene is determined to be the cumulative angular deviation of the sample scene. The target-oriented loss of the current iteration is then determined based on the cumulative angular deviations of all sample scenes in a batch.

[0048] In a feasible embodiment, S220 includes:

[0049] Determine the cosine angle between the trajectory direction of each mode output by the trajectory prediction model corresponding to each training sample and the target direction corresponding to the training sample;

[0050] Determining a modal direction loss value of each mode output by the trajectory prediction model corresponding to each training sample according to the difference between the first value and the cosine angle;

[0051] Determine the sum of the modal directional loss values ​​of each mode output by the trajectory prediction model corresponding to each training sample as the total directional loss;

[0052] The goal-oriented loss is determined by the ratio of the total direction loss to the total number of training samples.

[0053] Among them, the specific value of the first value can be determined according to the numerical range of the goal-oriented loss during actual scene training. There is no restriction on its specific numerical value. For example, the first value can be 1.

[0054] Optionally, the goal-oriented loss is determined according to the following formula:

[0055] ;

[0056] in, represents the goal-oriented loss, represents the sample index of the training sample, k∈[1,N], represents the modality index of the output trajectory modality of the trajectory prediction model, m∈[1,M], Indicates the The trajectory prediction model outputs the first The trajectory direction of each mode, Indicates the The target direction corresponding to the training samples.

[0057] The sample index is used to indicate the sample sequence number involved in the current iteration, for example, N is the total number of sample scenes in a batch; the modal index is used to indicate the different flight strategy identifiers output by the predetermined trajectory prediction model, for example, M represents the total number of flight strategies. ,in, Indicates the The trajectory prediction model outputs the first The trajectory end point position information of the trajectory prediction result of each mode, Indicates the The trajectory prediction model outputs the first The trajectory starting point position information of the trajectory prediction result of each mode, The determination can be made based on pre-labeled information of each sample scene, or based on the endpoint information of the true value of the trajectory in the sample. It indicates the cosine of the angle between the trajectory direction and the target direction. The smaller the value of the cosine of the angle, the smaller the deviation between the trajectory direction and the target direction.

[0058] S230. Identify training samples based on a pre-trained semantic segmentation model to determine obstacle location information; determine collision avoidance parameters based on the trajectory point location information and obstacle location information in the trajectory prediction results of each category; and determine the collision trajectory loss based on the collision avoidance parameters and a preset safety distance threshold.

[0059] A semantic segmentation model is used to identify the sample scene images in the training samples to obtain the obstacle position information in the sample scene images. The cumulative collision deviation of the sample scene is determined based on the sum of the distances between the trajectory point position information and the obstacle position information in the trajectory prediction results of all categories of each sample scene. The collision trajectory loss is determined based on the cumulative collision deviation of all sample scenes and the safety distance threshold.

[0060] Exemplarily, based on the above example, the target trajectory prediction result with the highest prediction probability value corresponding to each modality obtained for each sample scene is determined, and the sum of the minimum distances between the trajectory point position information and the obstacle position information of the target trajectory prediction result of each modality of each sample scene is determined as the collision avoidance parameter of the modality, and the collision trajectory loss is determined based on the comparison result of the collision avoidance parameter of each modality with the safety distance threshold. For example, if the collision avoidance parameter of a modality is greater than or equal to the safety distance threshold, the collision trajectory loss of the modality is determined to be 0, otherwise it is determined based on the pre-established mapping relationship between the collision avoidance parameter and the collision trajectory loss. The total collision trajectory loss of the current iteration is determined based on the sum of the collision trajectory losses of all modalities of all sample scenes in a batch.

[0061] In a feasible embodiment, the labeled obstacles in the training samples of the semantic segmentation model are areas in the foreground area of ​​the image that account for a proportion greater than a preset area threshold.

[0062] Among them, the deep learning-based semantic segmentation model can help the drone's autonomous driving system obtain accurate perception information and accurately predict the corresponding object mask pixel by pixel based on the distinction between different semantic objects, providing efficient feature utilization. Incorporating the extracted traversable areas and semantic information into the trajectory prediction model training process can effectively improve the success rate of obstacle avoidance compared to using only RGB visual input. The semantic segmentation model can use a segmentation model commonly used in the field. This model can recognize images pixel by pixel and assign a category label to each pixel in the image.

[0063] When predicting a drone's flight trajectory, distant obstacles have little impact on the current trajectory prediction. Therefore, if these obstacles are also included in the trajectory prediction, the accuracy of the drone's trajectory prediction for the current scenario will be affected. Therefore, in this embodiment, when annotating obstacles in the semantic segmentation model's training samples, only obstacle areas with a significant impact on the current flight path decision are annotated, while obstacle areas with a limited impact on the path decision are filtered out.

[0064] Specifically, when labeling the training samples for the semantic segmentation model, obstacle regions are first identified. Obstacle regions that occupy an area exceeding a preset threshold in the image foreground are marked as labeled regions. The semantic segmentation model is trained using the labeled regions in the training samples. The trained semantic segmentation model is then used to identify obstacles in the sample scene images in the training samples for the trajectory prediction model. The resulting obstacle locations are identified as those with a significant impact on the current UAV flight trajectory decision, thereby improving the accuracy of the current flight trajectory prediction decision.

[0065] In one feasible embodiment, S230 includes:

[0066] Determine the minimum distance between the trajectory point position information of each mode output by the trajectory prediction model corresponding to each training sample and the obstacle position information corresponding to the training sample, and use it as the collision avoidance parameter of each mode output by the trajectory prediction model corresponding to the training sample;

[0067] If the collision avoidance parameter is greater than or equal to the safety distance threshold, then the collision avoidance loss of the mode output by the trajectory prediction model corresponding to the training sample is determined to be zero;

[0068] Otherwise, the collision reference parameter is determined according to the ratio of the collision avoidance parameter to the safety distance threshold;

[0069] The first reference parameter is obtained by taking the natural constant as the base and the negative number of the collision reference parameter as the exponent.

[0070] The difference between the second value and the collision reference parameter is used as the second reference parameter;

[0071] The collision avoidance loss of each mode output by the trajectory prediction model corresponding to each training sample is determined according to the product of the first reference parameter and the second reference parameter.

[0072] Among them, the specific value of the second value can be determined according to the numerical range of the collision avoidance loss during actual scene training. There is no restriction on its specific numerical value here. For example, the second value can be 1.

[0073] Optionally, the collision avoidance loss is determined according to the following formula:

[0074]

[0075] in, represents the sample index of the training sample, k∈[1,N], represents the modality index of the output trajectory modality of the trajectory prediction model, m∈[1,M], Indicates the The trajectory prediction model outputs the first The collision avoidance loss of each mode, Indicates the safety distance threshold, represents the collision avoidance parameter, ,in, Indicates the The trajectory prediction model outputs the first The trajectory point position information of each mode, Indicates the The obstacle location information corresponding to the training samples.

[0076] Indicates the The trajectory prediction model outputs the first The minimum distance between the trajectory prediction result of each mode and the obstacle. For example, the trajectory prediction result output by the trajectory prediction model is a three-dimensional velocity vector, so the velocity vector needs to be converted from the world coordinate system to the camera coordinate system. , represents the rotation matrix, represents the velocity vector in the camera coordinate system, Represents the velocity vector in the world coordinate system; the corresponding trajectory point position information is determined based on the velocity vector in the camera coordinate system. The obstacle position information also needs to be projected onto the image plane. , K is the camera intrinsic parameter matrix, Represents three-dimensional point coordinate information, Indicates the image coordinate information after projection.

[0077] The collision avoidance loss in this embodiment is based on the spatial relationship between the trajectory prediction result and the obstacle position, and adopts a continuous distance penalty mechanism to ensure the safety of the predicted trajectory.

[0078] S240. Determine the predicted category probability distribution of each modality output by the trajectory prediction model according to the trajectory prediction results of multiple categories; determine the trajectory classification loss according to the cross entropy between the predicted category probability distribution of each modality and the corresponding true category label.

[0079] The initial trajectory prediction model outputs trajectory prediction results for multiple categories corresponding to multiple modalities in each sample scene. Based on the trajectory prediction results for multiple categories corresponding to each modality, the predicted category probability distribution of the modality is determined. The true trajectory value of each modality corresponding to each sample scene is pre-labeled in the training samples. Therefore, the true category label is determined from the trajectory prediction results for multiple categories corresponding to each modality based on the true trajectory value of each modality corresponding to each sample scene. The trajectory classification loss for each sample scene is determined based on the average cross-entropy between the predicted category probability distribution and the true category label corresponding to all modalities of each sample scene. The total trajectory classification loss for the current iteration is determined based on the sum of the trajectory classification losses for all sample scenes in a batch.

[0080] In one feasible embodiment, S240 includes:

[0081] Determine the sum of the cross entropy between the predicted category probability distribution of each modality output by the trajectory prediction model and the true category label corresponding to the modality as the total cross entropy parameter;

[0082] The trajectory classification loss is determined by the ratio of the total cross entropy parameter and the total number of modalities.

[0083] Optionally, the trajectory classification loss is determined according to the following formula:

[0084] ;

[0085] in, represents the trajectory classification loss, The output of the trajectory prediction model is The predicted category probability distribution of each modality, Indicates the The true category labels corresponding to the modalities, express and The cross entropy between Represents the modality index of the output trajectory modality of the trajectory prediction model, m∈[1,M], where M represents the total number of modalities.

[0086] The specific calculation formula of cross entropy is: ,

[0087] Among them, C is the total number of categories of trajectory prediction results output by the trajectory prediction model, represents the one-hot true label of the c-th category, Represents the predicted probability of the corresponding category.

[0088] Based on the above example, the trajectory classification loss of each sample scene is determined in turn according to the above formula, and the total trajectory classification loss is determined according to the trajectory classification losses of all sample scenes.

[0089] S250: Determine a target loss based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss.

[0090] S260 , iteratively adjust the parameters of the initial trajectory prediction model according to the target loss corresponding to the training samples in the sample set to obtain a trained target trajectory prediction model.

[0091] The technical solution of this embodiment improves the accuracy of the constraint conditions for multi-dimensional trajectory planning by further determining the target guidance loss, collision avoidance loss and trajectory classification loss, thereby improving the accuracy of trajectory prediction.

[0092] Figure 3 A schematic diagram of the structure of a training device for a trajectory prediction model provided by an embodiment of the present invention. Figure 3 As shown, the device includes:

[0093] The sample trajectory prediction module 310 is used to predict the training samples according to the initial trajectory prediction model to obtain trajectory prediction results of multiple categories;

[0094] a multi-loss determination module 320 for determining a target guidance loss, a collision avoidance loss, and a trajectory classification loss based on the trajectory prediction results of the multiple categories; wherein the target guidance loss is determined based on the trajectory direction of the trajectory prediction result and the target direction of the end point; the collision avoidance loss is determined based on the distance between the trajectory point position of the trajectory prediction result and the obstacle position; and the trajectory classification loss is determined based on the trajectory prediction results of different categories and the corresponding true category labels;

[0095] a target loss determination module 330, configured to determine a target loss based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss;

[0096] The model parameter iterative adjustment module 340 is used to iteratively adjust the parameters of the initial trajectory prediction model according to the target loss corresponding to the training samples in the sample set to obtain a trained target trajectory prediction model.

[0097] The technical solution of this embodiment uses the target direction of the end point as a constraint condition for trajectory planning through goal-oriented loss; uses the traversable area excluding obstacles as a constraint condition for trajectory planning through collision avoidance loss; and uses the classified true value trajectory as a constraint condition for trajectory planning through trajectory classification loss, thereby improving the accuracy of trajectory prediction model training and thus improving the accuracy of trajectory prediction based on the trajectory prediction model.

[0098] Optionally, the multi-loss determination module includes a target-oriented loss determination unit, which is specifically configured to:

[0099] Determine the trajectory direction according to the trajectory starting point and trajectory end point in the trajectory prediction results of each category;

[0100] The target guidance loss is determined according to an angular deviation between the trajectory direction and the target direction corresponding to the training sample.

[0101] Optionally, a goal-oriented loss determination unit is configured to:

[0102] Determine the cosine angle between the trajectory direction of each mode output by the trajectory prediction model corresponding to each training sample and the target direction corresponding to the training sample;

[0103] Determining a modal direction loss value of each mode output by the trajectory prediction model corresponding to each training sample according to a difference between the first value and the cosine angle;

[0104] Determine the sum of the modal directional loss values ​​of each mode output by the trajectory prediction model corresponding to each training sample as the total directional loss;

[0105] The target-oriented loss is determined according to a ratio of the total direction loss to the total number of training samples.

[0106] Optionally, the multi-loss determination module includes a collision avoidance loss determination unit, which is specifically configured to:

[0107] Identify the training samples according to a pre-trained semantic segmentation model to determine obstacle location information;

[0108] Determining collision avoidance parameters based on the trajectory point position information and the obstacle position information in the trajectory prediction results of each category;

[0109] The collision trajectory loss is determined according to the collision avoidance parameter and a preset safety distance threshold.

[0110] Optionally, a collision avoidance loss determination unit is configured to:

[0111] Determine the minimum distance between the trajectory point position information of each mode output by the trajectory prediction model corresponding to each training sample and the obstacle position information corresponding to the training sample, and use it as the collision avoidance parameter of each mode output by the trajectory prediction model corresponding to the training sample;

[0112] If the collision avoidance parameter is greater than or equal to the safety distance threshold, determining that the collision avoidance loss of the mode output by the trajectory prediction model corresponding to the training sample is zero;

[0113] otherwise, determining a collision reference parameter according to a ratio of the collision avoidance parameter to the safety distance threshold;

[0114] The first reference parameter is obtained by taking the natural constant as the base and the negative number of the collision reference parameter as the exponent.

[0115] Taking the difference between the second value and the collision reference parameter as the second reference parameter;

[0116] The collision avoidance loss of each mode output by the trajectory prediction model corresponding to each training sample is determined according to the product of the first reference parameter and the second reference parameter.

[0117] Optionally, the labeled obstacles in the training samples of the semantic segmentation model are areas in the foreground area of ​​the image that account for a proportion greater than a preset area threshold.

[0118] Optionally, the multi-loss determination module includes a trajectory classification loss determination unit, which is specifically used to:

[0119] Determine the predicted category probability distribution of each modality output by the trajectory prediction model according to the trajectory prediction results of multiple categories;

[0120] The trajectory classification loss is determined based on the cross entropy between the predicted class probability distribution of each modality and the corresponding true class labels.

[0121] Optionally, a trajectory classification loss determination unit is used to:

[0122] Determine the sum of the cross entropy between the predicted category probability distribution of each modality output by the trajectory prediction model and the true category label corresponding to the modality as the total cross entropy parameter;

[0123] The trajectory classification loss is determined according to the ratio of the total cross entropy parameter to the total number of modalities.

[0124] The trajectory prediction model training device provided by the embodiment of the present invention can execute the trajectory prediction model training method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0125] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0126] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0127] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0128] like Figure 4As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0129] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0130] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the method for training the trajectory prediction model.

[0131] In some embodiments, the training of the method trajectory prediction model can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the training of the method trajectory prediction model described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the training of the method trajectory prediction model in any other appropriate manner (e.g., by means of firmware).

[0132] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific reference products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes switch components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, switch components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0137] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0138] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0139] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0140] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for training a trajectory prediction model, characterized in that: The method includes: Predict the training samples according to the initial trajectory prediction model to obtain trajectory prediction results of multiple categories; Determining a target guidance loss, a collision avoidance loss, and a trajectory classification loss based on the trajectory prediction results of the multiple categories; wherein the target guidance loss is determined based on the trajectory direction of the trajectory prediction result and the target direction of the end point, the collision avoidance loss is determined based on the distance between the trajectory point position of the trajectory prediction result and the obstacle position, and the trajectory classification loss is determined based on the trajectory prediction results of different categories and the corresponding true category labels; determining a target loss based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss; The initial trajectory prediction model is iteratively parameterized according to the target loss corresponding to the training sample in the sample set to obtain a trained target trajectory prediction model.

2. The method according to claim 1, characterized in that Determining a target-oriented loss based on trajectory prediction results of the multiple categories includes: Determine the trajectory direction according to the trajectory starting point and trajectory end point in the trajectory prediction results of each category; The target guidance loss is determined according to an angular deviation between the trajectory direction and the target direction corresponding to the training sample.

3. The method according to claim 2, characterized in that Determining a target-oriented loss based on trajectory prediction results of the multiple categories includes: Determine the cosine angle between the trajectory direction of each mode output by the trajectory prediction model corresponding to each training sample and the target direction corresponding to the training sample; Determining a modal direction loss value of each mode output by the trajectory prediction model corresponding to each training sample according to a difference between the first value and the cosine angle; Determine the sum of the modal directional loss values ​​of each mode output by the trajectory prediction model corresponding to each training sample as the total directional loss; The target-oriented loss is determined according to a ratio of the total direction loss to the total number of training samples.

4. The method according to claim 1, wherein Determining a collision avoidance loss based on trajectory prediction results of the multiple categories includes: Identify the training samples according to a pre-trained semantic segmentation model to determine obstacle location information; Determining collision avoidance parameters based on the trajectory point position information and the obstacle position information in the trajectory prediction results of each category; The collision trajectory loss is determined according to the collision avoidance parameter and a preset safety distance threshold.

5. The method according to claim 4, characterized in that Determining a collision avoidance loss based on trajectory prediction results of the multiple categories includes: Determine the minimum distance between the trajectory point position information of each mode output by the trajectory prediction model corresponding to each training sample and the obstacle position information corresponding to the training sample, and use it as the collision avoidance parameter of each mode output by the trajectory prediction model corresponding to the training sample; If the collision avoidance parameter is greater than or equal to the safety distance threshold, determining that the collision avoidance loss of the mode output by the trajectory prediction model corresponding to the training sample is zero; otherwise, determining a collision reference parameter according to a ratio of the collision avoidance parameter to the safety distance threshold; The first reference parameter is obtained by taking the natural constant as the base and the negative number of the collision reference parameter as the exponent. Taking the difference between the second value and the collision reference parameter as the second reference parameter; The collision avoidance loss of each mode output by the trajectory prediction model corresponding to each training sample is determined according to the product of the first reference parameter and the second reference parameter.

6. The method according to claim 4, characterized in that The labeled obstacles in the training samples of the semantic segmentation model are areas in the foreground area of ​​the image that account for a proportion greater than a preset area threshold.

7. The method according to claim 1, characterized in that Determining a trajectory classification loss according to trajectory prediction results of the multiple categories includes: Determine the predicted category probability distribution of each modality output by the trajectory prediction model according to the trajectory prediction results of multiple categories; The trajectory classification loss is determined based on the cross entropy between the predicted class probability distribution of each modality and the corresponding true class labels.

8. The method according to claim 7, characterized in that Determining a trajectory classification loss according to trajectory prediction results of the multiple categories includes: Determine the sum of the cross entropy between the predicted category probability distribution of each modality output by the trajectory prediction model and the true category label corresponding to the modality as the total cross entropy parameter; The trajectory classification loss is determined according to the ratio of the total cross entropy parameter to the total number of modalities.

9. A training device for a trajectory prediction model, characterized in that: The device includes: The sample trajectory prediction module is used to predict the training samples based on the initial trajectory prediction model and obtain trajectory prediction results of multiple categories; a multi-loss determination module, configured to determine a target guidance loss, a collision avoidance loss, and a trajectory classification loss based on the trajectory prediction results of the multiple categories; wherein the target guidance loss is determined based on the trajectory direction of the trajectory prediction result and the target direction of the end point, the collision avoidance loss is determined based on the distance between the trajectory point position of the trajectory prediction result and the obstacle position, and the trajectory classification loss is determined based on the trajectory prediction results of different categories and the corresponding true category labels; a target loss determination module, configured to determine a target loss based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss; The model parameter iterative adjustment module is used to iteratively adjust the parameters of the initial trajectory prediction model according to the target loss corresponding to the training sample in the sample set to obtain a trained target trajectory prediction model.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the training method of the trajectory prediction model according to any one of claims 1 to 7 when executed.

Images