Trajectory prediction model training method, device and medium

Abstract

The application discloses a trajectory prediction model training method and device and a medium. The method comprises the following steps: predicting a training sample according to an initial trajectory prediction model to obtain trajectory prediction results of multiple categories; determining a target orientation loss, a collision avoidance loss and a trajectory classification loss according to the trajectory prediction results of the multiple categories; determining a target loss according to the target orientation loss, the collision avoidance loss and the trajectory classification loss; and iteratively adjusting parameters of the initial trajectory prediction model according to the target loss corresponding to the training sample in a sample set to obtain a target trajectory prediction model after training. The target orientation loss takes the target direction of the end point as a constraint condition for trajectory planning; the collision avoidance loss takes the passable area except the obstacle as a constraint condition for trajectory planning; and the trajectory classification loss takes the classified true value trajectory as a constraint condition for trajectory planning, thereby improving the accuracy of trajectory prediction according to the trajectory prediction model.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a training method, apparatus, and medium for a trajectory prediction model. Background Technology

[0002] Drones have wide applications in various fields such as agriculture, exploration, disaster relief, public safety, entertainment, and future warfare. Drones are required to complete flight missions autonomously and efficiently in a variety of complex scenarios. Furthermore, different flight missions have different requirements for the flight strategies of drones. For example, in disaster relief missions, the flight strategy is mainly time-sensitive, while in entertainment missions, the flight strategy is mainly safety-sensitive.

[0003] In the autonomous navigation system of UAVs, the trajectory prediction model is mainly used to predict the current flight trajectory. Therefore, in order to ensure the accuracy of the flight trajectory of UAVs under different flight strategies, it is necessary to improve the training effect of the trajectory prediction model. Summary of the Invention

[0004] This invention provides a training method, apparatus, and medium for a trajectory prediction model to improve the training effect of the trajectory prediction model.

[0005] According to one aspect of the present invention, a method for training a trajectory prediction model is provided, comprising:

[0006] Based on the initial trajectory prediction model, the training samples are predicted to obtain trajectory prediction results for multiple categories;

[0007] The target guidance loss, collision avoidance loss, and trajectory classification loss are determined based on the trajectory prediction results of the multiple categories; wherein, the target guidance loss is determined based on the trajectory direction and the target direction of the endpoint of the trajectory prediction result, the collision avoidance loss is determined based on the distance between the trajectory point position and the obstacle position of the trajectory prediction result, and the trajectory classification loss is determined based on the trajectory prediction results of different categories and the corresponding true category labels;

[0008] The target loss is determined based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss.

[0009] The initial trajectory prediction model is iteratively adjusted based on the target loss corresponding to the training samples in the sample set to obtain the trained target trajectory prediction model.

[0010] According to another aspect of the present invention, a training apparatus for a trajectory prediction model is provided, comprising:

[0011] The sample trajectory prediction module is used to predict the training samples based on the initial trajectory prediction model and obtain trajectory prediction results for multiple categories.

[0012] A multi-loss determination module is used to determine target guidance loss, collision avoidance loss, and trajectory classification loss based on the trajectory prediction results of the multiple categories; wherein, the target guidance loss is determined based on the trajectory direction and the target direction of the endpoint of the trajectory prediction result, the collision avoidance loss is determined based on the distance between the trajectory point position and the obstacle position of the trajectory prediction result, and the trajectory classification loss is determined based on the trajectory prediction results of different categories and the corresponding true category labels;

[0013] The target loss determination module is used to determine the target loss based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss.

[0014] The model parameter iterative adjustment module is used to iteratively adjust the parameters of the initial trajectory prediction model based on the target loss corresponding to the training samples in the sample set, so as to obtain the trained target trajectory prediction model.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a training method for a trajectory prediction model according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the training method of the trajectory prediction model according to any embodiment of the present invention.

[0018] The technical solution of this invention improves the accuracy of trajectory prediction model training by using the target direction of the endpoint as a constraint condition for trajectory planning through target orientation loss; using the passable area other than obstacles as a constraint condition for trajectory planning through collision avoidance loss; and using the classified ground truth trajectory as a constraint condition for trajectory planning through trajectory classification loss. This improves the accuracy of trajectory prediction based on the trajectory prediction model.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a training method for a trajectory prediction model according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart of another training method for a trajectory prediction model provided by an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the structure of a training device for a trajectory prediction model according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the training method of the trajectory prediction model in the embodiments of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "candidate," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Figure 1This invention provides a flowchart of a training method for a trajectory prediction model. This embodiment is applicable to optimizing the training of a trajectory prediction model that guides UAV flight. The method can be executed by a trajectory prediction model training device, which can be implemented in hardware and / or software and can be configured on a server with computing power. Figure 1 As shown, the method includes:

[0028] S110. Based on the initial trajectory prediction model, predict the training samples to obtain trajectory prediction results for multiple categories.

[0029] The initial trajectory prediction model can be an untrained network architecture or a pre-trained network architecture. The network architecture can be a classification network, and this invention does not limit the type of classification network. The training samples are trajectory paths and scene images obtained by controlling the drone's flight in different scenarios.

[0030] Specifically, scene images from different scenarios in the training samples are input into the initial trajectory prediction model. The output of the initial trajectory prediction model consists of multiple trajectory prediction results of different categories. Different categories of trajectory prediction results represent different trajectory direction angles, and the number of output categories can be adjusted according to actual training needs; there is no limitation here, for example, the number of categories could be 20. For instance, scene images from each scenario in the training samples are input into the initial trajectory prediction model, which then provides multiple trajectory prediction results for obstacle avoidance flight from different directions and angles. Each trajectory prediction result includes multiple trajectory point position information or multiple trajectory velocity vectors.

[0031] S120. Determine the target guidance loss, collision avoidance loss, and trajectory classification loss based on the trajectory prediction results of multiple categories.

[0032] Since the model training in this embodiment is supervised training, the training samples include pre-determined drone trajectory paths in different scenarios. These drone trajectory paths are ground truth values, used to guide the output trajectory prediction results of the initial trajectory prediction model to approach the ground truth values. Therefore, based on the trajectory prediction results of multiple categories for each scenario obtained from the initial trajectory prediction model, and the ground truth values ​​for each scenario, different types of losses are determined. These different types of losses are coordinated from multiple dimensions to simultaneously optimize the accuracy, safety, and target orientation of the trained trajectory prediction.

[0033] Different types of losses include target-oriented loss, collision avoidance loss, and trajectory classification loss. Target-oriented loss is determined based on the trajectory direction of the predicted trajectory and the target direction of the endpoint. The trajectory direction represents the predicted flight direction of the UAV, which can be a direction vector from the starting point to the endpoint in the predicted trajectory, or multiple direction vectors from the starting point to the endpoint at different time periods in the predicted trajectory. The target direction of the endpoint refers to the UAV's intended flight direction in different scenarios. The intended flight direction can be pre-labeled in different scenarios or determined based on the endpoint direction of the UAV trajectory path in each scenario in the sample. For example, target-oriented loss can be determined based on the cumulative deviation between the trajectory direction of each type of trajectory prediction result and the target direction of the endpoint in each scenario. The target-oriented loss guides the trajectory prediction result towards the desired target direction, thus optimizing the target orientation of the trajectory prediction.

[0034] Collision avoidance loss is determined based on the distance between the trajectory point positions in the trajectory prediction results and the obstacle positions. The trajectory point positions represent the predicted flight position of the UAV, while the obstacle positions are determined based on scene images for different scenarios. For example, the obstacle positions may be determined based on obstacle recognition results from scene images. For instance, the collision trajectory loss can be determined based on the cumulative distance between the trajectory point positions and obstacle positions for each type of trajectory prediction result in each scenario. Guided by the collision avoidance loss, the trajectory prediction results can safely avoid obstacles, thus optimizing the safety of trajectory prediction.

[0035] The trajectory classification loss is determined based on the trajectory prediction results of different categories and their corresponding ground truth class labels. The drone trajectory paths in each scenario of the training samples correspond to different modalities. Different modalities represent different flight strategies, such as aggressive strategies (minimizing flight time), safety strategies (ensuring a safe distance from obstacles), intermediate strategies (combining short flight time and safety), etc., which can be determined based on other flight strategies. Flight strategies can be adjusted according to the actual scenario, and the number of modalities is not limited. For example, the drone trajectory paths corresponding to different modalities in each scenario are pre-determined as the ground truth class labels for each modality. There is a correlation between the trajectory prediction results of different categories and different modalities. For example, when the trajectory prediction model outputs trajectory prediction results of different categories, the output information includes the modality information to which it belongs. For example, the trajectory prediction model outputs trajectory prediction results for different directions and angles during flight under different modalities; that is, each modality corresponds to multiple different types of trajectory prediction results. For example, the trajectory classification loss is determined based on the cumulative deviation between the trajectory prediction results of each modality in each scenario and their corresponding ground truth class labels. By using trajectory classification loss as a guide, the trajectory prediction results can optimize the deviation between the predicted results and the true values ​​of discrete trajectory categories, thereby improving the accuracy of trajectory prediction.

[0036] S130. Determine the target loss based on the target orientation loss, collision avoidance loss, and trajectory classification loss.

[0037] The target loss is determined based on the combined results of the target-oriented loss, collision avoidance loss, and trajectory classification loss. Specifically, the target loss is determined by a weighted sum of the target-oriented loss, collision avoidance loss, and trajectory classification loss. For example, L_total = α·L_target + β·L_collision + γ·L_classification, where L_total represents the target-oriented loss, α represents the weight of the target-oriented loss, L_target represents the collision avoidance loss, β represents the weight of the collision avoidance loss, L_classification represents the trajectory classification loss, and γ represents the weight of the trajectory classification loss. The weight values ​​can be determined according to the actual scenario requirements to balance the importance of the losses in different dimensions.

[0038] S140. Adjust the parameters of the initial trajectory prediction model iteratively based on the target loss corresponding to the training samples in the sample set to obtain the trained target trajectory prediction model.

[0039] The parameters of the initial trajectory prediction model are continuously adjusted based on the target loss obtained from the training samples in the sample set. For example, all training samples in the sample set are divided into multiple batches. The parameters in the initial trajectory prediction model are adjusted once based on the target loss obtained from the training samples in each batch. Then, the direction of parameter adjustment is determined based on the changing trend of the target loss obtained from the training samples in the next batch. By continuously iterating and training all training samples in the sample set until the loss converges or the preset number of iterations is reached, the trained target trajectory prediction model is obtained.

[0040] Optionally, after obtaining the trained target trajectory prediction model, the method further includes:

[0041] In the autonomous navigation system of unmanned aerial vehicles (UAVs), a target trajectory prediction model is used to determine the trajectory prediction results for different modes. This involves enabling obstacle avoidance flight using different flight strategies through the target trajectory prediction model, and generating flight trajectories that better reflect the actual environment through training with the aforementioned three dimensions of loss.

[0042] The technical solution of this embodiment improves the accuracy of trajectory prediction model training by using the target direction of the endpoint as the constraint condition for trajectory planning through target orientation loss; using the passable area other than obstacles as the constraint condition for trajectory planning through collision avoidance loss; and using the classified ground truth trajectory as the constraint condition for trajectory planning through trajectory classification loss. This improves the accuracy of trajectory prediction based on the trajectory prediction model.

[0043] Figure 2 This is a flowchart illustrating another training method for a trajectory prediction model provided in an embodiment of the present invention. This embodiment further refines the multi-loss determination process described in the above embodiments. Figure 2 As shown, the method includes:

[0044] S210. Based on the initial trajectory prediction model, predict the training samples to obtain trajectory prediction results for multiple categories.

[0045] S220. Determine the trajectory direction based on the trajectory start point and trajectory end point in the trajectory prediction results for each category; determine the target guidance loss based on the angle deviation between the trajectory direction and the target direction corresponding to the training sample.

[0046] The trajectory direction of each category is determined based on the trajectory prediction results of each category obtained for each sample scene, the trajectory start point and trajectory end point are determined, the target direction of the corresponding end point of each sample scene is determined, the cumulative angle deviation of each sample scene is determined based on the sum of the angle deviations between the trajectory direction and the target direction of all categories, and the target guidance loss is determined based on the cumulative angle deviation of all sample scenes.

[0047] For example, the initial trajectory prediction model outputs trajectory prediction results for multiple categories corresponding to multiple modalities. The trajectory prediction result with the highest prediction probability value for each modality is determined as the target trajectory prediction result for that modality. Since the trained trajectory prediction model outputs a trajectory prediction result corresponding to each modality, to improve the accuracy of the target-oriented loss, the target trajectory prediction result corresponding to each modality is used in the calculation process to determine the target-oriented loss. For instance, the target trajectory prediction result with the highest prediction probability value for each modality in each sample scene is determined. The sum of the angle deviations between the trajectory direction and the target direction of the target trajectory prediction results of all modalities in each sample scene is determined as the cumulative angle deviation for that sample scene. Then, the target-oriented loss for the current iteration is determined based on the cumulative angle deviation of all sample scenes in a batch.

[0048] In one feasible embodiment, S220 includes:

[0049] Determine the cosine angle between the trajectory direction of each modality output by the trajectory prediction model for each training sample and the target direction corresponding to that training sample;

[0050] The modal orientation loss value of each mode output by the trajectory prediction model for each training sample is determined based on the difference between the first value and the cosine angle.

[0051] The sum of the modal orientation loss values ​​of each mode output by the trajectory prediction model corresponding to each training sample is determined as the total orientation loss;

[0052] The target guidance loss is determined by the ratio of the total directional loss to the total number of training samples.

[0053] The specific value of the first value can be determined based on the range of values ​​of the target-oriented loss during actual training scenarios. There is no restriction on its specific value; for example, the first value can be 1.

[0054] Optionally, the target-oriented loss can be determined according to the following formula:

[0055] ;

[0056] in, Indicates goal-oriented loss. This represents the sample index of the training sample, k∈[1,N]. This represents the mode index of the output trajectory mode of the trajectory prediction model, m∈[1,M]. Indicates the first The trajectory prediction model output for the training sample is the first... The trajectory direction of each mode Indicates the first The target direction corresponding to each training sample.

[0057] The sample index is used to represent the sample sequence number involved in the current iteration. For example, N is the total number of sample scenarios in a batch. The modality index is used to represent the different flight strategy identifiers output by the pre-determined trajectory prediction model. For example, M represents the total number of flight strategies. ,in, Indicates the first The trajectory prediction model output for the training sample is the first... The trajectory prediction results for each modality include the trajectory end point location information. Indicates the first The trajectory prediction model output for the training sample is the first... The trajectory prediction results for each modality include the trajectory starting point location information. It can be determined based on the pre-labeled information of each sample scene, or based on the endpoint information of the true trajectory value in the sample. This represents the cosine of the angle between the trajectory direction and the target direction. The smaller the value of the cosine, the smaller the deviation between the trajectory direction and the target direction.

[0058] S230. Identify training samples based on a pre-trained semantic segmentation model to determine obstacle location information; determine collision avoidance parameters based on trajectory point location information and obstacle location information in the trajectory prediction results for each category; determine collision trajectory loss based on collision avoidance parameters and a preset safe distance threshold.

[0059] A semantic segmentation model is used to identify the sample scene images in the training samples to obtain the obstacle location information in the sample scene images. The cumulative collision deviation of the sample scene is determined by the sum of the distances between the trajectory point location information and the obstacle location information in the trajectory prediction results of all categories of each sample scene. The collision trajectory loss is determined by the cumulative collision deviation of all sample scenes and the safe distance threshold.

[0060] For example, based on the above example, the target trajectory prediction result with the highest predicted probability value for each modality obtained for each sample scene is determined. The minimum distance between the trajectory point position information and the obstacle position information of the target trajectory prediction result for each modality in each sample scene is determined as the collision avoidance parameter for that modality. The collision trajectory loss is determined based on the comparison result between the collision avoidance parameter of each modality and the safe distance threshold. For example, if the collision avoidance parameter of a modality is greater than or equal to the safe distance threshold, the collision trajectory loss of that modality is determined to be 0; otherwise, it is determined according to the pre-established mapping relationship between the collision avoidance parameter and the collision trajectory loss. The total collision trajectory loss of the current iteration is determined based on the sum of the collision trajectory losses of all modalities in all sample scenes in a batch.

[0061] In one feasible embodiment, the labeled obstacles in the training samples of the semantic segmentation model are regions that occupy a larger proportion of the foreground region of the image than a preset area threshold.

[0062] Among them, deep learning-based semantic segmentation models can help drone autopilot systems acquire accurate perception information and accurately predict the masks of corresponding objects pixel by pixel based on the differentiation of different semantic objects, demonstrating an efficient feature utilization method. Integrating the extracted passable areas and semantic information into the trajectory prediction model training process can effectively improve the obstacle avoidance success rate compared to using only RGB visual input. The semantic segmentation model can use commonly used segmentation models in this field, which can identify images pixel by pixel and assign a category label to each pixel in the image.

[0063] When predicting the flight trajectory of a drone, distant obstacles have a relatively small impact on the current trajectory prediction result. Therefore, if distant obstacles are identified and included in the trajectory prediction at the same time as the obstacle identification, it will actually affect the accuracy of the trajectory prediction result in the current scene. Therefore, in this embodiment, when labeling obstacles on the training samples of the semantic segmentation model, only obstacle regions that have a significant impact on the current flight path decision are labeled, while obstacle regions with limited impact on the path decision are filtered out.

[0064] Specifically, when labeling the training samples of the semantic segmentation model, obstacle regions are first identified. Obstacle regions whose area occupies the foreground region of the image exceeds a preset area threshold are designated as marked regions. The marked regions from the training samples are then used to train the semantic segmentation model. The trained semantic segmentation model is then used to identify obstacles in the sample scene images from the training samples of the trajectory prediction model. The resulting obstacle location information represents the obstacle regions that have a significant impact on the current UAV flight trajectory decision, thereby improving the accuracy of the current flight trajectory prediction decision.

[0065] In one feasible embodiment, S230 includes:

[0066] The minimum distance between the trajectory point position information of each modality output by the trajectory prediction model corresponding to each training sample and the obstacle position information corresponding to that training sample is determined and used as the collision avoidance parameter of each modality output by the trajectory prediction model corresponding to that training sample.

[0067] If the collision avoidance parameter is greater than or equal to the safe distance threshold, then the collision avoidance loss of the mode output by the trajectory prediction model corresponding to the training sample is determined to be zero.

[0068] Otherwise, the collision reference parameters are determined based on the ratio of the collision avoidance parameters to the safe distance threshold.

[0069] The first reference parameter is the result of raising the negative number of the collision reference parameter to the power of the natural constant as the base.

[0070] The difference between the second value and the collision reference parameter is used as the second reference parameter;

[0071] The collision avoidance loss of each mode output by the trajectory prediction model corresponding to each training sample is determined by the product of the first reference parameter and the second reference parameter.

[0072] The specific value of the second value can be determined based on the range of collision avoidance loss during actual scenario training. There is no restriction on its specific value. For example, the second value can be 1.

[0073] Optionally, the collision avoidance loss can be determined according to the following formula:

[0074]

[0075] in, This represents the sample index of the training sample, k∈[1,N]. This represents the mode index of the output trajectory mode of the trajectory prediction model, m∈[1,M]. Indicates the first The trajectory prediction model output for the training sample is the first... Collision avoidance loss for each modality Indicates the safe distance threshold. Indicates collision avoidance parameters. ,in, Indicates the first The trajectory prediction model output for the training sample is the first... The trajectory point position information of each modality, Indicates the first The obstacle location information corresponding to each training sample.

[0076] Indicates the first The trajectory prediction model output for the training sample is the first... The minimum distance between the trajectory prediction result and the obstacle for each modality. For example, the trajectory prediction model outputs a 3D velocity vector, therefore the velocity vector needs to be transformed from the world coordinate system to the camera coordinate system. , Represents the rotation matrix. This represents the velocity vector in the camera coordinate system. This represents the velocity vector in the world coordinate system; the position information of the corresponding trajectory points is determined based on the velocity vector in the camera coordinate system. Obstacle position information also requires 3D point projection onto the image plane. K is the camera intrinsic parameter matrix. Represents three-dimensional point coordinate information. This represents the coordinate information of the projected image.

[0077] In this embodiment, the collision avoidance loss is based on the spatial relationship between the trajectory prediction result and the obstacle position, and a continuous distance penalty mechanism is adopted to ensure the safety of the predicted trajectory.

[0078] S240. Determine the probability distribution of the predicted class for each modality output by the trajectory prediction model based on the trajectory prediction results for multiple categories; determine the trajectory classification loss based on the cross-entropy between the probability distribution of the predicted class for each modality and the corresponding true class label.

[0079] The initial trajectory prediction model outputs trajectory prediction results for multiple categories across multiple modalities in each sample scenario. Based on these prediction results, the predicted category probability distribution for each modality is determined. Furthermore, the ground truth trajectory values ​​for each modality in each sample scenario are pre-labeled in the training samples. Therefore, the true category label is determined from the trajectory prediction results for each modality based on the ground truth trajectory values ​​for each sample scenario. The trajectory classification loss for each sample scenario is determined by the average cross-entropy between the predicted category probability distributions and the true category labels across all modalities. The total trajectory classification loss for the current iteration is determined by the sum of the trajectory classification losses for all sample scenarios in a batch.

[0080] In one feasible embodiment, S240 includes:

[0081] The sum of the cross-entropy between the predicted class probability distribution of each modality output by the trajectory prediction model and the corresponding true class label of that modality is determined as the total cross-entropy parameter;

[0082] The trajectory classification loss is determined by the ratio of the total cross-entropy parameter to the total number of modes.

[0083] Optionally, the trajectory classification loss can be determined according to the following formula:

[0084] ;

[0085] in, Represents trajectory classification loss, The output of the trajectory prediction model represents the first... The predicted class probability distribution for each modality. Indicates the first The true category label corresponding to each modality express and Cross-entropy between The modality index represents the output trajectory mode of the trajectory prediction model, m∈[1,M], where M represents the total number of modes.

[0086] The specific formula for calculating cross-entropy is: ,

[0087] Where C represents the total number of categories in the trajectory prediction results output by the trajectory prediction model. This represents the one-hot true label of the c-th category. This represents the predicted probability for the corresponding category.

[0088] Based on the above example, the trajectory classification loss for each sample scenario is determined sequentially according to the above formula, and the total trajectory classification loss is determined based on the trajectory classification losses of all sample scenarios.

[0089] S250. Determine the target loss based on the target orientation loss, collision avoidance loss, and trajectory classification loss.

[0090] S260. Adjust the parameters of the initial trajectory prediction model iteratively based on the target loss corresponding to the training samples in the sample set to obtain the trained target trajectory prediction model.

[0091] The technical solution in this embodiment improves the accuracy of the constraints for multi-dimensional trajectory planning by further determining the target guidance loss, collision avoidance loss, and trajectory classification loss, thereby improving the accuracy of trajectory prediction.

[0092] Figure 3 This is a schematic diagram of the structure of a training device for a trajectory prediction model provided in an embodiment of the present invention. Figure 3 As shown, the device includes:

[0093] The sample trajectory prediction module 310 is used to predict the training samples based on the initial trajectory prediction model to obtain trajectory prediction results for multiple categories.

[0094] The multi-loss determination module 320 is used to determine target guidance loss, collision avoidance loss, and trajectory classification loss based on the trajectory prediction results of the multiple categories; wherein, the target guidance loss is determined based on the trajectory direction and the target direction of the endpoint of the trajectory prediction result, the collision avoidance loss is determined based on the distance between the trajectory point position and the obstacle position of the trajectory prediction result, and the trajectory classification loss is determined based on the trajectory prediction results of different categories and the corresponding true category labels;

[0095] The target loss determination module 330 is used to determine the target loss based on the target guidance loss, the collision avoidance loss and the trajectory classification loss;

[0096] The model parameter iterative adjustment module 340 is used to iteratively adjust the parameters of the initial trajectory prediction model according to the target loss corresponding to the training samples in the sample set, so as to obtain the trained target trajectory prediction model.

[0097] The technical solution of this embodiment improves the accuracy of trajectory prediction model training by using the target direction of the endpoint as the constraint condition for trajectory planning through target orientation loss; using the passable area other than obstacles as the constraint condition for trajectory planning through collision avoidance loss; and using the classified ground truth trajectory as the constraint condition for trajectory planning through trajectory classification loss. This improves the accuracy of trajectory prediction based on the trajectory prediction model.

[0098] Optionally, the multi-loss determination module includes a target-oriented loss determination unit, specifically used for:

[0099] The trajectory direction is determined based on the trajectory start and trajectory end points in the trajectory prediction results for each category;

[0100] The target guidance loss is determined based on the angular deviation between the trajectory direction and the target direction corresponding to the training sample.

[0101] Optionally, a target-oriented loss determination unit is used for:

[0102] Determine the cosine angle between the trajectory direction of each modality output by the trajectory prediction model for each training sample and the target direction corresponding to that training sample;

[0103] The modal orientation loss value of each mode output by the trajectory prediction model corresponding to each training sample is determined based on the difference between the first value and the cosine angle.

[0104] The sum of the modal orientation loss values ​​of each mode output by the trajectory prediction model corresponding to each training sample is determined as the total orientation loss;

[0105] The target guidance loss is determined based on the ratio of the total directional loss to the total number of training samples.

[0106] Optionally, the multi-loss determination module includes a collision avoidance loss determination unit, specifically used for:

[0107] The training samples are identified using a pre-trained semantic segmentation model to determine the location information of obstacles.

[0108] Collision avoidance parameters are determined based on the trajectory point location information and obstacle location information in the trajectory prediction results for each category, respectively.

[0109] The collision trajectory loss is determined based on the collision avoidance parameters and the preset safe distance threshold.

[0110] Optionally, a collision avoidance loss determination unit is used specifically for:

[0111] The minimum distance between the trajectory point position information of each modality output by the trajectory prediction model corresponding to each training sample and the obstacle position information corresponding to that training sample is determined and used as the collision avoidance parameter of each modality output by the trajectory prediction model corresponding to that training sample.

[0112] If the collision avoidance parameter is greater than or equal to the safe distance threshold, then the collision avoidance loss of the mode output by the trajectory prediction model corresponding to the training sample is determined to be zero.

[0113] Otherwise, the collision reference parameters are determined based on the ratio of the collision avoidance parameters to the safe distance threshold.

[0114] The first reference parameter is the result of raising the negative number of the collision reference parameter to the power of the natural constant as the base.

[0115] The difference between the second value and the collision reference parameter is taken as the second reference parameter;

[0116] The collision avoidance loss of each mode output by the trajectory prediction model corresponding to each training sample is determined based on the product of the first reference parameter and the second reference parameter.

[0117] Optionally, the labeled obstacles in the training samples of the semantic segmentation model are regions that occupy a larger proportion of the foreground region of the image than a preset area threshold.

[0118] Optionally, the multi-loss determination module includes a trajectory classification loss determination unit, specifically used for:

[0119] Based on the trajectory prediction results of multiple categories, determine the prediction category probability distribution of each modality output by the trajectory prediction model;

[0120] The trajectory classification loss is determined based on the cross-entropy between the predicted class probability distribution for each modality and the corresponding true class label.

[0121] Optional, the trajectory classification loss determination unit is specifically used for:

[0122] The sum of the cross-entropy between the predicted class probability distribution of each modality output by the trajectory prediction model and the corresponding true class label of that modality is determined as the total cross-entropy parameter;

[0123] The trajectory classification loss is determined based on the ratio of the total cross-entropy parameter to the total number of modes.

[0124] The training device for the trajectory prediction model provided in this embodiment of the invention can execute the training method for the trajectory prediction model provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0125] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations and do not violate public order and good morals.

[0126] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0127] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0128] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0129] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0130] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as training a method trajectory prediction model.

[0131] In some embodiments, training of the method trajectory prediction model may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of training the method trajectory prediction model described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform training of the method trajectory prediction model by any other suitable means (e.g., by means of firmware).

[0132] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific reference products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0136] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or computing systems that include switching components (e.g., application servers), or computing systems that include front-end components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such back-end, switching, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0137] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0138] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0139] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A training method for a trajectory prediction model, characterized in that, The method includes: The training samples are predicted based on the initial trajectory prediction model to obtain trajectory prediction results for multiple categories under different modes; different modes are used to represent different flight strategies. The target guidance loss, collision avoidance loss, and trajectory classification loss are determined based on the trajectory prediction results of the multiple categories. Specifically, the target guidance loss is determined based on the trajectory direction and the target direction of the endpoint of the trajectory prediction results, the collision avoidance loss is determined based on the distance between the trajectory point position and the obstacle position of the trajectory prediction results, and the trajectory classification loss is determined based on the trajectory prediction results of different categories under each modality and the corresponding true category labels. The target loss is determined based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss. The initial trajectory prediction model is iteratively adjusted based on the target loss corresponding to the training samples in the sample set to obtain a trained target trajectory prediction model. The target trajectory prediction model is then used to determine the trajectory prediction results for different modalities.

2. The method according to claim 1, characterized in that, The target guidance loss is determined based on the trajectory prediction results of the multiple categories, including: The trajectory direction is determined based on the trajectory start and trajectory end points in the trajectory prediction results for each category; The target guidance loss is determined based on the angular deviation between the trajectory direction and the target direction corresponding to the training sample.

3. The method according to claim 2, characterized in that, The target guidance loss is determined based on the trajectory prediction results of the multiple categories, including: Determine the cosine angle between the trajectory direction of each modality output by the trajectory prediction model for each training sample and the target direction corresponding to that training sample; The modal orientation loss value of each mode output by the trajectory prediction model corresponding to each training sample is determined based on the difference between the first value and the cosine angle. The sum of the modal orientation loss values ​​of each mode output by the trajectory prediction model corresponding to each training sample is determined as the total orientation loss; The target guidance loss is determined based on the ratio of the total directional loss to the total number of training samples.

4. The method according to claim 1, characterized in that, The collision avoidance loss is determined based on the trajectory prediction results of the multiple categories, including: The training samples are identified using a pre-trained semantic segmentation model to determine the location information of obstacles. Collision avoidance parameters are determined based on the trajectory point location information and obstacle location information in the trajectory prediction results for each category, respectively. The collision trajectory loss is determined based on the collision avoidance parameters and the preset safe distance threshold.

5. The method according to claim 4, characterized in that, The collision avoidance loss is determined based on the trajectory prediction results of the multiple categories, including: The minimum distance between the trajectory point position information of each modality output by the trajectory prediction model corresponding to each training sample and the obstacle position information corresponding to that training sample is determined and used as the collision avoidance parameter of each modality output by the trajectory prediction model corresponding to that training sample. If the collision avoidance parameter is greater than or equal to the safe distance threshold, then the collision avoidance loss of the mode output by the trajectory prediction model corresponding to the training sample is determined to be zero. Otherwise, the collision reference parameters are determined based on the ratio of the collision avoidance parameters to the safe distance threshold. The first reference parameter is the result of raising the negative number of the collision reference parameter to the power of the natural constant as the base. The difference between the second value and the collision reference parameter is taken as the second reference parameter; The collision avoidance loss of each mode output by the trajectory prediction model corresponding to each training sample is determined based on the product of the first reference parameter and the second reference parameter.

6. The method according to claim 4, characterized in that, The labeled obstacles in the training samples of the semantic segmentation model are regions whose proportion in the foreground region of the image is greater than a preset area threshold.

7. The method according to claim 1, characterized in that, The trajectory classification loss is determined based on the trajectory prediction results of the multiple categories, including: Based on the trajectory prediction results of multiple categories, determine the prediction category probability distribution of each modality output by the trajectory prediction model; The trajectory classification loss is determined based on the cross-entropy between the predicted class probability distribution for each modality and the corresponding true class label.

8. The method according to claim 7, characterized in that, The trajectory classification loss is determined based on the trajectory prediction results of the multiple categories, including: The sum of the cross-entropy between the predicted class probability distribution of each modality output by the trajectory prediction model and the corresponding true class label of that modality is determined as the total cross-entropy parameter; The trajectory classification loss is determined based on the ratio of the total cross-entropy parameter to the total number of modes.

9. A training device for a trajectory prediction model, characterized in that, The device includes: The sample trajectory prediction module is used to predict training samples based on the initial trajectory prediction model, and obtain trajectory prediction results for multiple categories under different modalities; different modalities are used to represent different flight strategies; A multi-loss determination module is used to determine target guidance loss, collision avoidance loss, and trajectory classification loss based on the trajectory prediction results of the multiple categories; wherein, the target guidance loss is determined based on the trajectory direction and the target direction of the endpoint of the trajectory prediction result, the collision avoidance loss is determined based on the distance between the trajectory point position and the obstacle position of the trajectory prediction result, and the trajectory classification loss is determined based on the trajectory prediction results of different categories under each modality and the corresponding true category label; The target loss determination module is used to determine the target loss based on the target guidance loss, the collision avoidance loss, and the trajectory classification loss. The model parameter iterative adjustment module is used to iteratively adjust the parameters of the initial trajectory prediction model based on the target loss corresponding to the training samples in the sample set, so as to obtain the trained target trajectory prediction model, and use the target trajectory prediction model to determine the trajectory prediction results of different modes.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the training method of the trajectory prediction model according to any one of claims 1-8.

Images