Training method and device of trajectory prediction model, equipment and medium

Abstract

The embodiment of the invention discloses a training method and device of a trajectory prediction model, equipment and a medium. The method comprises the steps of obtaining current environment information of a target object at a current moment and historical environment information of a historical moment; respectively inputting the current environment information and the historical environment information into a trajectory prediction model to obtain a current prediction trajectory and a historical prediction trajectory; determining similarity loss according to the current prediction trajectory and the historical prediction trajectory, wherein the similarity loss is used for describing the difference between the current prediction trajectory and the historical prediction trajectory; and training the trajectory prediction model based on the similarity loss, and performing motion trajectory prediction on the target object based on the trained trajectory prediction model. According to the scheme, the trajectory prediction model is trained based on the similarity loss determined according to the current prediction trajectory and the historical prediction trajectory, the continuity and stability of trajectory prediction in the time sequence can be effectively improved, and improvement of the safety and reliability of an automatic driving system is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to a training method, device, equipment and medium for a trajectory prediction model. Background Art

[0002] Motion prediction is a very important part of the autonomous driving system. It aims to use the motion trajectories observed in the past to build a model to enable the machine to understand the interactions in the surrounding complex traffic environment, so as to accurately predict the motion trajectory of obstacles around the autonomous driving vehicle in the future, thereby assisting the autonomous driving car to plan an accurate, safe and reasonable route.

[0003] In the related art, a deep learning-based method is used for trajectory prediction. This method uses a static training method, where the data of each scene is divided into historical trajectory data and future trajectory data, and each time the historical trajectory used to generate the future motion trajectory is a fixed-length historical trajectory. However, this static prediction method may lead to instability and inconsistency in continuous predictions, and cannot provide continuous and stable prediction results, thus affecting the safety and reliability of the autonomous driving system. Summary of the invention

[0004] The present invention provides a method, apparatus, device and medium for training a trajectory prediction model. The trajectory prediction model is trained based on the similarity loss determined according to the current predicted trajectory and the historical predicted trajectory, which can effectively improve the temporal consistency and stability of the trajectory prediction, and help improve the safety and reliability of the autonomous driving system.

[0005] According to one aspect of the present invention, a method for training a trajectory prediction model is provided, the method comprising:

[0006] Acquire current environmental information of the target object at the current moment and historical environmental information at historical moments; wherein the environmental information includes map road information and historical trajectory information of the target object and a reference object, wherein the reference object refers to other objects located within a preset spatial range of the target object;

[0007] Inputting the current environment information and the historical environment information into a trajectory prediction model respectively to obtain a current predicted trajectory and a historical predicted trajectory;

[0008] Determining a similarity loss according to the current predicted trajectory and the historical predicted trajectory; wherein the similarity loss is used to describe the difference between the current predicted trajectory and the historical predicted trajectory;

[0009] The trajectory prediction model is trained based on the similarity loss, so as to predict the motion trajectory of the target object based on the trained trajectory prediction model.

[0010] According to another aspect of the present invention, there is provided a training device for a trajectory prediction model, comprising:

[0011] An environmental information acquisition module, used to acquire the current environmental information of the target object at the current moment and the historical environmental information at the historical moments; wherein the environmental information includes map road information and the historical trajectory information of the target object and a reference object, wherein the reference object refers to other objects located within a preset spatial range of the target object;

[0012] A predicted trajectory determination module, used to input the current environment information and the historical environment information into a trajectory prediction model respectively to obtain a current predicted trajectory and a historical predicted trajectory;

[0013] A similarity loss determination module, configured to determine a similarity loss according to the current prediction trajectory and the historical prediction trajectory; wherein the similarity loss is used to describe the difference between the current prediction trajectory and the historical prediction trajectory;

[0014] A model training module is used to train the trajectory prediction model based on the similarity loss, so as to train the trajectory prediction model for the target object based on the trained trajectory prediction model.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the trajectory prediction model described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the training method of the trajectory prediction model described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention first obtains the current environmental information of the target object at the current moment and the historical environmental information at the historical moments; wherein the environmental information includes map road information and the historical trajectory information of the target object and the reference object, and the reference object refers to other objects located within the preset spatial range of the target object; then the current environmental information and the historical environmental information are respectively input into the trajectory prediction model to obtain the current predicted trajectory and the historical predicted trajectory; then the similarity loss is determined according to the current predicted trajectory and the historical predicted trajectory; wherein the similarity loss is used to describe the difference between the current predicted trajectory and the historical predicted trajectory; finally, the trajectory prediction model is trained based on the similarity loss to predict the motion trajectory of the target object based on the trained trajectory prediction model. This technical solution, based on the similarity loss determined according to the current predicted trajectory and the historical predicted trajectory, can effectively improve the temporal consistency and stability of the trajectory prediction, and help improve the safety and reliability of the autonomous driving system.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flowchart of a trajectory prediction model training method provided according to Embodiment 1 of the present invention;

[0024] Figure 2A is a schematic diagram of historical trajectory information provided according to Embodiment 1 of the present invention;

[0025] Figure 2B is a schematic diagram of map topology information provided according to the first embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of the composition of a trajectory prediction model provided according to the first embodiment of the present invention;

[0027] Figure 4 is a flowchart of a trajectory prediction model training method provided according to Embodiment 2 of the present invention;

[0028] Figure 5is a training schematic diagram of a trajectory prediction model provided according to Embodiment 2 of the present invention;

[0029] Figure 6 is a schematic diagram of the structure of a trajectory prediction model training device provided according to Embodiment 3 of the present invention;

[0030] Figure 7 It is a structural schematic diagram of an electronic device for implementing a trajectory prediction model training method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 This is a flow chart of a method for training a trajectory prediction model provided in the first embodiment of the present invention. This embodiment is applicable to the case where a trajectory prediction model is trained based on prediction coherence. The method can be executed by a trajectory prediction model training device. The trajectory prediction model training device can be implemented in the form of hardware and / or software. The trajectory prediction model training device can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0035] S110, obtaining current environment information of the target object at the current moment and historical environment information at historical moments.

[0036] Among them, the environmental information includes map road information and historical trajectory information of the target object and the reference object. The reference object refers to other objects located within the preset spatial range of the target object. Among them, the target object may refer to an object for which motion trajectory prediction is desired, such as a motor vehicle, a non-motor vehicle or a pedestrian. A historical moment may refer to a moment that is before the current moment and adjacent to the current moment. Exemplarily, assuming that the current moment is moment t, the historical moment may be moment t-1. The current environmental information and the historical environmental information may refer to environmental information at the current moment and historical moments, respectively. Specifically, the map road information may include lane position information and lane type information (such as going straight, turning left or turning right). The historical trajectory information may be characterized based on the position coordinate information at different moments. Exemplarily, such as Figure 2A As shown, a historical trajectory information is displayed. The preset spatial range may refer to a spatial range pre-set according to actual application requirements, and is used to represent the surrounding space of the target object.

[0037] S120, inputting the current environment information and the historical environment information into the trajectory prediction model respectively to obtain the current predicted trajectory and the historical predicted trajectory.

[0038] Among them, the trajectory prediction model may refer to a deep learning network model for realizing the trajectory prediction function. Optionally, the trajectory prediction model includes a feature extraction unit, a feature fusion unit and a trajectory prediction unit, and the feature extraction unit includes a map feature extraction unit and a trajectory feature extraction unit. Among them, the feature extraction unit can be used to extract features from environmental information, the map feature extraction unit can be used to extract features from map road information in environmental information, and the trajectory feature extraction unit can be used to extract features from historical trajectory information in environmental information. The feature fusion unit can be used to perform feature fusion on the extracted map features and trajectory features, and the trajectory prediction unit can be used to predict the motion trajectory of the target object based on the fused features.

[0039] In this embodiment, the current environment information and the historical environment information are respectively input into the trajectory prediction model, and the trajectory prediction at the corresponding moment is performed through the trajectory prediction model, thereby obtaining the current predicted trajectory and the historical predicted trajectory. Optionally, the current environment information and the historical environment information are respectively input into the trajectory prediction model to obtain the current predicted trajectory and the historical predicted trajectory, including the following steps A1-A4:

[0040] A1. Inputting the map road information in the current environment information and the historical environment information into the map feature extraction unit respectively to obtain the current map feature and the historical map feature.

[0041] Specifically, the map road information in the current environment information is input into the map feature extraction unit of the trajectory prediction model, and the map feature extraction unit performs feature extraction on the map road information in the current environment information to obtain the current map feature; the map road information in the historical environment information is input into the map feature extraction unit of the trajectory prediction model, and the map feature extraction unit performs feature extraction on the map road information in the historical environment information to obtain the historical map feature.

[0042] Optionally, the map feature extraction unit uses a lane graph convolutional network (Lane Graph Convolutional Network, LaneGCN). LaneGCN is a deep learning model based on a graph convolutional network.

[0043] Further, optionally, step A1 includes the following steps A11-A12:

[0044] A11. Extract lane topological relationships from the map road information in the current environment information and the historical environment information, respectively, to obtain current map topological information and historical map topological information.

[0045] The map topology information includes nodes and edges. Nodes represent lane center points, and edges represent lane connection relationships. Specifically, the lane topology relationship is extracted from the map road information in the current environment information, and nodes and edges are used to represent lane center points and lane connection relationships, respectively, so as to obtain the current map topology information; similarly, the lane topology relationship is extracted from the map road information in the historical environment information, so as to obtain the historical map topology information. The extraction of lane topology relationship helps to construct structured map data. For example, Figure 2B As shown, a map topology information is displayed.

[0046] A12. Input the current map topology information and the historical map topology information into the lane graph convolutional network respectively to obtain the current map features and the historical map features.

[0047] Specifically, after obtaining the current map topology information and the historical map topology information, the current map topology information can be input into the lane map convolution network, and the lane features of the current map topology information can be extracted through the lane map convolution network to obtain the current map features; and the historical map topology information can be input into the lane map convolution network, and the lane features of the historical map topology information can be extracted through the lane map convolution network to obtain the historical map features. For example, assuming that the current map topology information is represented by M t , the current map feature is represented as feature map , if the LaneGCN network is used for lane feature extraction, it can be expressed as feature map=LaneGCN(M t ).

[0048] A2. Input the current environment information and the historical trajectory information in the historical environment information into the trajectory feature extraction unit respectively to obtain the current trajectory feature and the historical trajectory feature.

[0049] Specifically, the historical trajectory information in the current environment information is input into the trajectory feature extraction unit, and the trajectory feature extraction unit performs trajectory feature extraction on the historical trajectory information in the current environment information to obtain the current trajectory feature; the historical trajectory information in the historical environment information is input into the trajectory feature extraction unit, and the trajectory feature extraction unit performs trajectory feature extraction on the historical trajectory information in the historical environment information to obtain the historical trajectory feature.

[0050] Optionally, the trajectory feature extraction unit includes a convolution unit and a feature pyramid unit, wherein the convolution unit adopts a convolutional neural network (CNN), and the feature pyramid unit adopts a feature pyramid network (FPN).

[0051] For example, it is assumed that the historical trajectory information in the current environment information is represented as The current trajectory feature is represented as feature obs , if CNN and FPN are used for trajectory feature extraction, it can be expressed as

[0052] A3. Inputting the current map features and the current trajectory features into a feature fusion unit to obtain current fusion features, and inputting the historical map features and the historical trajectory features into a feature fusion unit to obtain historical fusion features.

[0053] For example, the feature fusion unit may use FusionNet (a convolutional neural network), and the current fusion feature is represented as feature fus =FusionNet(feature obs , feature map ). Furthermore, the feature fusion unit includes a first fusion unit, a second fusion unit, a third fusion unit and a fourth fusion unit; wherein the first fusion unit is used to fuse the trajectory feature into the map feature, the second fusion unit is used to fuse different map features, the third fusion unit is used to fuse the map feature into the trajectory feature, and the fourth fusion unit is used to fuse different trajectory features.

[0054] Exemplarily, the first fusion unit adopts A2M (Actor to Map), the second fusion unit adopts M2M (Map to Map), the third fusion unit adopts M2A (Map to Actor), and the fourth fusion unit adopts A2A (Actor to Actor). Among them, A2M, M2M, M2A and A2A are four different types of interaction modules, which together constitute a complex interaction network for capturing and fusing real-time information between participants and lanes to improve the accuracy and reliability of action prediction. Specifically, A2M is used to introduce the real-time traffic information (such as position, speed, direction, etc.) of participants into lane nodes; M2M is used to propagate traffic information on the lane map to update the features of lane nodes, including propagating long-range dependencies between lanes and considering the mutual influence between different lanes; M2A is used to fuse the updated map features (including the features of lane nodes) with real-time traffic information and feed this information back to the participant nodes; A2A is used to process the interactions between participants and generate output participant features, including considering factors such as the relative position, speed, direction, etc. between participants, as well as the potential interaction relationship between them.

[0055] For example, the map features and trajectory features can be fused in the order of A2M, M2M, M2A and A2A. Specifically, taking the current fused features as an example, the current trajectory features are first fused into the current map features using the attention mechanism through the A2M module, which can be specifically expressed as

[0056] Among them, Δ i,j =MLP(v i ,v j ), MLP represents multi-layer perceptron, v represents node position, W represents weight, represents normalization, and concat represents the connection function. Then, the LaneGCN network is used to fuse different current map features through the M2M module, which can be specifically represented as feature map_f =LaneGCN(feature map ). Then, the M2A module uses the attention mechanism to fuse the fused current map features into the current trajectory features, which can be specifically expressed as

[0057] Finally, the A2A module uses the attention mechanism to fuse different current trajectory features, which can be specifically expressed as It should be noted that both the current fusion features and the historical fusion features can be obtained through the above method.

[0058] A4. Input the current fusion feature and the historical fusion feature into the trajectory prediction unit respectively to obtain the current predicted trajectory and the historical predicted trajectory.

[0059] Specifically, the current fusion feature is input into the trajectory prediction unit, and the trajectory prediction unit is used to obtain one or more current prediction trajectories; the historical fusion feature is input into the trajectory prediction unit, and the trajectory prediction unit is used to obtain one or more historical prediction trajectories. The number of current prediction trajectories is the same as that of historical prediction trajectories. Optionally, the trajectory prediction unit includes a residual unit, a linear unit, and an activation function unit. The residual unit can be used to describe a residual connection. Exemplarily, the linear unit can use a multi-layer perceptron (Multi-Layer Perceptron, MLP), and the activation function unit can use a RELU (Rectified Linear Unit, rectified linear unit) function.

[0060] Specifically, a network including residual connections and linear layers is used for trajectory prediction, and a classification probability network is used to determine the probability of the predicted trajectory, wherein the activation function of the classification probability network uses a RELU function. For example, taking the current predicted trajectory as an example, it can be specifically expressed as follows:

[0061]

[0062] Among them, traj current Indicates the current predicted trajectory, cls current Represents the probability of getting the current predicted trajectory, Indicates the current fusion feature.

[0063] Figure 3 FIG. 1 is a schematic diagram of a trajectory prediction model provided in the first embodiment of the present invention. Figure 3 As shown, firstly, the map road information and the historical trajectory information are respectively input into the map feature extraction unit and the trajectory feature extraction unit of the trajectory prediction model, and the map features and trajectory features are obtained through feature extraction. Then, the map features and trajectory features are input into the feature fusion unit of the trajectory prediction model, and the fusion features are obtained through feature fusion. Then, the fusion features are input into the trajectory prediction unit of the trajectory prediction model, and the predicted trajectory is obtained through trajectory prediction.

[0064] S130, determining a similarity loss according to the current prediction trajectory and the historical prediction trajectory.

[0065] The similarity loss is used to describe the difference between the current prediction trajectory and the historical prediction trajectory. For example, the similarity loss can be calculated based on the smooth-L1 loss, which is specifically expressed as follows:

[0066]

[0067] Among them, traj current Represents the current predicted trajectory, traj history Represents the historical prediction trajectory.

[0068] S140, training the trajectory prediction model based on the similarity loss, so as to predict the motion trajectory of the target object based on the trained trajectory prediction model.

[0069] In this embodiment, after obtaining the similarity loss, the similarity loss can be used as one of the loss parameters for model training to train the trajectory prediction model. Optionally, after respectively inputting the current environment information and the historical environment information into the trajectory prediction model to obtain the current predicted trajectory and the historical predicted trajectory, it also includes: determining the first regression loss and the first classification loss corresponding to the current predicted trajectory, and determining the second regression loss and the second classification loss corresponding to the historical predicted trajectory; accordingly, training the trajectory prediction model based on the similarity loss includes: training the trajectory prediction model based on the first regression loss, the first classification loss, the second regression loss, the second classification loss and the similarity loss.

[0070] The first regression loss and the first classification loss may refer to the regression loss and the classification loss corresponding to the current prediction trajectory, respectively. The second regression loss and the second classification loss may refer to the regression loss and the classification loss corresponding to the historical prediction trajectory, respectively.

[0071] For example, the regression loss can be calculated based on the smooth-L1 loss, which is specifically expressed as:

[0072]

[0073] Among them, Traj pre Indicates the predicted trajectory (such as the current predicted trajectory or the historical predicted trajectory), Traj gt represents the true trajectory label.

[0074] For example, if there are multiple current prediction trajectories and multiple historical prediction trajectories, the classification loss can be calculated based on the maximum boundary loss, which is specifically expressed as:

[0075]

[0076] Where M represents the number of reference objects (i.e., the number of obstacles predicted in a set of data), K represents the number of current predicted trajectories or the number of historical predicted trajectories of a reference object (i.e., obstacle), and k ′represents the maximum probability, c represents the probability of the predicted trajectory (i.e., cls in the above text), and ε is the preset parameter value. For example, assuming that for a set of current information input (i.e., current environment information and historical environment information), there are a total of 10 obstacles, of which 8 obstacles are predicted, and 6 trajectories are predicted for each obstacle, then M is equal to 8 and K is equal to 6.

[0077] In this embodiment, a training data set is first obtained, and then the training data set is divided according to a preset ratio (such as 8:1:1) to obtain a training set, a validation set, and a test set corresponding to two adjacent moments. Each set includes multiple input-output groups, the input parameter is the environmental information of the target object, and the output parameter is the motion trajectory of the target object corresponding to the environmental information. Then, the trajectory prediction model is trained based on the training set and the validation set, and the model performance is verified based on the test set.

[0078] In each model training process, the current environment information and historical environment information of the target object are respectively input into the trajectory prediction model to obtain the current predicted trajectory and the historical predicted trajectory, and then the first regression loss and the first classification loss corresponding to the current predicted trajectory, as well as the second regression loss and the second classification loss corresponding to the historical predicted trajectory are calculated, and the similarity loss is calculated at the same time, and then the trajectory prediction model is trained based on the first regression loss, the first classification loss, the second regression loss, the second classification loss and the similarity loss. When the number of model training times reaches the preset number of times or the model training loss (including the above 5 types of losses) is less than the preset loss threshold, the model training can be stopped, and the model at this time is used as the final trajectory prediction model, so as to continuously predict the motion trajectory of the target object at future moments through the trained trajectory prediction model. It should be noted that by combining the trajectory prediction results of the previous and next moments, the robustness of the model is increased while enhancing the correlation between the previous and next trajectory predictions.

[0079] The technical solution of the embodiment of the present invention first obtains the current environmental information of the target object at the current moment and the historical environmental information at the historical moments; wherein the environmental information includes map road information and the historical trajectory information of the target object and the reference object, and the reference object refers to other objects located within the preset spatial range of the target object; then the current environmental information and the historical environmental information are respectively input into the trajectory prediction model to obtain the current predicted trajectory and the historical predicted trajectory; then the similarity loss is determined according to the current predicted trajectory and the historical predicted trajectory; wherein the similarity loss is used to describe the difference between the current predicted trajectory and the historical predicted trajectory; finally, the trajectory prediction model is trained based on the similarity loss to predict the motion trajectory of the target object based on the trained trajectory prediction model. This technical solution, based on the similarity loss determined according to the current predicted trajectory and the historical predicted trajectory, can effectively improve the temporal consistency and stability of the trajectory prediction, and help improve the safety and reliability of the autonomous driving system.

[0080] Embodiment 2

[0081] Figure 4 A flowchart of a trajectory prediction model training method provided in the second embodiment of the present invention, which is optimized based on the above embodiment. The specific optimization is: the number of current predicted trajectories and historical predicted trajectories is the same and is at least two; accordingly, the similarity loss is determined according to the current predicted trajectory and the historical predicted trajectory, including: matching the current predicted trajectory with the historical predicted trajectory; determining the similarity loss according to the successfully matched current predicted trajectory and the historical predicted trajectory.

[0082] like Figure 4 As shown, the method of this embodiment specifically includes the following steps:

[0083] S210, obtaining current environment information of the target object at the current moment and historical environment information at historical moments.

[0084] The environmental information includes map road information and historical trajectory information of the target object and the reference object. The reference object refers to other objects located within the preset spatial range of the target object.

[0085] S220, inputting the current environment information and the historical environment information into the trajectory prediction model respectively to obtain the current predicted trajectory and the historical predicted trajectory.

[0086] The number of the current prediction track and the number of the historical prediction track are the same and are at least two, thereby realizing multimodal prediction of the model, thereby improving the applicability of the model. For example, the number of the current prediction track and the number of the historical prediction track can both be set to 6.

[0087] S230: Match the current predicted trajectory with the historical predicted trajectory.

[0088] Exemplarily, a two-way matching algorithm can be used to match multiple current prediction trajectories with historical prediction trajectories. Specifically, multiple current prediction trajectories are taken as set C, and multiple historical prediction trajectories are taken as set D. If the c-trajectory in set C and the d-trajectory in set D are the closest prediction points in both sets, it means that the c-trajectory and the d-trajectory are matched successfully.

[0089] S240, determining a similarity loss according to the successfully matched current prediction trajectory and the historical prediction trajectory.

[0090] The similarity loss is used to describe the difference between the current prediction trajectory and the historical prediction trajectory. In this embodiment, the similarity loss is calculated for each group of successfully matched current prediction trajectory and historical prediction trajectory, and multiple similarity losses can be obtained.

[0091] S250, training the trajectory prediction model based on the similarity loss, so as to predict the motion trajectory of the target object based on the trained trajectory prediction model.

[0092] Figure 5 A training diagram of a trajectory prediction model provided in the second embodiment of the present invention. Figure 5 As shown, first, the current environment information and the historical environment information are respectively input into the shared trajectory prediction model to obtain multiple current prediction trajectories and multiple historical prediction trajectories of the same number; then, the multiple current prediction trajectories and the multiple historical prediction trajectories are matched, and the similarity loss is calculated based on the successfully matched current prediction trajectories and historical prediction trajectories; then, the trajectory prediction model is trained based on the similarity loss, so that the model can learn the trajectory similarity relationship between the previous and next moments, thereby improving the temporal continuity and stability of trajectory prediction.

[0093] In the technical solution of the embodiment of the present invention, the number of the current predicted trajectory and the number of the historical predicted trajectory are the same and are at least two; when determining the similarity loss based on the current predicted trajectory and the historical predicted trajectory, the current predicted trajectory and the historical predicted trajectory are first track matched; then the similarity loss is determined based on the successfully matched current predicted trajectory and the historical predicted trajectory; and then the trajectory prediction model is trained based on the similarity loss. This technical solution can realize multi-modal prediction of the trajectory prediction model, thereby improving the applicability of the trajectory prediction model.

[0094] Embodiment 3

[0095] Figure 6This is a schematic diagram of the structure of a trajectory prediction model training device provided in the third embodiment of the present invention. The device can execute the trajectory prediction model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 6 As shown, the device comprises:

[0096] The environment information acquisition module 310 is used to acquire the current environment information of the target object at the current moment and the historical environment information at the historical moments; wherein the environment information includes map road information and the historical trajectory information of the target object and the reference object, wherein the reference object refers to other objects located within the preset spatial range of the target object;

[0097] A predicted trajectory determination module 320, configured to input the current environment information and the historical environment information into a trajectory prediction model to obtain a current predicted trajectory and a historical predicted trajectory;

[0098] A similarity loss determination module 330, configured to determine a similarity loss according to the current prediction trajectory and the historical prediction trajectory; wherein the similarity loss is used to describe the difference between the current prediction trajectory and the historical prediction trajectory;

[0099] The model training module 340 is used to train the trajectory prediction model based on the similarity loss, so as to train the trajectory prediction model for the target object based on the trained trajectory prediction model.

[0100] Optionally, the trajectory prediction model includes a feature extraction unit, a feature fusion unit and a trajectory prediction unit, and the feature extraction unit includes a map feature extraction unit and a trajectory feature extraction unit;

[0101] The predicted trajectory determination module 320 is used to:

[0102] Inputting the map road information in the current environment information and the historical environment information into the map feature extraction unit respectively to obtain the current map feature and the historical map feature;

[0103] Inputting the current environment information and the historical trajectory information in the historical environment information into the trajectory feature extraction unit respectively to obtain the current trajectory feature and the historical trajectory feature;

[0104] Inputting the current map feature and the current trajectory feature into the feature fusion unit to obtain a current fusion feature, and inputting the historical map feature and the historical trajectory feature into the feature fusion unit to obtain a historical fusion feature;

[0105] The current fusion feature and the historical fusion feature are respectively input into the trajectory prediction unit to obtain a current predicted trajectory and a historical predicted trajectory.

[0106] Optionally, the map feature extraction unit adopts a lane graph convolutional network;

[0107] The predicted trajectory determination module 320 is further used to:

[0108] Extracting lane topology relationships from the map road information in the current environment information and the historical environment information, respectively, to obtain current map topology information and historical map topology information; wherein the map topology information includes nodes and edges, the nodes represent lane center points, and the edges represent lane connection relationships;

[0109] The current map topology information and the historical map topology information are respectively input into the lane graph convolutional network to obtain current map features and historical map features.

[0110] Optionally, the feature fusion unit includes a first fusion unit, a second fusion unit, a third fusion unit and a fourth fusion unit; wherein the first fusion unit is used to fuse trajectory features into map features, the second fusion unit is used to fuse different map features, the third fusion unit is used to fuse map features into trajectory features, and the fourth fusion unit is used to fuse different trajectory features.

[0111] Optionally, the trajectory feature extraction unit includes a convolution unit and a feature pyramid unit; the trajectory prediction unit includes a residual unit, a linear unit and an activation function unit.

[0112] Optionally, the number of the current prediction trajectory and the number of the historical prediction trajectory are the same and are at least two;

[0113] The similarity loss determination module 330 is used to:

[0114] Track matching between the current predicted track and the historical predicted track;

[0115] The similarity loss is determined based on the current prediction trajectory and the historical prediction trajectory that are successfully matched.

[0116] Optionally, the device further comprises:

[0117] A regression and classification loss determination module, configured to determine a first regression loss and a first classification loss corresponding to the current predicted trajectory, and determine a second regression loss and a second classification loss corresponding to the historical predicted trajectory, after respectively inputting the current environment information and the historical environment information into a trajectory prediction model to obtain a current predicted trajectory and a historical predicted trajectory;

[0118] The model training module 340 is used to:

[0119] The trajectory prediction model is trained based on the first regression loss, the first classification loss, the second regression loss, the second classification loss, and the similarity loss.

[0120] A trajectory prediction model training device provided in an embodiment of the present invention can execute a trajectory prediction model training method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0121] Embodiment 4

[0122] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0123] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0125] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the training method of the trajectory prediction model.

[0126] In some embodiments, the training method of the trajectory prediction model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method of the trajectory prediction model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the training method of the trajectory prediction model in any other appropriate manner (e.g., by means of firmware).

[0127] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0129] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0131] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0132] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0133] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0134] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for training a trajectory prediction model, characterized in that: The method comprises: Acquire current environmental information of the target object at the current moment and historical environmental information at historical moments; wherein the environmental information includes map road information and historical trajectory information of the target object and a reference object, wherein the reference object refers to other objects located within a preset spatial range of the target object; Inputting the current environment information and the historical environment information into a trajectory prediction model respectively to obtain a current predicted trajectory and a historical predicted trajectory; Determining a similarity loss according to the current prediction trajectory and the historical prediction trajectory; wherein the similarity loss is used to describe the difference between the current prediction trajectory and the historical prediction trajectory; The trajectory prediction model is trained based on the similarity loss, so as to predict the motion trajectory of the target object based on the trained trajectory prediction model.

2. The method according to claim 1, characterized in that The trajectory prediction model includes a feature extraction unit, a feature fusion unit and a trajectory prediction unit, and the feature extraction unit includes a map feature extraction unit and a trajectory feature extraction unit; Inputting the current environment information and the historical environment information into the trajectory prediction model respectively to obtain the current predicted trajectory and the historical predicted trajectory, including: Inputting the map road information in the current environment information and the historical environment information into the map feature extraction unit respectively to obtain the current map feature and the historical map feature; Inputting the current environment information and the historical trajectory information in the historical environment information into the trajectory feature extraction unit respectively to obtain the current trajectory feature and the historical trajectory feature; Inputting the current map feature and the current trajectory feature into the feature fusion unit to obtain a current fusion feature, and inputting the historical map feature and the historical trajectory feature into the feature fusion unit to obtain a historical fusion feature; The current fusion feature and the historical fusion feature are respectively input into the trajectory prediction unit to obtain a current predicted trajectory and a historical predicted trajectory.

3. The method according to claim 2, characterized in that The map feature extraction unit adopts a lane graph convolutional network; Inputting the map road information in the current environment information and the historical environment information into the map feature extraction unit respectively to obtain the current map feature and the historical map feature, including: Extracting lane topology relationships from the map road information in the current environment information and the historical environment information, respectively, to obtain current map topology information and historical map topology information; wherein the map topology information includes nodes and edges, the nodes represent lane center points, and the edges represent lane connection relationships; The current map topology information and the historical map topology information are respectively input into the lane graph convolutional network to obtain current map features and historical map features.

4. The method according to claim 2, characterized in that: The feature fusion unit includes a first fusion unit, a second fusion unit, a third fusion unit and a fourth fusion unit; wherein the first fusion unit is used to fuse trajectory features into map features, the second fusion unit is used to fuse different map features, the third fusion unit is used to fuse map features into trajectory features, and the fourth fusion unit is used to fuse different trajectory features.

5. The method according to claim 2, characterized in that: The trajectory feature extraction unit includes a convolution unit and a feature pyramid unit; the trajectory prediction unit includes a residual unit, a linear unit and an activation function unit.

6. The method according to claim 1, characterized in that The number of the current prediction trajectories and the number of the historical prediction trajectories are the same and are at least two; Determining a similarity loss according to the current prediction trajectory and the historical prediction trajectory includes: Track matching between the current predicted track and the historical predicted track; The similarity loss is determined based on the current prediction trajectory and the historical prediction trajectory that are successfully matched.

7. The method according to any one of claims 1 to 6, characterized in that: After the current environment information and the historical environment information are respectively input into the trajectory prediction model to obtain the current predicted trajectory and the historical predicted trajectory, the method further includes: Determine a first regression loss and a first classification loss corresponding to the current prediction trajectory, and determine a second regression loss and a second classification loss corresponding to the historical prediction trajectory; Training the trajectory prediction model based on the similarity loss includes: The trajectory prediction model is trained based on the first regression loss, the first classification loss, the second regression loss, the second classification loss, and the similarity loss.

8. A training device for a trajectory prediction model, characterized in that: The device comprises: An environmental information acquisition module, used to acquire the current environmental information of the target object at the current moment and the historical environmental information at the historical moments; wherein the environmental information includes map road information and the historical trajectory information of the target object and the reference object, wherein the reference object refers to other objects located within the preset spatial range of the target object; A predicted trajectory determination module, used to input the current environment information and the historical environment information into a trajectory prediction model respectively to obtain a current predicted trajectory and a historical predicted trajectory; A similarity loss determination module, configured to determine a similarity loss according to the current prediction trajectory and the historical prediction trajectory; wherein the similarity loss is used to describe the difference between the current prediction trajectory and the historical prediction trajectory; A model training module is used to train the trajectory prediction model based on the similarity loss, so as to train the trajectory prediction model for the target object based on the trained trajectory prediction model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the trajectory prediction model training method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the training method of the trajectory prediction model according to any one of claims 1 to 7 when executed.

Images