Trajectory prediction method and device, electronic equipment and storage medium

Abstract

The application provides a trajectory prediction method and device, electronic equipment and a storage medium, and relates to the technical field of automatic driving. The method comprises the following steps: acquiring a first monitoring trajectory of a first obstacle around a target vehicle in a first target sampling period; acquiring at least one historical prediction trajectory of the first obstacle; the historical prediction trajectory is obtained by predicting the trajectory of the first obstacle after a second monitoring trajectory of the first obstacle in a historical sampling period before the first target sampling period; the first monitoring trajectory and the at least one historical prediction trajectory of the first obstacle are fused to obtain a fused trajectory feature; and the first prediction trajectory of the first obstacle after the first monitoring trajectory is predicted according to the fused trajectory feature. The first prediction trajectory is predicted through the fused trajectory feature, the first prediction trajectory is smoothed based on the historical prediction trajectory, and the accuracy of continuous trajectory prediction is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a trajectory prediction method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Trajectory prediction is an important link in an automatic driving system. In related technologies, the driving trajectory of a surrounding obstacle in the future several seconds is inferred by obtaining the perception information of the surrounding obstacle by an upstream perception module. In actual application scenarios, the driving trajectory of the surrounding obstacle in the future needs to be continuously predicted, and when continuous trajectory prediction is performed, the position and quantity of the obstacle do not change greatly, but the predicted trajectory is completely different, resulting in a jump in the continuously predicted trajectory and poor accuracy. SUMMARY

[0003] The present application provides a trajectory prediction method, device, electronic device, and storage medium, which avoids the problem of a jump in the predicted trajectory when continuous trajectory prediction is performed, and improves the accuracy of continuous trajectory prediction.

[0004] In one aspect, an embodiment of the present application provides a trajectory prediction method, comprising:

[0005] obtaining a first monitoring trajectory of a first obstacle around a target vehicle in a first target sampling period;

[0006] obtaining at least one historical predicted trajectory of the first obstacle; the historical predicted trajectory is obtained by predicting the trajectory of the first obstacle after a second monitoring trajectory of the first obstacle in a historical sampling period before the first target sampling period;

[0007] performing feature fusion on the first monitoring trajectory and the at least one historical predicted trajectory of the first obstacle to determine a fusion trajectory feature of the first obstacle;

[0008] predicting a first predicted trajectory of the first obstacle after the first monitoring trajectory according to the fusion trajectory feature of the first obstacle.

[0009] In another aspect, an embodiment of the present application provides a trajectory prediction device, comprising:

[0010] a first obtaining module configured to obtain a first monitoring trajectory of a first obstacle around a target vehicle in a first target sampling period;

[0011] a second obtaining module, configured to obtain at least one historical predicted trajectory of the first obstacle; the historical predicted trajectory is obtained by predicting a trajectory of the first obstacle after a second monitoring trajectory of the first obstacle in a historical sampling period before the first target sampling period according to the second monitoring trajectory;

[0012] a fusion module, configured to perform feature fusion on the first monitoring trajectory of the first obstacle and the at least one historical predicted trajectory to determine a fused trajectory feature of the first obstacle;

[0013] a prediction module, configured to predict a first predicted trajectory of the first obstacle after the first monitoring trajectory according to the fused trajectory feature of the first obstacle.

[0014] Another aspect of the present application provides a vehicle, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the program, the trajectory prediction method in the foregoing aspect is implemented.

[0015] Another aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the program, the trajectory prediction method in the foregoing aspect is implemented.

[0016] Another aspect of the present application provides a non-transitory computer readable storage medium, having a computer program stored thereon, and when the computer program is executed by a processor, the trajectory prediction method in the foregoing aspect is implemented.

[0017] Another aspect of the present application provides a computer program product, having a computer program stored thereon, and when the computer program is executed by a processor, the trajectory prediction method in the foregoing aspect is implemented.

[0018] The trajectory prediction method, device, electronic equipment and storage medium provided in the application obtain a first monitoring trajectory of a first obstacle around a target vehicle in a first target sampling period, obtain at least one historical prediction trajectory of the first obstacle, wherein the historical prediction trajectory is obtained by predicting the trajectory of the first obstacle after a second monitoring trajectory of the first obstacle in a historical sampling period before the first target sampling period, the first monitoring trajectory and the at least one historical prediction trajectory of the first obstacle are fused to determine a fusion trajectory feature of the first obstacle, and the first prediction trajectory of the first obstacle after the first monitoring trajectory is predicted according to the fusion trajectory feature of the first obstacle. By fusing the at least one historical prediction trajectory and the first monitoring trajectory and predicting the first prediction trajectory after the first monitoring trajectory based on the fusion trajectory feature, the first prediction trajectory is smoothed based on the historical prediction trajectory, the jumping phenomenon is avoided, and the accuracy of continuous trajectory prediction is improved.

[0019] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and / or additional aspects and advantages of the application will become apparent and be readily understood by considering the following detailed description, including the accompanying drawings, in which:

[0021] Figure 1 A flowchart of a trajectory prediction method provided by an embodiment of the application;

[0022] Figure 2 A timing diagram of a trajectory prediction provided by an embodiment of the application;

[0023] Figure 3 A flowchart of another estimation prediction method provided by an embodiment of the application;

[0024] Figure 4 One of the scene diagrams of a trajectory prediction provided by an embodiment of the application;

[0025] Figure 5 Another scene diagram of a trajectory prediction provided by an embodiment of the application;

[0026] Figure 6 A flowchart of a training method of a prediction model provided by an embodiment of the application;

[0027] Figure 7 A flowchart of another training method of a prediction model provided by an embodiment of the application;

[0028] Figure 8A structural schematic diagram of a trajectory prediction device provided by an embodiment of the present application is shown in the figure.

[0029] Figure 9 A block diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0030] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, in which the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0031] A trajectory prediction method, device, electronic device and storage medium of an embodiment of the present application are described below with reference to the accompanying drawings.

[0032] Figure 1 A flowchart of a trajectory prediction method provided by an embodiment of the present application is shown in the figure.

[0033] The execution subject of the trajectory prediction method of the present embodiment is a trajectory prediction device, which can be arranged in an electronic device. The electronic device can be any stationary or mobile computing device capable of data processing, such as a notebook computer, a smart phone, a wearable device, or a stationary computing device such as a desktop computer, or a server, or other types of vehicle-mounted devices, etc., and the present disclosure does not limit the electronic device. The vehicle-mounted terminal mentioned above refers to a terminal device that can be installed in a vehicle, such as a smart vehicle system.

[0034] As shown in the figure, the method can include the following steps: Figure 1

[0035] Step 101: Obtain a first monitoring trajectory of a first obstacle around a target vehicle in a first target sampling period.

[0036] The target vehicle is a vehicle whose trajectory around obstacles needs to be predicted.

[0037] The surrounding environment of the target vehicle usually contains at least one obstacle, and the at least one obstacle around the target vehicle can be determined based on the perception information corresponding to the first target sampling period. In the present embodiment, one obstacle is taken as an example for illustration, which is referred to as the first obstacle. The first obstacle can be other vehicles, pedestrians, buildings, etc. around the target vehicle, and the present embodiment does not limit the first obstacle.

[0038] ​In an implementation form of the embodiment of the application, the perception information of the surrounding environment collected by the plurality of sensors installed on the target vehicle at the first target sampling time is acquired, wherein the sensors include image sensors, laser radar sensors, etc. The perception information contains information of at least one first obstacle around the target vehicle. Then, based on the perception information of the surrounding environment collected at the first target sampling time, the first obstacle around the target vehicle is identified, and a first monitoring trajectory corresponding to the first obstacle at the first target sampling time is determined, wherein the first monitoring trajectory of the first obstacle contains trajectory points of the obstacle, and each trajectory point indicates the position of the obstacle in the vehicle coordinate system.

[0039] In step 102, at least one historical prediction trajectory of the first obstacle is acquired.

[0040] The historical prediction trajectory is obtained by predicting the trajectory of the first obstacle after the second monitoring trajectory according to the second monitoring trajectory of the first obstacle in the historical sampling period before the first target sampling period. The historical prediction trajectory can be one or more, which is not limited in the embodiment.

[0041] In one scenario, if the first target sampling time is the initial sampling time, there is no historical sampling time before the first target sampling time, and the first obstacle has no corresponding at least one historical prediction trajectory. That is, the trajectory obtained by subsequent trajectory fusion is still the first monitoring trajectory corresponding to the first obstacle.

[0042] In another scenario, if the first target sampling time is not the initial sampling time, there is at least one historical sampling time before the first target sampling time, the second monitoring trajectory corresponding to the at least one historical sampling time can be acquired, and the corresponding historical prediction trajectory is predicted based on the second monitoring trajectory.

[0043] As an example, Figure 2 A timing diagram of trajectory prediction provided by the embodiment of the application is as follows, Figure 2As shown, taking 2 historical sampling time points as an example, in order to distinguish, they are called historical sampling period 1 and historical sampling period 2, and the historical sampling period 2 is a historical period before the historical sampling period 1. Wherein, the first target sampling period is a certain target period, in the embodiment, the frame indicates the time length of the first target sampling period, for example, the first target sampling period corresponds to the time of t100-t150 frames, then the historical sampling period 1 corresponds to the time of t50-t100 frames, and the historical sampling period 2 corresponds to the time of t0-t50 frames, so that the first monitoring trajectory is determined according to the perception information corresponding to the time of t100-t150 frames, and the historical second monitoring trajectory 1 is determined according to the perception information corresponding to the time of t50-t100 frames, and the historical second monitoring trajectory 2 is determined according to the perception information corresponding to the time of t0-t50 frames, in the embodiment, the historical prediction trajectory 1 corresponding to the time of t100-t150 frames is obtained based on the second monitoring trajectory 1, and the historical prediction trajectory 2 corresponding to the time of t50-t100 frames is obtained based on the second monitoring trajectory 2, and then the historical prediction trajectory 1, the historical prediction trajectory 2 and the first monitoring trajectory are fused to obtain the fusion trajectory feature, and the fusion trajectory feature is used to predict the first prediction trajectory.

[0044] It should be noted that, in the embodiment, the frame time length corresponding to the monitoring trajectory and the prediction trajectory is taken as an example for illustration, for example, the historical sampling period 1, that is, the second monitoring trajectory 1 corresponding to the time of t50-t100 frames, the historical prediction trajectory 1 corresponding to the time of t100-t150 frames is predicted, in the actual application scenario, the frame time length corresponding to the prediction trajectory can be longer than the monitoring trajectory, for example, Figure 2 As shown, the historical sampling period 2, that is, the second monitoring trajectory 2 corresponding to the time of t0-t50 frames, corresponds to the historical prediction trajectory 2' corresponding to the time of t50-t120 frames, and the historical sampling period 1, that is, the second monitoring trajectory 1 corresponding to the time of t50-t100 frames, corresponds to the historical prediction trajectory 1' corresponding to the time of t100-t170 frames, and then the historical prediction trajectory 1', the historical prediction trajectory 2' and the first monitoring trajectory are fused to obtain the fusion trajectory feature, and the fusion trajectory feature is used to predict the first prediction trajectory'.

[0045] Step 103, fusing the first monitoring trajectory of the first obstacle and at least one historical prediction trajectory to determine the fusion trajectory feature of the first obstacle.

[0046] In the embodiment of the present application, the first monitoring track corresponding to the first obstacle is converted into a corresponding feature vector, and the feature vectors corresponding to at least one historical prediction track are fused, and the fusion can be based on the same dimension feature vectors to obtain the fusion track feature corresponding to the first obstacle. The fusion track feature contains the features of the historical prediction track of the first obstacle, so that when the track of the obstacle is continuously predicted subsequently, the features of multiple historical prediction tracks are referred to, and the difference between the prediction results is avoided when the future track of the same first obstacle is continuously predicted, which leads to the jump of the prediction result and reduces the prediction accuracy.

[0047] As an example, the at least one historical prediction track is two historical prediction tracks, as shown in the figure, which are historical prediction track 1 and historical prediction track 2. The historical prediction track 2, the historical prediction track 1 and the first monitoring track are fused to obtain the fusion track feature corresponding to the first obstacle. Figure 2

[0048] Step 104, according to the fusion track feature of the first obstacle, a first prediction track of the first obstacle after the first monitoring track is predicted.

[0049] In actual scenarios, the positions of surrounding obstacles will not change greatly in a short time, and the track of the obstacle should not change greatly during the continuous prediction of the track of the obstacle around the target vehicle, so that in the embodiment of the present application, in the scenario of continuous prediction of the future track based on the first monitoring track of the first obstacle, the features of at least one historical prediction track corresponding to the first obstacle and the track features corresponding to the first monitoring track of the first obstacle are fused to obtain the fusion track feature, and based on the fusion track feature, a smooth first prediction track of the first obstacle after the first target sampling period is predicted, that is, the first prediction track after the first monitoring track. At the same time, in actual scenarios, the tracks of at least one obstacle around the target vehicle need to be referred to each other, therefore, based on the fusion track feature of at least one first obstacle, the mutual influence between at least one first obstacle is considered, and the accuracy of the first prediction track predicted by each first obstacle is improved.

[0050] ​In the trajectory prediction method of the embodiment of the present application, the first monitoring trajectory of the first obstacle around the target vehicle in the first target sampling period is obtained, and at least one historical prediction trajectory of the first obstacle is obtained, wherein the historical prediction trajectory is obtained by predicting the trajectory of the first obstacle after the second monitoring trajectory according to the second monitoring trajectory of the first obstacle in the historical sampling period before the first target sampling period, the fusion trajectory feature is obtained by performing feature fusion on the first monitoring trajectory and the at least one historical prediction trajectory of the first obstacle, and the first prediction trajectory of the first obstacle after the first monitoring trajectory is predicted according to the fusion trajectory feature. The first prediction trajectory is predicted through the fusion trajectory feature after the first monitoring trajectory, which realizes the smoothing of the first prediction trajectory based on the historical prediction trajectory, makes the continuous trajectory predicted more stable, avoids the jump phenomenon of the predicted trajectory, and improves the accuracy of the continuous trajectory prediction.

[0051] Based on the above embodiment, Figure 3 Another flowchart of an estimation prediction method provided by the embodiment of the present application is provided, which is described by taking the trajectory prediction based on the prediction model as an example. As shown in the figure, Figure 3 The method comprises the following steps:

[0052] In step 301, the first monitoring trajectory of the first obstacle around the target vehicle in the first target sampling period is obtained.

[0053] In step 302, at least one historical prediction trajectory of the first obstacle is obtained.

[0054] The steps 301 and 302 can refer to the explanation and description in the foregoing embodiment, and the principle is the same, which will not be described here.

[0055] In step 303, the first monitoring trajectory and each historical prediction trajectory corresponding to the first obstacle are input into the fusion module of the prediction model for feature extraction, so as to extract the first trajectory feature of the first monitoring trajectory and the historical prediction trajectory feature of each historical prediction trajectory.

[0056] The fusion module is a neural network constructed based on a multilayer perceptron (MLP).

[0057] The fusion module is used for feature extraction of the first monitoring trajectory, and the extracted feature is used to indicate the information of the trajectory change of the first obstacle in the first target sampling period.

[0058] The fusion module is used for feature extraction of each historical prediction trajectory, and the extracted feature is used to indicate the information of the trajectory change predicted based on the second monitoring trajectory monitored by the first obstacle in the historical sampling period.

[0059] At step 304, the first track feature and the historical predicted track features of each segment are input into a fusion module of the prediction model for feature fusion to obtain a fused track feature of the first obstacle.

[0060] The fusion weights of the historical predicted track features of each segment can be set. As an example, the fusion weights of the historical predicted track features of each segment can be set to be the same. As another example, the fusion weights of the historical predicted track features of each segment can be determined based on the prediction time periods corresponding to the historical predicted track features. The closer the prediction time period and the first target sampling time period, the greater the weight of the historical predicted track feature corresponding to the prediction time period. The historical predicted track of the adjacent time can be used to smooth and correct the first track feature, which can improve the accuracy of the fused track feature. Compared with the method of predicting the first predicted track only based on the first monitored track, the application fully considers the change of the historical predicted track, which can improve the accuracy of the subsequent first predicted track prediction and avoid the jump of the continuous track prediction.

[0061] The prediction model is pre-trained, and the training method will be described in detail in subsequent embodiments.

[0062] At step 305, the fused track feature of the first obstacle is input into a prediction module of the prediction model to predict a first predicted track of the first obstacle after the first monitored track.

[0063] The prediction module can be a Transform-based model.

[0064] To further illustrate, as an example, Figure 4 As shown in FIG. 1, a scene diagram of a track prediction provided by an embodiment of the application is shown. Figure 4 As shown in FIG. 1, the first obstacle around the target vehicle can be multiple. In this embodiment, two first obstacles are taken as an example for illustration, which are referred to as first obstacle 1 and first obstacle 2. In the feature fusion stage by the fusion module, at least one historical predicted track of the first obstacle 1 is, for example, four historical predicted tracks obtained by continuous prediction, which are referred to as historical predicted track 1, historical predicted track 2, historical predicted track 3 and historical predicted track 4. For the first obstacle 1, the first monitored track corresponding to the first obstacle 1, the historical predicted track 1, the historical predicted track 2, the historical predicted track 3 and the historical predicted track 4 are input into the fusion module for feature fusion to obtain the fused track feature of the first obstacle 1 after fusion. Similarly, the fused track feature of the first obstacle 2 can be obtained.

[0065] Further, Figure 5 As shown in FIG. 2, a scene diagram of a track prediction provided by an embodiment of the application is shown. Figure 5As shown, still taking the first obstacle 1 and the first obstacle 2 as an example for trajectory prediction, the trajectory prediction principles of other first obstacles are the same, which will not be repeated here. Wherein, the fusion trajectory features of the first obstacle 1 and the fusion trajectory features of the first obstacle 2 are input into the prediction module for prediction to obtain the first predicted trajectory of the first obstacle 1 and the first predicted trajectory of the first obstacle 2, realizing trajectory prediction in the future set time based on the fusion trajectory features, improving the stability of the predicted trajectory predicted each time in continuous prediction, and improving the reliability in continuous trajectory prediction.

[0066] In the embodiment of the application, the fusion trajectory features of the first obstacle are predicted based on the trained prediction model to predict the first predicted trajectory of the first obstacle after the first monitoring trajectory. Since the historical predicted trajectory is considered when performing trajectory prediction, the current first monitoring trajectory is smoothed and corrected through the historical predicted trajectory, avoiding a large change of the currently predicted trajectory relative to the historical predicted trajectory, so that when continuously predicting the trajectory of the same obstacle, the problem of jump of the predicted trajectory is avoided, and the reliability of trajectory prediction is improved.

[0067] In the trajectory prediction method of the embodiment of the application, the historical predicted trajectory is considered when performing trajectory prediction, the current first monitoring trajectory is smoothed and corrected through the historical predicted trajectory, a large change of the currently predicted trajectory relative to the historical predicted trajectory is avoided, so that when continuously predicting the trajectory of the same obstacle, the problem of jump of the predicted trajectory is avoided, and the reliability of trajectory prediction is improved.

[0068] Based on the above embodiment, the embodiment of the application provides a training method of a prediction model, Figure 6 A flowchart of a training method of a prediction model provided by the embodiment of the application is shown in FIG. 6. Figure 6 As shown, the method comprises the following steps:

[0069] Step 601, obtaining a training sample.

[0070] The training sample comprises a third monitoring trajectory of the second obstacle in a second target sampling period and at least one historical predicted trajectory of the second obstacle, wherein the historical predicted trajectory is obtained by predicting the trajectory of the second obstacle after a fourth monitoring trajectory of the second obstacle in a historical sampling period before the second target sampling period. For details, refer to the description of the at least one historical predicted trajectory of the first obstacle in the foregoing embodiment, which has the same principle and will not be repeated here.

[0071] The training sample is labeled with a target real trajectory of the second obstacle after the third monitoring trajectory and a historical real trajectory of the second obstacle in a historical sampling period before the second target sampling period. The target real trajectory is a real trajectory in a set period after the second target sampling period, and the time length of the set period and the time length of the second target sampling period can be the same or different. The time length of the historical sampling period and the second target sampling period can be the same.

[0072] It should be noted that the second obstacle included in the training sample used in the prediction model training process can be the same obstacle as the first obstacle or a different obstacle, and is referred to as the second obstacle for distinction. The second target sampling time can be the same as the first target sampling period, or can be different, which is not limited here. Therefore, the monitoring trajectory of the second obstacle in the second target sampling period is referred to as the second monitoring trajectory for distinction.

[0073] In step 602, the training sample is input into the prediction model to obtain a second predicted trajectory of the second obstacle after the third monitoring trajectory.

[0074] The process of predicting the second predicted trajectory of the second obstacle after the second target sampling period, i.e., after the third monitoring trajectory, based on the third monitoring trajectory of the second obstacle and at least one historical predicted trajectory of the second obstacle can be referred to the explanation and description in the foregoing embodiments, and the principle is the same, which will not be repeated here.

[0075] In step 603, a target loss function of the prediction model is determined according to the second predicted trajectory of the second obstacle, the target real trajectory, and the historical real trajectory.

[0076] In an implementation manner of the embodiment of the application, a first loss function corresponding to the second obstacle is determined according to the difference between the second predicted trajectory of the second obstacle and the target real trajectory, a second loss function corresponding to the second obstacle is determined according to the similarity between the second predicted trajectory of the second obstacle and the historical real trajectory, and a target loss function of the prediction model is determined according to the first loss function and the second loss function corresponding to the second obstacle. In the embodiment of the application, in the process of determining the target loss function of the prediction model, the similarity between the second predicted trajectory of the second obstacle predicted by the prediction model for the third monitoring trajectory of the second obstacle in the second target sampling period and the historical real trajectory is calculated, and the similarity between the trajectories is used as an evaluation index to determine the loss function. That is, the similarity between the current predicted trajectory and the historical real trajectory corresponding to the historical predicted trajectory is used as an evaluation index in the prediction model training process, so as to pay more attention to the stability of the trajectory prediction result output in the continuous trajectory prediction in the model training process, and the training effect of the prediction model is improved.

[0077] Step 604, the prediction model is trained by using the target loss function.

[0078] In the embodiment of the application, the parameters of the prediction model are adjusted according to the target loss function, and based on the adjusted parameters, other training samples are used for continuous training, the parameters of the prediction model are continuously adjusted based on the target loss function determined in each training process, until the loss function of the prediction model is minimized, or the preset training number is reached, or the change of the loss function of the prediction model is less than a set threshold, and then it is determined that the training of the prediction model is completed.

[0079] In the training method of the prediction model in the embodiment of the application, in the process of determining the target loss function of the prediction model, the similarity between the second predicted trajectory predicted by the prediction model for the third monitoring trajectory of the second target sampling period of the second obstacle and the historical real trajectory is calculated, and the similarity between the trajectories is used as an evaluation index to determine the loss function, that is, the similarity between the predicted trajectory currently predicted by the prediction model and the historical real trajectory corresponding to the historical predicted trajectory is used as an evaluation index in the training process of the prediction model, so that the currently predicted predicted trajectory is smoothed based on the historical real trajectory in the model training process, and the training effect of the prediction model is improved.

[0080] Based on the above embodiment, Figure 7 Another flowchart of the training method of the prediction model provided in the embodiment of the application is provided, and how to determine the target loss function of the prediction model in the case where the second obstacles are multiple is described, as shown in Figure 7 Step 603 includes the following steps:

[0081] Step 701, according to the difference between the second predicted trajectory of each second obstacle and the target real trajectory, the first loss function corresponding to each second obstacle is determined.

[0082] In this embodiment, for each second obstacle, the difference between the second predicted trajectory of the second obstacle and the corresponding target real trajectory is determined to determine the first loss function corresponding to the second obstacle.

[0083] Step 702, according to the similarity between the second predicted trajectory of each second obstacle and the historical real trajectory, the second loss function corresponding to each second obstacle is determined.

[0084] In this embodiment, for each second obstacle, the similarity between the second predicted trajectory of the second obstacle and the historical real trajectory is taken as the second loss function corresponding to the second obstacle. Wherein, for the determination of the similarity, as an implementation manner, the second predicted trajectory of the second obstacle can be converted into a first feature vector, the historical real trajectory can be converted into a second feature vector, the Euclidean distance between the first feature vector and the second feature vector is calculated, and the similarity is determined according to the calculated Euclidean distance.

[0085] Wherein, for each second obstacle, the historical real trajectory can be one or more, and the historical sampling period adjacent to the second target sampling period is relatively close in time, and the trajectory of the second obstacle will not change greatly, so that the historical real trajectory corresponding to the historical sampling period adjacent to the second target sampling period in the historical real trajectory is taken as a reference, the first similarity between the second predicted trajectory corresponding to the second obstacle and the historical real trajectory is calculated, the second loss function corresponding to the second obstacle is determined based on the first similarity, and the accuracy of the determination of the second loss function is improved.

[0086] In step 703, the candidate loss function corresponding to each second obstacle is determined according to the first loss function and the second loss function corresponding to each second obstacle.

[0087] In the embodiment of the application, for each second obstacle, the first loss function and the second loss function corresponding to the second obstacle are weighted and summed to determine the candidate loss function corresponding to the second obstacle.

[0088] In step 704, the target loss function of the prediction model is determined according to the candidate loss functions corresponding to the plurality of second obstacles and the set weight values corresponding to each candidate loss function.

[0089] In the embodiment of the application, in the scene where a plurality of second obstacles exist around the vehicle, the target loss function of the prediction model needs to be determined according to the candidate loss functions corresponding to the plurality of second obstacles, and as an implementation manner, the target loss function of the prediction model is determined according to the candidate loss functions corresponding to the plurality of second obstacles and the set weight values corresponding to each candidate loss function.

[0090] Wherein, as an implementation manner, the set weight value corresponding to each candidate loss function is determined according to the number of trajectory points contained in the second target predicted trajectory corresponding to each second obstacle, and the set weight value and the number of trajectory points are in a positive correlation relationship, because the more the trajectory points in the predicted trajectory, the more accurate the prediction is. Therefore, the more the number of trajectory points contained in the second target predicted trajectory corresponding to the second obstacle, the greater the set weight value of the candidate loss function corresponding to the second obstacle, and vice versa.

[0091] In the training method of the prediction model, in the process of model training, the target loss function of the prediction model is determined by weighted summation of the determined candidate loss functions of the plurality of second obstacles and the weight values corresponding to each candidate loss function, thereby improving the accuracy of the determination of the target loss function and the training effect of the prediction model.

[0092] To achieve the above-embodiment, the application also provides a trajectory prediction device.

[0093] Figure 8 A structural schematic diagram of a trajectory prediction device provided by the application is shown.

[0094] As shown in Figure 8 The device can include:

[0095] The first acquisition module 81 is configured to acquire a first monitoring trajectory of a first obstacle around a target vehicle in a first target sampling period.

[0096] The second acquisition module 82 is configured to acquire at least one historical predicted trajectory of the first obstacle; the historical predicted trajectory is obtained by predicting a trajectory of the first obstacle after a second monitoring trajectory according to the second monitoring trajectory of the first obstacle in a historical sampling period before the first target sampling period.

[0097] The fusion module 83 is configured to perform feature fusion on the first monitoring trajectory of the first obstacle and the at least one historical predicted trajectory to determine a fusion trajectory feature of the first obstacle.

[0098] The prediction module 84 is configured to predict a first predicted trajectory of the first obstacle after the first monitoring trajectory according to the fusion trajectory feature of the first obstacle.

[0099] Further, in an implementation manner of the application, the fusion module 83 is specifically configured to:

[0100] input the first monitoring trajectory corresponding to the first obstacle and each historical predicted trajectory into a fusion module of a prediction model for feature extraction, respectively, to extract a first trajectory feature of the first monitoring trajectory and historical predicted trajectory features of each historical predicted trajectory;

[0101] fuse the first trajectory feature and each historical predicted trajectory feature to obtain the fusion trajectory feature of the first obstacle.

[0102] In an implementation form of the embodiment of the application, the prediction module 84 is specifically configured to input the fusion track feature of the first obstacle into a prediction model of the prediction module to obtain a first predicted track of the first obstacle after the first monitoring track.

[0103] In an implementation form of the embodiment of the application, the device further comprises a training module, wherein the training module comprises:

[0104] The acquisition unit is configured to acquire a training sample, wherein the training sample comprises a third monitoring track of a second obstacle in a second target sampling period and at least one historical predicted track of the second obstacle; the training sample is labeled with a target real track of the second obstacle after the third monitoring track and a historical real track of the second obstacle in a historical sampling period before the second target sampling period.

[0105] The prediction unit is configured to input the training sample into a prediction model to obtain a second predicted track of the second obstacle after the third monitoring track.

[0106] The first determination unit is configured to determine a target loss function of the prediction model according to the second predicted track, the target real track and the historical real track of the second obstacle.

[0107] The training unit is configured to train the prediction model by using the target loss function.

[0108] In an implementation form of the embodiment of the application, the first determination unit is specifically configured to:

[0109] determine a first loss function corresponding to the second obstacle according to a difference between the second predicted track and the target real track; determine a second loss function corresponding to the second obstacle according to a similarity between the second predicted track and the historical real track; and determine the target loss function of the prediction model according to the first loss function and the second loss function corresponding to the second obstacle.

[0110] In an implementation form of the embodiment of the application, the second obstacle is a plurality of obstacles, and the first determination unit is specifically further configured to:

[0111] determine a candidate loss function corresponding to each of the second obstacles according to the first loss function and the second loss function corresponding to each of the second obstacles; and determine the target loss function of the prediction model according to the candidate loss functions corresponding to the plurality of second obstacles and a set weight value corresponding to each of the candidate loss functions.

[0112] In an implementation form of the embodiment of the application, the training module further comprises:

[0113] The second determining unit is configured to determine a set weight value corresponding to each second obstacle according to a number of track points contained in a second target prediction track corresponding to each second obstacle, wherein the set weight value and the number of track points are in a positive correlation.

[0114] In an implementation form of the embodiment of the application, the historical real track includes a historical real track corresponding to a previous historical sampling period adjacent to the second target sampling period, and the first determining unit is specifically further configured to:

[0115] determine a first similarity between the second prediction track and a historical real track corresponding to a previous historical sampling period adjacent to the second target sampling period, and determine the second loss function corresponding to the second obstacle according to the first similarity.

[0116] It should be noted that the foregoing explanation and description of the method embodiment are also applicable to the device embodiment, which will not be described here again.

[0117] In the trajectory prediction device of the embodiment of the application, the first monitoring track of the first obstacle around the target vehicle in the first target sampling period is obtained, at least one historical prediction track of the first obstacle is obtained, wherein the historical prediction track is obtained by predicting the track of the first obstacle after the second monitoring track according to the second monitoring track in the historical sampling period before the first target sampling period, the fusion track feature is obtained by performing feature fusion on the first monitoring track and the at least one historical prediction track of the first obstacle, and the first prediction track of the first obstacle after the first monitoring track is predicted according to the fusion track feature. The first prediction track after the first monitoring track is predicted by fusing the track features, which realizes smoothing of the first prediction track based on the historical prediction track, avoids the jumping phenomenon, and improves the accuracy of continuous trajectory prediction.

[0118] In order to implement the above-mentioned embodiments, the application further provides a vehicle, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to the foregoing method embodiments when executing the program.

[0119] In order to implement the above-mentioned embodiments, the application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to the foregoing method embodiments when executing the program.

[0120] In order to implement the above-mentioned embodiments, the application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method according to the foregoing method embodiments.

[0121] To achieve the above-mentioned embodiments, the present application also provides a computer program product, which has a computer program stored thereon, and the computer program is executed by a processor to implement the method according to the foregoing method embodiments.

[0122] Figure 9 A block diagram of an electronic device provided by an embodiment of the present application is shown. For example, the electronic device 800 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0123] Referring to Figure 9 , the electronic device 800 can include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0124] The processing component 802 usually controls overall operations of the electronic device 800, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the methods described above. In addition, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0125] The memory 804 is configured to store various types of data to support operations of the electronic device 800. Examples of these data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0126] The power component 806 provides power to various components of the electronic device 800. The power component 806 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0127] The multimedia component 808 includes a screen to provide an output interface between the electronic device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and intensity of the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a camera mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0128] The audio component 810 is configured to output and / or input an audio signal. For example, the audio component 810 includes a microphone (MIC) to receive an external audio signal when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker to output an audio signal.

[0129] The input / output interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0130] The sensor component 814 includes one or more sensors to provide various state assessments for the electronic device 800. For example, the sensor component 814 can detect an open / closed state of the electronic device 800, relative positioning of components, such as a display and a keypad of the electronic device 800, a change in position of the electronic device 800 or a component of the electronic device 800, presence or absence of user contact with the electronic device 800, an orientation or acceleration / deceleration of the electronic device 800, and a temperature change of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 814 can further include a light sensor such as a CMOS or CCD image sensor for use in an imaging application. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0131] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0132] In an example embodiment, the electronic device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing the above-described methods.

[0133] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the electronic device 800 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0134] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Illustrative expressions of the above terms in the specification do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0135] In addition, the terms "first", "second", etc. are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly specified.

[0136] Any processes or methods described in the flowcharts or elsewhere herein can be understood as representing a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function(s) or process(es). The scope of a preferred embodiment of this application includes any other implementations of the processes or methods described herein, including those that do not use the same order of steps, those that use a different order of steps, those that use additional or fewer steps, those that use one or more different steps, or those that use one or more different structures for carrying out the processes or methods described herein, and the like, as would be understood by those skilled in the art.

[0137] The logic and / or steps represented in the flowcharts or elsewhere herein, for example, can be embodied in computer-readable medium, which can be any device or apparatus that can store the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any tangible apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium, which can include any device or apparatus that can store the program of instructions. The computer-readable medium can be a computer- readable transmission medium, which can include any apparatus that can communicate, propagate, or transport the program of instructions. The computer-readable medium is not, and should not be construed as being a computer-readable storage medium, unless the computer-readable medium is a device or apparatus that can store the program of instructions. The computer-readable medium can be a computer-readable storage medium that can communicate, propagate, or transport the program of instructions. The computer-readable medium can be a computer-readable transmission medium that can store the program of instructions. The computer-readable medium can be a combination of computer-readable storage medium and computer-readable transmission medium, which can communicate, propagate, or transport the program of instructions. Examples (non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a RAM (random access memory), a ROM (read-only memory), an EPROM (erasable programmable ROM), an EEPROM (electrically erasable programmable ROM), and a portable compact disc read-only memory (CD-ROM), all of which are tangible. Note that the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0138] It should be understood that aspects of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As well, if implemented in hardware, as in another embodiment, any of the following can be used: discrete hardware components, an application-specific integrated circuit (ASIC), a programmed digital signal processor (DSP), a field programmable gate array (FPGA) for the implementation of custom logic, and / or combinations thereof.

[0139] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0140] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0141] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A trajectory prediction method, characterized in that: include: Obtain a first monitoring trajectory of a first obstacle around the target vehicle during a first target sampling period; Obtaining at least two historical predicted trajectories of the first obstacle; the historical predicted trajectories are obtained by predicting a trajectory of the first obstacle after the second monitored trajectory based on a second monitored trajectory of the first obstacle during a historical sampling period before the first target sampling period; Inputting the first monitoring trajectory and each segment of the historical predicted trajectory into a fusion module of a prediction model for feature extraction, respectively, to obtain first trajectory features of the first monitoring trajectory and historical predicted trajectory features of each segment of the historical predicted trajectory; and fusing the first trajectory features with the features of each segment of the historical predicted trajectory to determine a fused trajectory feature of the first obstacle; A first predicted trajectory of the first obstacle following the first monitored trajectory is predicted based on the fused trajectory feature of the first obstacle.

2. The method according to claim 1, wherein The predicting, based on the fused trajectory feature of the first obstacle, a first predicted trajectory of the first obstacle following the first monitored trajectory includes: The fused trajectory feature of the first obstacle is input into a prediction module of a prediction model to predict a first predicted trajectory of the first obstacle following the first monitored trajectory.

3. The method according to claim 2, wherein The prediction model is trained using the following method: Acquire a training sample, the training sample comprising a third monitored trajectory of a second obstacle during a second target sampling period and at least one historical predicted trajectory of the second obstacle; the training sample is annotated with a target true trajectory of the second obstacle after the third monitored trajectory and a historical true trajectory of the second obstacle during a historical sampling period before the second target sampling period; Inputting the training sample into a prediction model to predict a second predicted trajectory of the second obstacle following the third monitoring trajectory; determining a target loss function of the prediction model according to the second predicted trajectory of the second obstacle, the target true trajectory, and the historical true trajectory; The prediction model is trained using the target loss function.

4. The method according to claim 3, wherein The determining, according to the second predicted trajectory of the second obstacle, the target actual trajectory, and the historical actual trajectory, of the target loss function of the prediction model includes: determining a first loss function corresponding to the second obstacle based on a difference between the second predicted trajectory and the actual trajectory of the target; determining a second loss function corresponding to the second obstacle based on a similarity between the second predicted trajectory and the historical true trajectory; Determine a target loss function of the prediction model according to the first loss function and the second loss function corresponding to the second obstacle.

5. The method according to claim 4, wherein There are multiple second obstacles, and determining the target loss function of the prediction model according to the first loss function and the second loss function corresponding to the second obstacles includes: Determining a candidate loss function corresponding to each second obstacle according to the first loss function and the second loss function corresponding to each second obstacle; The target loss function of the prediction model is determined according to the candidate loss functions corresponding to the plurality of second obstacles and the set weight values ​​corresponding to the respective candidate loss functions.

6. The method according to claim 5, wherein The method further comprises: The set weight value corresponding to each second obstacle is determined according to the number of trajectory points included in the second target predicted trajectory corresponding to each second obstacle; wherein the set weight value is positively correlated with the number of trajectory points.

7. The method according to claim 4, wherein The historical real trajectory includes a historical real trajectory corresponding to a previous historical sampling period adjacent to the second target sampling period, and determining a second loss function corresponding to the second obstacle based on a similarity between the second predicted trajectory and the historical real trajectory includes: Determining a first similarity between the second predicted trajectory and a historical real trajectory corresponding to a previous historical sampling period adjacent to the second target sampling period; Determine the second loss function corresponding to the second obstacle according to the first similarity.

8. A trajectory prediction device, characterized in that: include: A first acquisition module is used to obtain a first monitoring trajectory of a first obstacle around the target vehicle during a first target sampling period; a second acquisition module configured to acquire at least two historical predicted trajectories of the first obstacle; the historical predicted trajectories being predicted based on a second monitored trajectory of the first obstacle during a historical sampling period before the first target sampling period, and a trajectory of the first obstacle after the second monitored trajectory; a fusion module, configured to input the first monitoring trajectory and each segment of the historical predicted trajectory into a fusion module of a prediction model for feature extraction, respectively, to obtain first trajectory features of the first monitoring trajectory and historical predicted trajectory features of each segment of the historical predicted trajectory, and to perform feature fusion on the first trajectory features and the historical predicted trajectory features of each segment to determine a fused trajectory feature of the first obstacle; A prediction module is configured to predict a first predicted trajectory of the first obstacle following the first monitored trajectory based on a fused trajectory feature of the first obstacle.

9. A vehicle, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images