Trajectory prediction model training method, trajectory prediction method, device and medium

Abstract

A trajectory prediction model training method, a trajectory prediction method, an apparatus and a medium are disclosed. The method comprises: obtaining first extracted features corresponding to a first image sequence; applying random disturbances to the first extracted features via a trajectory prediction model to be trained to obtain first disturbance features corresponding to the random disturbances respectively; generating, via the trajectory prediction model to be trained, multi-modal trajectory prediction results of obstacles in the first image sequence based on the first disturbance features corresponding to the random disturbances respectively; training the trajectory prediction model to be trained based on a second image sequence and the multi-modal trajectory prediction results; and determining the trained trajectory prediction model to be trained as a target trajectory prediction model in response to the trained trajectory prediction model to be trained satisfying a preset training end condition. The disclosed embodiments can directly perform trajectory prediction under the camera visual angle, thus eliminating the need to restore depth information and improving the accuracy of the trajectory prediction results.

Description

Technical Field

[0001] This disclosure relates to machine vision technology, and in particular to a training method, trajectory prediction method, apparatus and medium for a trajectory prediction model. Background Technology

[0002] Autonomous driving technology is being used more and more widely in vehicles and other mobile devices. Trajectory prediction is a crucial part of autonomous driving technology, and the results of trajectory prediction have a significant impact on vehicle safety and comfort.

[0003] It should be noted that traditional trajectory prediction schemes are based on bird's-eye view (BEV) perspective. This scheme requires the reconstruction of depth information, and the results of depth information reconstruction are often inaccurate, which reduces the accuracy of trajectory prediction results. Summary of the Invention

[0004] To address the problem of low accuracy in trajectory prediction results obtained using traditional trajectory prediction schemes, this disclosure is proposed. Embodiments of this disclosure provide a training method for a trajectory prediction model, a trajectory prediction method, an apparatus, and a medium.

[0005] According to one aspect of the present disclosure, a method for training a trajectory prediction model is provided, comprising:

[0006] Obtain a first extracted feature corresponding to a first image sequence, wherein the first image sequence includes: images captured by a first camera set on a first mobile device at multiple historical moments;

[0007] By applying random perturbations to the first extracted features through the trajectory prediction model to be trained, multiple first perturbation features corresponding to each random perturbation are obtained;

[0008] Based on the first perturbation features corresponding to each of the multiple random perturbations, the multimodal trajectory prediction results of obstacles in the first image sequence are generated through the trajectory prediction model to be trained.

[0009] Based on the second image sequence and the multimodal trajectory prediction results, the trajectory prediction model to be trained is trained. The second image sequence includes images captured by the first camera at multiple future times after the multiple historical times.

[0010] In response to the training trajectory prediction model meeting the preset training termination condition, the training trajectory prediction model is determined as the target trajectory prediction model.

[0011] According to another aspect of the present disclosure, a trajectory prediction method is provided, comprising:

[0012] Obtain the third extracted feature corresponding to the third image sequence, wherein the third image sequence includes: images captured by a second camera set on a second mobile device at multiple times;

[0013] By applying random perturbations to the third extracted features through the target trajectory prediction model, a second perturbation feature corresponding to each of the multiple random perturbations is obtained;

[0014] Based on the second perturbation features corresponding to each of the multiple random perturbations, the multimodal trajectory prediction results of the obstacles in the third image sequence are generated via the target trajectory prediction model.

[0015] According to another aspect of the present disclosure, a training apparatus for a trajectory prediction model is provided, comprising:

[0016] The first acquisition module is used to acquire the first extractable feature corresponding to the first image sequence, wherein the first image sequence includes: images acquired by a first camera set on a first mobile device at multiple historical moments;

[0017] The first perturbation application module is used to apply random perturbations to the first extracted features obtained by the first acquisition module via the trajectory prediction model to be trained, so as to obtain the first perturbation features corresponding to each of the multiple random perturbations.

[0018] The first generation module is used to generate multimodal trajectory prediction results of obstacles in the first image sequence based on the first perturbation features corresponding to the plurality of random perturbations obtained by the first perturbation application module and via the trajectory prediction model to be trained.

[0019] The training module is used to train the trajectory prediction model to be trained based on the second image sequence and the multimodal trajectory prediction results generated by the first generation module. The second image sequence includes images captured by the first camera at multiple future times after the multiple historical times.

[0020] The determination module is used to determine the trained trajectory prediction model as the target trajectory prediction model in response to the training module meeting the preset training termination condition.

[0021] According to another aspect of the present disclosure, a trajectory prediction apparatus is provided, comprising:

[0022] The second acquisition module is used to acquire the third extracted feature corresponding to the third image sequence, wherein the third image sequence includes: images captured by the second camera set on the second mobile device at multiple times;

[0023] The second perturbation application module is used to apply random perturbations to the third extracted features obtained by the second acquisition module via the target trajectory prediction model, thereby obtaining a second perturbation feature corresponding to each of the multiple random perturbations.

[0024] The second generation module is used to generate multimodal trajectory prediction results of obstacles in the third image sequence based on the second perturbation features corresponding to the plurality of random perturbations obtained by the second perturbation application module, via the target trajectory prediction model.

[0025] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for executing the training method of the trajectory prediction model or the trajectory prediction method described above.

[0026] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0027] processor;

[0028] Memory used to store the processor's executable instructions;

[0029] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the training method of the trajectory prediction model or the trajectory prediction method described above.

[0030] Based on the trajectory prediction model training method, trajectory prediction method, apparatus, medium, and electronic device provided in the above embodiments of this disclosure, by providing the first extracted features corresponding to the first image sequence as input data to the trajectory prediction model to be trained, the trajectory prediction model to be trained can obtain multiple corresponding first perturbation features by applying random perturbations, and further perform trajectory prediction based on the multiple first perturbation features to obtain the multimodal trajectory prediction result of the obstacle in the first image sequence. The multimodal trajectory prediction result can be used together with the second image sequence to train the trajectory prediction model to be trained, thereby obtaining the trained target trajectory prediction model. In this way, the trained target trajectory prediction model can efficiently and reliably output the multimodal trajectory prediction result of the obstacle based on the third extracted features corresponding to the input third image sequence. That is, the trajectory prediction scheme provided in the embodiments of this disclosure performs trajectory prediction directly from the camera's perspective, rather than from the BEV perspective, so there is no need to undergo depth information restoration, which is beneficial to improving the accuracy of the trajectory prediction result. For example, using the trajectory prediction scheme provided in the embodiments of this disclosure, accurate trajectory prediction can also be performed for small obstacles at a distance (such as pedestrians).

[0031] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0032] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0033] Figure 1 This is a flowchart illustrating a training method for a trajectory prediction model provided in an exemplary embodiment of this disclosure.

[0034] Figure 2 This is a flowchart illustrating a training method for a trajectory prediction model provided in another exemplary embodiment of this disclosure.

[0035] Figure 3 This is a flowchart illustrating a training method for a trajectory prediction model provided in another exemplary embodiment of this disclosure.

[0036] Figure 4 This is a flowchart illustrating a training method for a trajectory prediction model provided in yet another exemplary embodiment of this disclosure.

[0037] Figure 5 This is a flowchart illustrating a training method for a trajectory prediction model provided in yet another exemplary embodiment of this disclosure.

[0038] Figure 6 This is a flowchart illustrating a training method for a trajectory prediction model provided in yet another exemplary embodiment of this disclosure.

[0039] Figure 7 This is a schematic flowchart of a trajectory prediction method provided in an exemplary embodiment of this disclosure.

[0040] Figure 8 This is a schematic diagram illustrating the implementation principle of trajectory prediction in an exemplary embodiment of this disclosure.

[0041] Figure 9 This is a schematic diagram of the structure of a training device for a trajectory prediction model provided in an exemplary embodiment of this disclosure.

[0042] Figure 10 This is a schematic diagram of the structure of a training device for a trajectory prediction model provided in another exemplary embodiment of this disclosure.

[0043] Figure 11 This is a schematic diagram of the structure of a training device for a trajectory prediction model provided in another exemplary embodiment of this disclosure.

[0044] Figure 12This is a schematic diagram of the structure of a training device for a trajectory prediction model provided in yet another exemplary embodiment of this disclosure.

[0045] Figure 13 This is a schematic diagram of the structure of a training device for a trajectory prediction model provided in yet another exemplary embodiment of this disclosure.

[0046] Figure 14 This is a schematic diagram of the trajectory prediction device provided in an exemplary embodiment of the present disclosure.

[0047] Figure 15 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0048] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0049] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0050] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent specific technical meanings or necessary logical sequences.

[0051] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0052] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0053] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0054] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0055] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0056] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0057] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0058] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0059] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0060] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0061] Application Overview

[0062] Trajectory prediction is a crucial component of autonomous driving technology, and the results of trajectory prediction can significantly impact vehicle safety and comfort.

[0063] It should be noted that traditional trajectory prediction schemes are based on the BEV perspective. This scheme uses detection boxes derived from the surround view image information. However, converting the surround view image information to the BEV perspective requires the restoration of depth information. Depth information is discarded during the image capture stage, and the restoration results are usually inaccurate. Therefore, the accuracy of the trajectory prediction results obtained by this scheme is difficult to guarantee.

[0064] Exemplary methods

[0065] Figure 1 This is a flowchart illustrating a training method for a trajectory prediction model provided in an exemplary embodiment of this disclosure. Figure 1 The method shown may include steps 110, 120, 130, 140 and 150, each of which will be explained below.

[0066] Step 110: Obtain the first extracted feature corresponding to the first image sequence. The first image sequence includes images captured by the first camera set on the first mobile device at multiple historical moments.

[0067] Optionally, the first mobile device can be a vehicle, train, ship, or aircraft, etc. For ease of understanding, the embodiments of this disclosure are all described using the case of a vehicle as the first mobile device.

[0068] Optionally, the first camera can be a front-view camera, left-view camera, right-view camera, etc. of the first mobile device; the first camera can periodically capture images and store the captured images in a storage area.

[0069] In step 110, multiple images (e.g., N images) can be retrieved from the storage area. The acquisition time corresponding to each of the N images can be considered a historical moment. By arranging the N images in chronological order of their acquisition times, a first image sequence comprising the N images can be formed. Feature extraction is then performed on the first image sequence to obtain a first extracted feature. This first extracted feature can be a multi-dimensional feature. Here, the first extracted feature can be a 128-dimensional feature or a 256-dimensional feature.

[0070] Step 120: Apply random perturbations to the first extracted features through the trajectory prediction model to be trained to obtain the first perturbation features corresponding to each of the multiple random perturbations.

[0071] It should be noted that the trajectory prediction model to be trained can be an obstacle trajectory prediction model to be trained, which is used to predict the trajectory of obstacles.

[0072] In step 120, the first extracted features can be provided to the trajectory prediction model to be trained. The trajectory prediction model to be trained can determine multiple random perturbations (e.g., M random perturbations) for the first extracted features. For each of the M random perturbations, the random perturbation can be applied to the first extracted features to obtain the first perturbation feature corresponding to the random perturbation. Thus, M first perturbation features corresponding one-to-one with the M random perturbations can be obtained. The dimension of each first perturbation feature can be the same as that of the first extracted features.

[0073] Step 130: Based on the first perturbation features corresponding to each of the multiple random perturbations, the multimodal trajectory prediction results of the obstacles in the first image sequence are generated through the trajectory prediction model to be trained.

[0074] Optionally, the trajectory prediction model to be trained may include a trajectory predictor, such as a Long Short-Term Memory (LSTM) trajectory predictor.

[0075] In step 130, the M first perturbation features obtained in step 120 can be provided to the trajectory predictor, which can then perform calculations to generate multimodal trajectory prediction results.

[0076] It should be noted that, in the embodiments of this disclosure, multimodal refers to outputting multiple different prediction results for the same obstacle requiring trajectory prediction, in order to cover more possibilities. Optionally, the multimodal trajectory prediction results may include M predicted trajectories for the same obstacle, and the M predicted trajectories may correspond one-to-one with M first perturbation features.

[0077] Step 140: Based on the second image sequence and the multimodal trajectory prediction results, train the trajectory prediction model to be trained. The second image sequence includes images acquired by the first camera at multiple future moments after multiple historical moments.

[0078] In step 140, K images can be obtained from the storage area that were acquired later than the N images mentioned above and are adjacent to the N images mentioned above. The acquisition time corresponding to each of the K images can be regarded as a future time. By arranging the K images in the order of their acquisition times, a second image sequence including the K images can be formed.

[0079] It should be noted that since the K images were acquired later than the N images mentioned above and are adjacent to the N images mentioned above, the second image sequence including the K images can be considered to carry the future true trajectory of the obstacle. The second image sequence can also be called the future frame true sequence.

[0080] By combining the future real trajectory carried in the second image sequence with the multimodal trajectory prediction results obtained by the trajectory prediction model to be trained through trajectory prediction, the prediction accuracy of the trajectory prediction model to be trained can be evaluated. Based on the evaluation results, the parameters of the trajectory prediction model to be trained can be updated to improve the prediction accuracy of the trajectory prediction model to be trained, thereby realizing the training of the trajectory prediction model to be trained.

[0081] Step 150: In response to the trained trajectory prediction model meeting the preset training termination condition, the trained trajectory prediction model is determined as the target trajectory prediction model.

[0082] It should be noted that when training the trajectory prediction model to be trained, a large amount of sample data can be used. Each sample data includes a first image sequence and a corresponding second image sequence. In this way, for each sample data, steps 110 to 140 above can be executed. The process of executing steps 110 to 140 above for each sample data can be regarded as an iterative process.

[0083] After several iterations, if the trained trajectory prediction model converges at a certain point, it can be determined that the trained trajectory prediction model meets the preset training termination condition. At this time, the trained trajectory prediction model can be directly identified as the target trajectory prediction model.

[0084] Of course, the preset training termination condition is not limited to this. For example, it can also be determined that the trained trajectory prediction model meets the preset training termination condition when the number of iterations reaches a preset number.

[0085] The training method for the trajectory prediction model provided in the above embodiments of this disclosure involves providing the first extracted features corresponding to the first image sequence as input data to the trajectory prediction model to be trained. The trajectory prediction model to be trained can obtain multiple first perturbation features by applying random perturbations, and further perform trajectory prediction based on the multiple first perturbation features to obtain the multimodal trajectory prediction result of the obstacle in the first image sequence. The multimodal trajectory prediction result can be used together with the second image sequence to train the trajectory prediction model to be trained, thereby obtaining the trained target trajectory prediction model. In this way, the trained target trajectory prediction model can efficiently and reliably output the multimodal trajectory prediction result of the obstacle based on the third extracted features corresponding to the input third image sequence. That is, the trajectory prediction scheme provided in the embodiments of this disclosure performs trajectory prediction directly from the camera's perspective, rather than from the BEV perspective. Therefore, it does not require the restoration of depth information, which is beneficial to improving the accuracy of the trajectory prediction result. For example, using the trajectory prediction scheme provided in the embodiments of this disclosure, accurate trajectory prediction can also be performed for small obstacles at a distance (such as pedestrians).

[0086] exist Figure 1 Based on the illustrated embodiments, as Figure 2 As shown, step 120 may include steps 1201 and 1203.

[0087] Step 1201: Determine the first feature perturbation distribution parameters corresponding to the first extracted feature via the Conditional Variational Autoencoder (CVAE) in the trajectory prediction model to be trained.

[0088] The trajectory prediction model to be trained may include a conditional variational autoencoder. In step 1201, the first extracted features can be provided to the conditional variational autoencoder, which can then perform calculations to generate the first feature perturbation distribution parameters corresponding to the first extracted features.

[0089] Assuming the first extracted feature is a 128-dimensional feature, the first extracted feature includes 128 feature values ​​corresponding one-to-one with the 128 dimensions. The first feature perturbation distribution parameters may include: 128 means corresponding one-to-one with the 128 dimensions, and 128×128 covariances. Each of the 128 means is used to represent the perturbation of one dimension, and each of the 128×128 covariances is used to represent the degree of correlation between two dimensions.

[0090] Step 1203: Based on the first feature perturbation distribution parameters, random perturbations are applied to the first extracted features via a conditional variational autoencoder to obtain the first perturbation features corresponding to each of the multiple random perturbations.

[0091] Continuing with the example above, the first feature perturbation distribution parameters include: 128 means corresponding one-to-one with the 128 dimensions, and 128×128 covariances. The first feature perturbation distribution parameters can be used to represent a high-dimensional Gaussian distribution. By randomly sampling from this high-dimensional Gaussian distribution, M random perturbations can be obtained, each of which includes 128 perturbation values ​​corresponding one-to-one with the 128 dimensions.

[0092] Assuming the 128 feature values ​​in the first extracted features are represented as x1, x2, x3, ..., x128, and the 128 perturbation values ​​corresponding to a certain random perturbation are represented as a1, a2, a3, ..., a128, then x1, x2, x3, ..., x128 can be normalized to obtain 128 normalized feature values ​​x1′, x2′, x3′, ..., x128′. For x1′ among the 128 normalized feature values, x1′ can be multiplied by a first preset coefficient, and the result of the multiplication can be added to or subtracted from a1. The result of the addition or subtraction can be used as the first feature value in the first perturbation feature corresponding to the random perturbation. In a similar way, 128 feature values ​​can be determined, thus obtaining the complete first perturbation feature corresponding to the random perturbation.

[0093] As described above, the process of obtaining the first perturbation feature from the first extracted feature and the random perturbation uses one of the addition and subtraction operations, multiplication, and normalization operations. In specific implementation, the process can also use exponential, logarithmic, division and other operational logics. The embodiments of this disclosure do not limit this.

[0094] Step 140 includes step 1401.

[0095] Step 1401: Train the trajectory prediction model to be trained based on the second image sequence, the multimodal trajectory prediction results, and the first feature perturbation distribution parameters.

[0096] In one alternative implementation, such as Figure 3 As shown, step 1401 includes steps 14011, 14013, 14015 and 14017.

[0097] Step 14011: Obtain the second extracted feature corresponding to the second image sequence.

[0098] In step 14011, feature extraction can be performed on the second image sequence to obtain the second extracted features corresponding to the second image sequence.

[0099] Optionally, the method for feature extraction of the second image sequence can be the same as or different from the method for feature extraction of the first image sequence. For example, a simple feature extractor (such as a simple convolutional neural network) can be used to extract features from the second image sequence, while a more complex feature extractor can be used to extract features from the first image sequence.

[0100] Step 14013: Determine the second feature perturbation distribution parameters corresponding to the second extracted feature via a conditional variational autoencoder.

[0101] It should be noted that the specific implementation of step 14013 can be referred to the description of the implementation of step 1201 above, and will not be repeated here.

[0102] Step 14015: Determine the first model loss value by comparing the first feature perturbation distribution parameter and the second feature perturbation distribution parameter.

[0103] It should be noted that the first feature perturbation distribution parameter and the second feature perturbation distribution parameter can be regarded as two probability distributions. The similarity between different probability distributions can usually be measured by KL divergence (Kullback-Leibler divergence). Therefore, in step 14015, KL divergence can be calculated for the first feature perturbation distribution parameter and the second feature perturbation distribution parameter. Then, the calculated KL divergence can be directly used as the first model loss value, or the calculated KL divergence can be multiplied by the second preset coefficient, and the result of the multiplication can be used as the first model loss value.

[0104] Step 14017: Train the trajectory prediction model to be trained based on the first model loss value.

[0105] In step 14017, referring to the first model loss value, gradient descent (e.g., stochastic gradient descent, steepest gradient descent, etc.) can be used to update the parameters of the trajectory prediction model to be trained, thereby achieving the training of the trajectory prediction model to be trained.

[0106] In this implementation, by sequentially acquiring the second extracted feature and the second feature perturbation distribution parameter, the comparison result of the first feature perturbation distribution parameter and the second feature perturbation distribution parameter can be used to determine the first model loss value, so as to train the trajectory prediction model to be trained. In this way, through training, the first feature perturbation distribution parameter and the second feature perturbation distribution parameter can be made sufficiently close, which helps to ensure the prediction accuracy of the trained target trajectory prediction model.

[0107] It should be noted that the implementation of step 1401 is not limited to this. For example, when training the trajectory prediction model to be trained, not only can the first model loss value obtained based on the second image sequence, the multimodal trajectory prediction result and the first feature perturbation distribution parameter be referenced, but other loss values ​​can also be referenced, such as the second model loss value mentioned below.

[0108] In the embodiments of this disclosure, the first feature perturbation distribution parameter corresponding to the first extracted feature can be determined efficiently and reliably via the conditional variational autoencoder in the trajectory prediction model to be trained. The first feature perturbation distribution parameter can be used together with the second image sequence and the multimodal trajectory prediction result for training the trajectory prediction model to be trained. This is beneficial for the conditional variational autoencoder to generate appropriate perturbation features based on the extracted features, thereby helping to ensure the prediction accuracy of the trained target trajectory prediction model.

[0109] In an optional example, the multimodal trajectory prediction result includes: multiple predicted trajectories of the obstacle, and the prediction probability value corresponding to each of the multiple predicted trajectories. Here, the multimodal trajectory prediction result may include: M predicted trajectories of the obstacle, and M prediction probability values ​​corresponding one-to-one with the M predicted trajectories, whereby the prediction probability value corresponding to any predicted trajectory is used to represent the confidence level of the predicted trajectory.

[0110] exist Figure 1 Based on the illustrated embodiments, as Figure 4 As shown, step 140 includes steps 1403, 1405, 1407, 1409 and 1411.

[0111] Step 1403: Determine the future true trajectory of the obstacle based on the second image sequence.

[0112] As described above, the second image sequence contains the future true trajectory of the obstacle. Therefore, in step 1403, the future true trajectory of the obstacle can be directly extracted from the second image sequence.

[0113] Step 1405: Determine the loss value of the first trajectory based on the future real trajectory and multiple predicted trajectories.

[0114] In one alternative implementation, step 1405 includes:

[0115] Based on the future actual trajectory and multiple predicted trajectories, the loss value of the first trajectory is determined, including:

[0116] The similarity between multiple predicted trajectories and future actual trajectories is determined separately;

[0117] Based on the similarity of each of the multiple predicted trajectories, weights are determined for each of the multiple predicted trajectories;

[0118] Based on the similarity of each of the multiple predicted trajectories, the loss value of the second trajectory is determined for each of the multiple predicted trajectories;

[0119] By using the weights corresponding to each of the multiple predicted trajectories, the loss values ​​of the second trajectory corresponding to each of the multiple predicted trajectories are weighted to obtain the loss value of the first trajectory.

[0120] Optionally, for each of the M predicted trajectories, the similarity between the predicted trajectory and the future real trajectory can be determined to obtain M similarity scores corresponding one-to-one with the M predicted trajectories.

[0121] Next, the M similarities can be normalized. The normalized result of the similarity corresponding to any predicted trajectory can be used as the weight corresponding to that predicted trajectory. In this way, the M weights corresponding one-to-one with the M predicted trajectories can be obtained. Of course, other methods can also be used to determine the M weights, as long as the weights are positively correlated with the similarity.

[0122] In addition, for each of the M predicted trajectories, a second trajectory loss value can be determined based on the similarity corresponding to the predicted trajectory. For example, the similarity corresponding to the predicted trajectory can be directly used as the second trajectory loss value, or the result of multiplying the similarity corresponding to the predicted trajectory with a third preset coefficient can be used as the second trajectory loss value. In this way, M second trajectory loss values ​​corresponding one-to-one with the M predicted trajectories can be obtained.

[0123] Then, the first trajectory loss value can be obtained by using M weights to perform a weighted sum or weighted average of the M second trajectory loss values.

[0124] In this implementation, by determining the similarity between each predicted trajectory and the future true trajectory, and based on the obtained similarity, weights and a second trajectory loss value can be determined for each predicted trajectory efficiently and reliably. The obtained weights are then used for weighting the second trajectory loss value, thereby obtaining a first trajectory loss value efficiently and reliably. The first trajectory loss value can effectively characterize the accuracy of the trajectory prediction model to be trained in predicting future trajectories.

[0125] Of course, the implementation of step 1405 is not limited to this. For example, in step 1405, M similarities can be calculated, and M second trajectory loss values ​​can be calculated based on the M similarities. Then, the summation or average of the M second trajectory loss values ​​can be directly used as the first trajectory loss value.

[0126] Step 1407: Determine the probability loss value based on the future real trajectory, multiple predicted trajectories, and the predicted probability values ​​corresponding to each of the multiple predicted trajectories.

[0127] In one alternative implementation, step 1407 includes:

[0128] The similarity between multiple predicted trajectories and future actual trajectories is determined separately;

[0129] Based on the similarity of each of the multiple predicted trajectories, a reference probability value is determined for each of the multiple predicted trajectories.

[0130] The probability loss value is determined based on the predicted probability value and reference probability value corresponding to each of the multiple predicted trajectories.

[0131] Optionally, for each of the M predicted trajectories, the similarity between the predicted trajectory and the future real trajectory can be determined to obtain M similarity scores corresponding one-to-one with the M predicted trajectories.

[0132] Next, for each of the multiple predicted trajectories, a reference probability value can be determined based on the similarity corresponding to the predicted trajectory. For example, the similarity can be directly used as the reference probability value corresponding to the predicted trajectory, or the result of multiplying the similarity with the fourth preset coefficient can be used as the reference probability value corresponding to the predicted trajectory. In this way, M reference probability values ​​corresponding one-to-one with M predicted trajectories are obtained, and the M reference probability values ​​can form a first probability value sequence.

[0133] In addition, the M predicted probability values ​​corresponding one-to-one with the M predicted trajectories can form a second probability value sequence. In this way, the KL divergence can be calculated for the first probability value sequence and the second probability value sequence. Then, the calculated KL divergence can be directly used as the probability loss value, or the calculated KL divergence can be multiplied by the fifth preset coefficient and the result of the multiplication can be used as the probability loss value.

[0134] In this implementation, by determining the similarity between each predicted trajectory and the future true trajectory, a reference probability value can be efficiently and reliably determined for each predicted trajectory based on the obtained similarity. This allows the reference probability value and the predicted probability value corresponding to each predicted trajectory to be used together to determine the probability loss value. The probability loss value can effectively characterize the prediction accuracy of the trajectory prediction model under training in predicting the probability of the future trajectory.

[0135] Of course, the implementation of step 1407 is not limited to this. For example, after obtaining M reference probability values ​​that correspond one-to-one with the M predicted trajectories, the difference between the reference probability value and the predicted probability value corresponding to each of the M predicted trajectories can be calculated to obtain M differences that correspond one-to-one with the M predicted trajectories. The M differences can then be summed or averaged to obtain the probability loss value.

[0136] Step 1409: Determine the second model loss value based on the first trajectory loss value and the probability loss value.

[0137] In step 1409, the first trajectory loss value and the probability loss value can be directly added together and the result can be used as the second model loss value. Alternatively, the average value of the first trajectory loss value and the probability loss value can be calculated and the average value can be used as the second model loss value. Or, the first trajectory loss value and the probability loss value can be weighted and the weighted result can be used as the second model loss value.

[0138] Step 1411: Train the trajectory prediction model to be trained based on the second model loss value.

[0139] In step 1411, referring to the second model loss value, gradient descent (e.g., stochastic gradient descent, steepest gradient descent, etc.) can be used to update the parameters of the trajectory prediction model to be trained, thereby achieving the training of the trajectory prediction model to be trained.

[0140] In the embodiments of this disclosure, referring to the second image sequence, the future true trajectory of the obstacle can be determined efficiently and reliably. The future true trajectory can be used together with multiple predicted trajectories in the multimodal trajectory prediction results to determine the first trajectory loss value. The future true trajectory can also be used together with multiple predicted trajectories and the prediction probability values ​​corresponding to each of the multiple predicted trajectories to determine the probability loss value. Combining the first trajectory loss value and the probability loss value, the second model loss value can be determined efficiently and reliably. The second model loss value can effectively reflect the prediction accuracy of the trajectory prediction model to be trained for the future trajectory and the prediction accuracy of the prediction probability corresponding to the future trajectory. By using the second model loss value to train the trajectory prediction model to be trained, the prediction accuracy of the target trajectory prediction model obtained by the final training for the future trajectory and the prediction probability corresponding to the future trajectory can be improved through backpropagation.

[0141] exist Figure 1 Based on the illustrated embodiments, as Figure 5 As shown, step 110 includes steps 1101 and 1103.

[0142] Step 1101: Superimpose multiple images from the first image sequence along the channel direction to obtain a superimposed image.

[0143] In step 1101, the N images in the first image sequence can be stacked along the channel direction according to the order of acquisition time to obtain a superimposed image. Assuming that the size of each of the N images is represented as H×W×C, the size of the superimposed image can be represented as H×W×(N×C).

[0144] Step 1103: Extract features from the superimposed images to obtain the first extracted features corresponding to the first image sequence.

[0145] In step 1103, a temporal convolutional network (TCN) can be used to extract features from the superimposed images in the time dimension to obtain the first extracted features corresponding to the first image sequence.

[0146] In the embodiments of this disclosure, combining image stacking operations and feature extraction operations on the stacking results enables efficient and reliable acquisition of the first extracted features, thereby improving the efficiency of obtaining the first extracted features.

[0147] exist Figure 1 Based on the illustrated embodiments, as Figure 6 As shown, step 110 includes steps 1105 and 1107.

[0148] Step 1105: Obtain obstacle information corresponding to each of the multiple images in the first image sequence to obtain multiple obstacle information. Each obstacle information includes: obstacle bounding box information or obstacle segmentation result information.

[0149] In step 1105, obstacle information corresponding to each of the N images in the first image sequence can be obtained, thus obtaining N obstacle information corresponding one-to-one with the N images. The obstacle information corresponding to any image may include: obstacle bounding box information obtained by obstacle detection of the image; or obstacle segmentation result information obtained by obstacle segmentation of the image. The obstacle bounding box information corresponding to any image may include: the coordinates of the four angles of the obstacle's bounding box; or; the coordinates of the top-left corner of the obstacle's bounding box, as well as the width and height of the bounding box. The obstacle segmentation result corresponding to any image may include: information used to distinguish between obstacles and foreground in the image; that is, based on the obstacle segmentation result corresponding to the image, it can be determined which pixels in the image correspond to obstacles and which pixels correspond to foreground.

[0150] Step 1107: Extract features from the obstacle information sequence composed of multiple obstacle information to obtain the first extracted features corresponding to the first image sequence.

[0151] In one alternative implementation, step 1107 includes:

[0152] Each obstacle in the obstacle information sequence is used as the current obstacle information in turn;

[0153] In response to the fact that the current obstacle information is the first obstacle information in the obstacle information sequence, based on the current obstacle information, the feature extractor in the trajectory prediction model to be trained generates the extracted features corresponding to the current obstacle information;

[0154] In response to the fact that the current obstacle information is not the first obstacle information in the obstacle information sequence, the extracted features corresponding to the previous obstacle information and the current obstacle information are generated by the feature extractor.

[0155] In response to the fact that the current obstacle information is the last obstacle information in the obstacle information sequence, the extracted feature corresponding to the current obstacle information is determined as the first extracted feature corresponding to the first image sequence.

[0156] Optionally, the feature extractor in the trajectory prediction model to be trained can be an LSTM feature extractor.

[0157] In one example, if N is 5, the obstacle information sequence includes 5 obstacles: obstacle 1, obstacle 2, obstacle 3, obstacle 4, and obstacle 5. First, obstacle 1 is used as the current obstacle and fed as the first input to the LSTM feature extractor. The LSTM feature extractor performs calculations based on obstacle 1 to generate and record the corresponding extracted feature 1. Next, obstacle 2 is used as the current obstacle and fed as the second input to the LSTM feature extractor. The LSTM feature extractor performs calculations based on obstacle 2 and extracted feature 1 to generate and record the corresponding extracted feature 2. Finally, obstacle 3 is used as the current obstacle and fed as the third input to the LSTM feature extractor. The LSTM feature extractor performs calculations based on obstacle 3 and extracted feature 2 to generate and record the corresponding extracted feature 3. This process continues in the same manner. After the obstacle information 5 is provided as the fifth input to the LSTM feature extractor, the LSTM feature extractor can generate the corresponding extracted feature 5 through calculation. The extracted feature 5 can then be used as the first extracted feature corresponding to the first image sequence.

[0158] In this implementation, a feature extractor can be used to iteratively extract features from N obstacle information. Each subsequent feature extraction can utilize the extraction results obtained from the previous feature extraction. This time recursion method can ensure the accuracy and reliability of the final determined first extracted feature.

[0159] It should be noted that in step 1107, in addition to using the LSTM feature extractor, a gated recurrent neural network (GRN) can also be used, or other types of recurrent neural networks besides the LSTM feature extractor and the GRN can be used to obtain the first extracted features corresponding to the first image sequence.

[0160] In the embodiments of this disclosure, by combining the acquisition operation of obstacle information corresponding to each of the multiple images in the first image sequence, and the feature extraction operation of the obstacle information sequence composed of the acquired obstacle information, the first extracted feature can be obtained efficiently and reliably. The first extracted feature can effectively reflect the information carried in the first image sequence.

[0161] In one optional example,

[0162] The multimodal trajectory prediction results include: multiple predicted trajectories of the obstacle;

[0163] Each of the multiple predicted trajectories is represented using target data, which includes: each of the multiple future moments, and the coordinates of the four corner points of the obstacle's bounding box.

[0164] It should be noted that each obstacle information mentioned above can be represented by obstacle bounding box information. Obstacle bounding box information can include the coordinates of the four angles of the obstacle's bounding box. Thus, in the multimodal trajectory prediction result, the target data used to represent any one of the M predicted trajectories can include the coordinates of the four corner points of the obstacle's bounding box at each of the K future time points. In this way, the target data can represent not only the position of the obstacle's bounding box but also its size. Therefore, using the finally trained target trajectory prediction model, it is possible to accurately predict whether the obstacle is approaching or moving away from the vehicle.

[0165] The training method for any trajectory prediction model provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the training method for any trajectory prediction model provided in this disclosure can be executed by a processor, such as by a processor executing the training method for any trajectory prediction model mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0166] Figure 7 This is a schematic flowchart of a trajectory prediction method provided in an exemplary embodiment of this disclosure. Figure 7 The method shown may include steps 710, 720 and 730, which are described below.

[0167] Step 710: Obtain the third extracted feature corresponding to the third image sequence. The third image sequence includes images captured by the second camera set on the second mobile device at multiple times.

[0168] It should be noted that the method for obtaining the third image sequence is the same as the method for obtaining the first image sequence described above, and the method for obtaining the third extracted features is the same as the method for obtaining the first extracted features described above, and will not be repeated here.

[0169] Step 720: Through the target trajectory prediction model, random perturbations are applied to the third extracted features to obtain the second perturbation features corresponding to each of the multiple random perturbations.

[0170] It should be noted that the target trajectory prediction model is a trajectory prediction model trained using the training method of the trajectory prediction model mentioned above. The method for obtaining the second perturbation feature is the same as the method for obtaining the first perturbation feature mentioned above, and will not be repeated here.

[0171] Step 730: Based on the second perturbation features corresponding to each of the multiple random perturbations, the multimodal trajectory prediction results of the obstacles in the third image sequence are generated through the target trajectory prediction model.

[0172] It should be noted that the method for generating the multimodal trajectory prediction results of obstacles in the third image sequence is the same as the method for generating the multimodal trajectory prediction results of obstacles in the first image sequence described above, and will not be repeated here.

[0173] The trajectory prediction scheme provided in the embodiments of this disclosure performs trajectory prediction directly from the camera's perspective, rather than from the BEV's perspective. Therefore, it does not require the reconstruction of depth information, which helps to improve the accuracy of the trajectory prediction results.

[0174] In an optional example, such as Figure 8As shown, during the model training phase, multiple obstacle information can be obtained through obstacle detection or obstacle segmentation. By aggregating this information, a historical frame feature sequence (equivalent to the obstacle information sequence mentioned above) can be obtained. The first extracted feature is obtained by using an LSTM feature extractor to extract features from this historical frame feature sequence. Based on the first extracted feature, the CVAE distribution 1 (equivalent to the first feature perturbation distribution parameter mentioned above) can be obtained through the conditional variational autoencoder in the trajectory prediction model to be trained. CVAE distribution 1 can be used to determine the first perturbation feature. Based on multiple first perturbation features, the LSTM trajectory generator in the trajectory prediction model to be trained can generate multimodal trajectory prediction results. Additionally, a future frame ground truth sequence (equivalent to the second image sequence mentioned above) can be obtained. Based on this future frame ground truth sequence, the CVAE distribution 2 (equivalent to the second feature perturbation distribution parameter mentioned above) can also be obtained through the conditional variational autoencoder in the trajectory prediction model to be trained. Through learning during the model training phase, CVAE distribution 1 and CVAE distribution 2 can be made sufficiently close.

[0175] Optionally, each obstacle information in the obstacle information sequence may include obstacle bounding box information or obstacle segmentation result information, as well as obstacle orientation information or other feature information suitable for aggregation into a sequence.

[0176] The operations performed during the actual model execution phase are basically similar to those during the model training phase. The main difference is that the second image sequence and CVAE distribution2 are no longer needed during the actual model execution phase, thus eliminating the need for... Figure 8 The operation is represented by the dashed line.

[0177] Any trajectory prediction method provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any trajectory prediction method provided in this disclosure can be executed by a processor, such as by a processor executing any trajectory prediction method mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0178] Exemplary device

[0179] Figure 9 This is a schematic diagram of the structure of a training device for a trajectory prediction model provided in an exemplary embodiment of this disclosure. Figure 9 The apparatus shown includes a first acquisition module 910, a first disturbance application module 920, a first generation module 930, a training module 940, and a determination module 950.

[0180] The first acquisition module 910 is used to acquire the first extractable feature corresponding to the first image sequence, wherein the first image sequence includes: images captured by a first camera set on a first mobile device at multiple historical moments;

[0181] The first perturbation application module 920 is used to apply random perturbations to the first extracted features obtained by the first acquisition module 910 via the trajectory prediction model to be trained, so as to obtain the first perturbation features corresponding to each of the multiple random perturbations.

[0182] The first generation module 930 is used to generate multimodal trajectory prediction results of obstacles in the first image sequence based on the first perturbation features corresponding to the plurality of random perturbations obtained by the first perturbation application module 920, via the trajectory prediction model to be trained.

[0183] Training module 940 is used to train the trajectory prediction model to be trained based on the second image sequence and the multimodal trajectory prediction result generated by the first generation module 930. The second image sequence includes images captured by the first camera at multiple future times after the multiple historical times.

[0184] The determination module 950 is used to determine the trained trajectory prediction model as the target trajectory prediction model in response to the training module 940 meeting the preset training termination condition.

[0185] In an optional example, such as Figure 10 As shown,

[0186] The first disturbance application module 920 includes:

[0187] The first determining submodule 9201 is used to determine the first feature perturbation distribution parameters corresponding to the first extracted feature via the conditional variational autoencoder in the trajectory prediction model to be trained.

[0188] The perturbation application submodule 9203 is used to apply random perturbations to the first extracted features obtained by the first acquisition module 910 via a conditional variational autoencoder based on the first feature perturbation distribution parameters determined by the first determination submodule 9201, thereby obtaining the first perturbation features corresponding to each of the multiple random perturbations.

[0189] Training module 940 is specifically used for:

[0190] The trajectory prediction model to be trained is trained based on the second image sequence, the multimodal trajectory prediction results generated by the first generation module 930, and the first feature perturbation distribution parameters determined by the first determination submodule 9201.

[0191] In an optional example, such as Figure 10 As shown, training module 940 includes:

[0192] The first acquisition submodule 9401 is used to acquire the second extracted features corresponding to the second image sequence;

[0193] The second determining submodule 9403 is used to determine the second feature perturbation distribution parameters corresponding to the second extracted feature obtained by the first obtaining submodule 9401 via a conditional variational autoencoder.

[0194] The third determining submodule 9405 is used to determine the first model loss value by comparing the first characteristic perturbation distribution parameters determined by the first determining submodule 9201 with the second characteristic perturbation distribution parameters determined by the second determining submodule 9403.

[0195] The first training submodule 9407 is used to train the trajectory prediction model to be trained based on the first model loss value determined by the third determination submodule 9405.

[0196] In one optional example, the multimodal trajectory prediction results include: multiple predicted trajectories of the obstacle, and the prediction probability values ​​corresponding to each of the multiple predicted trajectories;

[0197] like Figure 11 As shown, training module 940 includes:

[0198] The fourth determining submodule 9409 is used to determine the future true trajectory of the obstacle based on the second image sequence;

[0199] The fifth determining submodule 9411 is used to determine the first trajectory loss value based on the future true trajectory determined by the fourth determining submodule 9409 and multiple predicted trajectories;

[0200] The sixth determining submodule 9413 is used to determine the probability loss value based on the future true trajectory determined by the fourth determining submodule 9409, multiple predicted trajectories, and the predicted probability values ​​corresponding to each of the multiple predicted trajectories.

[0201] The seventh determining submodule 9415 is used to determine the second model loss value based on the first trajectory loss value determined by the fifth determining submodule 9411 and the probability loss value determined by the sixth determining submodule 9413;

[0202] The second training submodule 9417 is used to train the trajectory prediction model to be trained based on the second model loss value determined by the seventh determination submodule 9415.

[0203] In one optional example, the fifth determining submodule 9411 includes:

[0204] The first determining unit is used to determine the similarity between multiple predicted trajectories and the future true trajectories determined by the fourth determining submodule 9409;

[0205] The second determining unit is used to determine the weights of the multiple predicted trajectories based on the similarity of each of the multiple predicted trajectories determined by the first determining unit.

[0206] The third determining unit is used to determine the second trajectory loss value for each of the multiple predicted trajectories based on the similarity of each predicted trajectory determined by the first determining unit.

[0207] The weighting unit is used to weight the second trajectory loss values ​​corresponding to the multiple predicted trajectories determined by the third determining unit using the weights corresponding to the multiple predicted trajectories determined by the second determining unit, so as to obtain the first trajectory loss value.

[0208] In one optional example, the sixth determining submodule 9413 includes:

[0209] The fourth determining unit is used to determine the similarity between multiple predicted trajectories and the future true trajectories determined by the fourth determining submodule 9409;

[0210] The fifth determining unit is used to determine reference probability values ​​for the multiple predicted trajectories based on the similarity values ​​corresponding to each of the multiple predicted trajectories determined by the fourth determining unit.

[0211] The sixth determining unit is used to determine the probability loss value based on the prediction probability values ​​corresponding to each of the multiple prediction trajectories and the reference probability values ​​corresponding to each of the multiple prediction trajectories determined by the fifth determining unit.

[0212] In an optional example, such as Figure 12 As shown, the first acquisition module 910 includes:

[0213] The overlay submodule 9101 is used to overlay multiple images in the first image sequence along the channel direction to obtain an overlay image;

[0214] The first feature extraction submodule 9103 is used to extract features from the superimposed image obtained by the superimposition submodule 9101 to obtain the first extracted features corresponding to the first image sequence.

[0215] In an optional example, such as Figure 13 As shown, the first acquisition module 910 includes:

[0216] The second acquisition submodule 9105 is used to acquire obstacle information corresponding to each of the multiple images in the first image sequence, and obtain multiple obstacle information. Each obstacle information includes: obstacle bounding box information or obstacle segmentation result information.

[0217] The second feature extraction submodule 9107 is used to extract features from the obstacle information sequence composed of multiple obstacle information acquired by the second acquisition submodule 9105 to obtain the first extracted features corresponding to the first image sequence.

[0218] In one optional example, the second feature extraction submodule 9107 includes:

[0219] The seventh determining unit is used to sequentially take each obstacle information in the obstacle information sequence as the current obstacle information;

[0220] The first generation unit is used to respond to the fact that the current obstacle information determined by the seventh determining unit is the first obstacle information in the obstacle information sequence, and to generate the extracted features corresponding to the current obstacle information based on the current obstacle information determined by the seventh determining unit, through the feature extractor in the trajectory prediction model to be trained.

[0221] The second generation unit is used to respond to the fact that the current obstacle information determined by the seventh determining unit is not the first obstacle information in the obstacle information sequence, and to generate the extracted features corresponding to the previous obstacle information of the current obstacle information determined by the seventh determining unit, and the current obstacle information, through the feature extractor.

[0222] The eighth determining unit is configured to, in response to the seventh determining unit determining the current obstacle information as the last obstacle information in the obstacle information sequence, determine the extracted features corresponding to the current obstacle information determined by the seventh determining unit as the first extracted features corresponding to the first image sequence.

[0223] In one optional example,

[0224] The multimodal trajectory prediction results include: multiple predicted trajectories of the obstacle;

[0225] Each of the multiple predicted trajectories is represented using target data, which includes: each of the multiple future moments, and the coordinates of the four corner points of the obstacle's bounding box.

[0226] Figure 14 This is a schematic diagram of the trajectory prediction device provided in an exemplary embodiment of the present disclosure. Figure 14 The apparatus shown includes a second acquisition module 1410, a second disturbance application module 1420, and a second generation module 1430.

[0227] The second acquisition module 1410 is used to acquire the third extracted feature corresponding to the third image sequence. The third image sequence includes images captured by the second camera set on the second mobile device at multiple times.

[0228] The second perturbation application module 1420 is used to apply random perturbations to the third extracted features obtained by the second acquisition module 1410 via the target trajectory prediction model, so as to obtain the second perturbation features corresponding to each of the multiple random perturbations.

[0229] The second generation module 1430 is used to generate multimodal trajectory prediction results of obstacles in the third image sequence based on the second perturbation features corresponding to the multiple random perturbations obtained by the second perturbation application module 1420 and the target trajectory prediction model.

[0230] In the apparatus disclosed herein, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve the corresponding functions and effects, and this disclosure does not list them all.

[0231] Exemplary electronic devices

[0232] Below, for reference Figure 15 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0233] Figure 15 A block diagram of an electronic device 1500 according to an embodiment of the present disclosure is shown.

[0234] like Figure 15 As shown, the electronic device 1500 includes one or more processors 1510 and memory 1520.

[0235] The processor 1510 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1500 to perform desired functions.

[0236] The memory 1520 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1510 may execute the program instructions to implement the image processing methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0237] In one example, the electronic device 1500 may also include an input device 1530 and an output device 1540, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0238] For example, when electronic device 1500 is a first device or a second device, the input device 1530 may be a microphone or a microphone array. When electronic device 1500 is a standalone device, the input device 1530 may be a communication network connector for receiving acquired input signals from the first device and the second device.

[0239] In addition, the input device 1530 may also include, for example, a keyboard, a mouse, etc.

[0240] The output device 1540 can output various information to the outside. The output device 1540 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0241] Of course, for the sake of simplicity, Figure 15 Only some of the components of the electronic device 1500 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1500 may include any other suitable components depending on the specific application.

[0242] Exemplary computer program products and computer-readable storage media

[0243] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the training method or trajectory prediction method of the trajectory prediction model according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0244] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0245] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the training method or trajectory prediction method of the trajectory prediction model according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0246] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0247] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. The specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0248] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0249] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0250] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0251] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0252] The above description of the disclosed aspects is provided to enable those skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. This disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0253] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A training method for a trajectory prediction model, comprising: Obtain a first extracted feature corresponding to a first image sequence, wherein the first image sequence includes: images captured by a first camera set on a first mobile device at multiple historical moments; By applying random perturbations to the first extracted features through the trajectory prediction model to be trained, multiple first perturbation features corresponding to each random perturbation are obtained; Based on the first perturbation features corresponding to each of the multiple random perturbations, the multimodal trajectory prediction results of obstacles in the first image sequence are generated through the trajectory prediction model to be trained. Based on the second image sequence and the multimodal trajectory prediction results, the trajectory prediction model to be trained is trained. The second image sequence includes images captured by the first camera at multiple future times after the multiple historical times. In response to the training trajectory prediction model meeting the preset training termination condition, the training trajectory prediction model is determined as the target trajectory prediction model.

2. The method according to claim 1, wherein, The process involves applying random perturbations to the first extracted features via a trajectory prediction model to be trained, resulting in multiple first perturbation features corresponding to each random perturbation, including: The first feature perturbation distribution parameters corresponding to the first extracted feature are determined by the conditional variational autoencoder in the trajectory prediction model to be trained; Based on the first feature perturbation distribution parameters, random perturbations are applied to the first extracted features via the conditional variational autoencoder to obtain first perturbation features corresponding to multiple random perturbations. The step of training the trajectory prediction model based on the second image sequence and the multimodal trajectory prediction results includes: The trajectory prediction model to be trained is trained based on the second image sequence, the multimodal trajectory prediction result, and the first feature perturbation distribution parameters.

3. The method according to claim 2, wherein, The step of training the trajectory prediction model based on the second image sequence, the multimodal trajectory prediction result, and the first feature perturbation distribution parameters includes: Obtain the second extracted feature corresponding to the second image sequence; The second feature perturbation distribution parameters corresponding to the second extracted feature are determined via the conditional variational autoencoder. The first model loss value is determined by comparing the first feature perturbation distribution parameters and the second feature perturbation distribution parameters; The trajectory prediction model to be trained is trained based on the loss value of the first model.

4. The method according to claim 1, wherein, The multimodal trajectory prediction results include: multiple predicted trajectories of the obstacle, and the prediction probability values ​​corresponding to each of the multiple predicted trajectories; The step of training the trajectory prediction model based on the second image sequence and the multimodal trajectory prediction results includes: Based on the second image sequence, determine the future true trajectory of the obstacle; Based on the future true trajectory and the multiple predicted trajectories, determine the first trajectory loss value; Based on the future true trajectory, the multiple predicted trajectories, and the predicted probability values ​​corresponding to each of the multiple predicted trajectories, the probability loss value is determined. Based on the first trajectory loss value and the probability loss value, determine the second model loss value; The trajectory prediction model to be trained is trained based on the loss value of the second model.

5. The method according to claim 4, wherein, The step of determining the first trajectory loss value based on the future true trajectory and the multiple predicted trajectories includes: The similarity between the multiple predicted trajectories and the future actual trajectories is determined respectively; Based on the similarity corresponding to each of the multiple predicted trajectories, weights are determined for each of the multiple predicted trajectories; Based on the similarity corresponding to each of the multiple predicted trajectories, a second trajectory loss value is determined for each of the multiple predicted trajectories; By using the weights corresponding to each of the multiple predicted trajectories, the second trajectory loss values ​​corresponding to each of the multiple predicted trajectories are weighted to obtain the first trajectory loss value.

6. The method according to claim 4, wherein, The determination of the probability loss value based on the future true trajectory, the multiple predicted trajectories, and the predicted probability values ​​corresponding to each of the multiple predicted trajectories includes: The similarity between the multiple predicted trajectories and the future actual trajectories is determined respectively; Based on the similarity corresponding to each of the multiple predicted trajectories, a reference probability value is determined for each of the multiple predicted trajectories; Based on the predicted probability value and the reference probability value corresponding to each of the multiple predicted trajectories, the probability loss value is determined.

7. The method according to claim 1, wherein, The step of obtaining the first extracted feature corresponding to the first image sequence includes: Multiple images from the first image sequence are superimposed along the channel direction to obtain a superimposed image; Feature extraction is performed on the superimposed image to obtain the first extracted feature corresponding to the first image sequence; or, The step of obtaining the first extracted feature corresponding to the first image sequence includes: Obtain obstacle information corresponding to each of the multiple images in the first image sequence to obtain multiple obstacle information. Each obstacle information includes: obstacle bounding box information or obstacle segmentation result information. Feature extraction is performed on the obstacle information sequence composed of the multiple obstacle information to obtain the first extracted feature corresponding to the first image sequence.

8. The method according to claim 7, wherein, The step of extracting features from the obstacle information sequence composed of the multiple obstacle information to obtain the first extracted features corresponding to the first image sequence includes: Each obstacle information in the obstacle information sequence is used as the current obstacle information in turn; In response to the fact that the current obstacle information is the first obstacle information in the obstacle information sequence, based on the current obstacle information, the feature extractor in the trajectory prediction model to be trained generates the extracted features corresponding to the current obstacle information; In response to the fact that the current obstacle information is not the first obstacle information in the obstacle information sequence, the extracted features corresponding to the previous obstacle information of the current obstacle information are generated by the feature extractor based on the extracted features of the current obstacle information and the current obstacle information. In response to the fact that the current obstacle information is the last obstacle information in the obstacle information sequence, the extracted feature corresponding to the current obstacle information is determined as the first extracted feature corresponding to the first image sequence.

9. The method according to claim 1, wherein, The multimodal trajectory prediction results include: multiple predicted trajectories of the obstacle; Each of the multiple predicted trajectories is characterized using target data, which includes: the coordinates of the four corner points of the bounding box of the obstacle at each of the multiple future moments.

10. A trajectory prediction method, comprising: Obtain the third extracted feature corresponding to the third image sequence, wherein the third image sequence includes: images captured by a second camera set on a second mobile device at multiple times; By applying random perturbations to the third extracted features through the target trajectory prediction model, a second perturbation feature corresponding to each of the multiple random perturbations is obtained; Based on the second perturbation features corresponding to each of the multiple random perturbations, the multimodal trajectory prediction results of the obstacles in the third image sequence are generated via the target trajectory prediction model; The target trajectory prediction model is trained using the training method for trajectory prediction models as described in any one of claims 1-9.

11. A training device for a trajectory prediction model, comprising: The first acquisition module is used to acquire the first extractable feature corresponding to the first image sequence, wherein the first image sequence includes: images acquired by a first camera set on a first mobile device at multiple historical moments; The first perturbation application module is used to apply random perturbations to the first extracted features obtained by the first acquisition module via the trajectory prediction model to be trained, so as to obtain the first perturbation features corresponding to each of the multiple random perturbations. The first generation module is used to generate multimodal trajectory prediction results of obstacles in the first image sequence based on the first perturbation features corresponding to the plurality of random perturbations obtained by the first perturbation application module and via the trajectory prediction model to be trained. The training module is used to train the trajectory prediction model to be trained based on the second image sequence and the multimodal trajectory prediction results generated by the first generation module. The second image sequence includes images captured by the first camera at multiple future times after the multiple historical times. The determination module is used to determine the trained trajectory prediction model as the target trajectory prediction model in response to the training module meeting the preset training termination condition.

12. A trajectory prediction device, comprising: The second acquisition module is used to acquire the third extracted feature corresponding to the third image sequence, wherein the third image sequence includes: images captured by the second camera set on the second mobile device at multiple times; The second perturbation application module is used to apply random perturbations to the third extracted features obtained by the second acquisition module via the target trajectory prediction model, thereby obtaining a second perturbation feature corresponding to each of the multiple random perturbations. The second generation module is used to generate multimodal trajectory prediction results of obstacles in the third image sequence based on the second perturbation features corresponding to the plurality of random perturbations obtained by the second perturbation application module and via the target trajectory prediction model. The target trajectory prediction model is trained using the training method for trajectory prediction models as described in any one of claims 1-9.

13. A computer-readable storage medium storing a computer program for executing a training method for a trajectory prediction model according to any one of claims 1-9, or executing a trajectory prediction method according to claim 10.

14. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the training method of the trajectory prediction model according to any one of claims 1-9, or to execute the trajectory prediction method according to claim 10.

Images