Computer-implemented method for training a model, as well as method and system for predicting the behavior of a road user.

By transforming training data into a Gaussian mixture distribution and using a CVAE with a Gaussian mixing context likelihood distribution, the method addresses the limitations of unimodal representations in CVAE models, enabling accurate and diverse behavioral predictions for automated vehicles.

DE102023210641A1Pending Publication Date: 2025-04-30ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE102023210641
Authority / Receiving Office
DE · DE
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-10-27
Publication Date
2025-04-30

AI Technical Summary

Technical Problem

Existing CVAE models struggle to accurately predict the future behavior of participants in a traffic scene due to their reliance on continuous, unimodal latent representations, which fail to capture the multimodal nature of output information, leading to unrealistic predictions that can impact downstream behavior planning for automated vehicles.

Method used

Transform training data elements into parameters of a Gaussian mixture distribution with multiple components and use a conditional variational autoencoder (CVAE) to model the context likelihood distribution as a Gaussian mixing distribution, allowing for the generation of output information with a multi-modal distribution.

Benefits of technology

The proposed method enables the CVAE to generate realistic and diverse behavioral predictions by accounting for multiple possible actions in a traffic scenario, improving the accuracy and reliability of automated vehicle behavior planning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Measures are proposed that enable the use of a CVAE model to generate output information with a multimodal distribution, particularly in the context of predicting the behavior of participants in a traffic scene and also for generating digital images as training data for a machine learning (ML) component. The claimed training method for such a CVAE model provides that the training data elements, each comprising at least one ground truth information Y* (11) with associated training context information X* (12), are represented in parameters of a Gaussian mixture distribution q (3) with at least one component N ( μ q , ∑ q ) are transformed and that the training context information X* (12) is transformed into parameters of a Gaussian mixture distribution p (30) with at least two components N ( μ pi , ∑ pi ) (31, 32, 33) are transformed.
Need to check novelty before this filing date? Find Prior Art

Description

State of the art

[0001] The invention is based on the concept of a Conditional Variational Auto-Encoder, as described, for example, in Ghosh et al., “From variational to deterministic autoencoder”, https: / / arxiv.org / abs / 1903.12436.

[0002] Accordingly, the invention relates to a computer-implemented method for training a model comprising a Conditional Variational Auto Encoder (CVAE) structure with a label encoder ϕ, a prior encoder γ, and a decoder θ. For the training method, a set of training data elements is provided, each comprising at least one ground truth information Y* with associated training context information X*. The physical meaning of the ground truth information Y* and the associated context information X* depends on the specific task or application for which the model is being trained. An important application for such models is the prediction of the behavior of participants in a traffic scene within the context of behavior planning for an automated vehicle.

[0003] To train the model, the following steps are performed for each training data element: - Use of the label encoder ϕ to transform the training data element into parameters of a multivariate label probability distribution of latent features, - Use of the prior encoder γ to transform the training context information X* into parameters of a multivariate context probability distribution of latent traits, - Determining an initial distance between the label probability distribution and the context probability distribution and comparing this initial distance with a given initial optimization criterion, - Generate at least one label sample Z ϕ the multivariate label probability distribution, - Use of the decoder θ to generate at least one output piece of information Y based on the label sample Z ϕ , - Determining a second distance between the ground truth information Y* and the generated output information Y and comparing the second distance with a given second optimization criterion, and - Modify at least one parameter of the label encoder ϕ, the prior encoder γ and / or the decoder θ if the first and / or the second optimization criterion is not met.

[0004] To plan safe and predictable maneuvers, automated vehicles must be able to anticipate how the current traffic situation will develop. This is particularly important when other participants, such as other vehicles, cyclists, and pedestrians, are also present in the traffic scene. Therefore, attempts are made to predict the future behavior of as many participants in the traffic scene as possible, usually in the form of trajectories. This prediction is then taken into account when planning the behavior of the automated vehicle. Classical prediction methods generally use simple kinematic models for the individual participants in the traffic scene and can only partially model interactions between participants. For this reason, the use of machine learning, especially deep learning (DL), has become the de facto standard for prediction in recent years.

[0005] In practice, a Conditional Variational Auto-Encoder (CVAE) model is often used to predict trajectories, as such models capture the relationship between the context of the traffic scene and the ground truth future in a probabilistic latent space and take this into account in the prediction.

[0006] The relationship between the traffic scene context and the future ground truth is essentially learned through two measures of the training process. One of these measures involves using training data elements as input for the label encoder ϕ, which include both ground truth information Y* and the corresponding training context information X*. The label encoder ϕ then maps these training data elements to parameters of a multivariate label probability distribution. The decoder θ then processes the label samples Z. ϕThis label probability distribution is used to generate output information Y. The other measure is that the multivariate context probability distribution is learned during training so that it is as similar as possible to the label probability distribution. The relationship between the training context information X* and the ground truth information Y* is thus learned primarily by the label encoder ϕ using both the ground truth information Y* and the corresponding training context information X*, thereby modeling a joint probability from which label samples Z are generated. ϕ The context probability distribution is drawn for the decoder θ. Since the context probability distribution is trained to be as similar as possible to the label probability distribution, output information Y can later be inferred from context information X* using the prior encoder γ.

[0007] The idea behind a CVAE is to condition to a context and to represent "nuances" in a distribution of latent characteristics. For this purpose, the label probability distribution and the context probability distribution are usually modeled as a multivariate normal distribution, specifically as a standard normal distribution. The latent representation in this case is therefore continuous and unimodal.

[0008] However, a unimodal latent representation does not do justice to some use cases, especially use cases with a highly multimodal distribution of the output information Y.

[0009] Behavioral prediction is one such application. The future behavior of a participant in a traffic scenario is always significantly determined by the available route options and lanes. For example, a participant's future behavior depends largely on whether they decide to turn right or left at an intersection, or to continue straight ahead. These three options correspond to three completely different, but realistic, behavioral modes. However, if the samples for prediction are generated from a continuous unimodal probability distribution in latent space, the prediction inevitably also produces forecasts that lie between the different behavioral modes and consequently represent not realistic, but nonsensical behaviors, such as driving "diagonally" across the intersection.This can have serious implications for downstream behavior planning for an automated vehicle. Disclosure of the invention

[0010] Measures are proposed that enable the use of a CVAE model to generate output information with a multi-modal distribution. These measures specifically allow the use of a suitably configured CVAE for predicting the behavior of participants in a traffic scenario, as well as for generating digital images as training data for a machine learning (ML) component.

[0011] This is achieved according to the invention by defining the training data elements as parameters of a Gaussian mixture distribution q with at least one component N(μq,∑q) are transformed and that the training context information X* is converted into parameters of a Gaussian mixture distribution p with at least two components N(μpi,∑pi) be transformed.

[0012] The invention is based on the assumption that a multimodality of the output information is already evident in the context probability distribution. Therefore, according to the invention, it is proposed to represent the training context information X* in parameters of a Gaussian mixture distribution p with several components. N(μpi,∑pi) to transform. Ideally, the individual components of this mixture distribution could be assigned to the different modes of the output information.

[0013] According to the invention, the training data elements are also transformed into parameters of a Gaussian mixture distribution q. This can comprise several components or just a single one. In the latter case, the training data elements are then transformed into parameters of a simple Gaussian normal distribution.

[0014] Particularly for behavioral prediction, it proves advantageous if the given number N of components of the context probability distribution p is greater than the given number M of components of the label probability distribution q. This takes into account the fact that a single training context, such as a traffic scene in the area of ​​a four-way intersection, can often lead to several different behavioral modes, such as "straight ahead", "turn right" or "turn left", while the label probability distribution is less "fuzzy" and, for example, only reflects the uncertainty within the "straight ahead" mode.

[0015] To determine the first distance between the label probability distribution q and the context probability distribution p, any distance measure can be used. Preferably, this first distance is determined based on the Kullback-Leibler (KL) divergence. The KL divergence for two multivariate unimodal Gaussian distributions N(μp,∑p) and N(μq,∑q) The dimension k is given by: DKL(p‖q)=12[log|∑q||∑p|−k+(μp−μq)T∑q−1(μp−μq)+tr{∑q−1∑p}]

[0016] The first distance can also be determined, for example, based on the Wasserstein distance.

[0017] In a preferred embodiment of the invention, a partial distance is first established between each of the individual components. N(μq,∑q) the label probability distribution q and the individual components N(μpi,∑pi) The context probability distribution p is determined. To determine the first distance between the label probability distribution q and the context probability distribution p, these partial distances are then added together, whereby a weighting of the individual partial distances can be carried out.

[0018] Weighting allows for considering only a selection of partial distances when determining the first distance. For example, in the case of a label probability distribution q in the form of a simple Gaussian distribution, the first distance can also be determined as the distance between this simple Gaussian distribution q and the "most fitting" component of the context probability distribution p.

[0019] It should also be noted that any suitable distance measure can be chosen for determining the partial distances, such as the KL divergence mentioned above. The weighted sum of the KL divergence can then be represented as ∑iN wiDKL(pi‖q), and the selection of the "most suitable" component as DKL(pi*‖q) where i* either describes the component with the smallest KL divergence, i.e. i*=argminiDKL(pi‖q) or is otherwise determined, e.g. by comparing the trajectories θ(p) decoded from a component using the decoder θ i ) with the ground-truth trajectory t gt In this case, it would be i∗=argminiD(θ(pi);tgt), where D is a suitable distance measure.

[0020] In a preferred embodiment of the invention, for each component N(μpi,∑pi) the context probability distribution p a fashion probability w i learned by training the context probability distribution p as a Gaussian mixture distribution p=∑i=1NwiN(μpi,∑pi).

[0021] A preferred way to learn fashion probabilities w i consists of performing the following steps for each training data element: - Generate at least one context sample Z γ i for each component N(μpi,∑pi) the context probability distribution p, - Use of the decoder θ to generate at least one output piece of information Y i for each context sample generated in this way Z γ i , - Determine a third distance between the output information Y thus generated. iand the ground truth information Y* and comparing the third distance with a given third optimization criterion and - Modify at least one parameter of the prior encoder γ if the third optimization criterion is not met.

[0022] In this case, the Prior Encoder γ determines the components for each training data element. pi=N(μpi,∑pi),i=1,…,N the context probability distribution p and the fashion probabilities w i , so that the context probability distribution p can be represented as p=∑i=1NwiN(μpi,∑pi).

[0023] The fashion probabilities w i They can be trained by optimizing the KL divergence. For this, the component i* of the context probability distribution p is determined, which in the trajectory space best corresponds to the ground truth trajectory, i.e., i*argmini D(θ(pi);tgt).

[0024] The vector m is then used as the "ground-truth" weights for the individual modes, where mi=0 for all components i=1,…,N;i≠i* andmi∗=1.

[0025] For training with weights w, a comparative measure between m and w is then optimized. For example, KL divergence or cross-entropy can be used as a comparative measure.

[0026] The label samples Z ϕ and / or the context samples Z γ i can be generated randomly and / or deterministically, for example using the Unscented Transformation method.

[0027] It proves particularly advantageous if the decoder θ generates at least one output piece of information Y using not only the generated label sample Z. ϕ not only uses the training context information X*, but also, if necessary, the generated context sample Z. y .

[0028] As mentioned at the outset, the invention is particularly suitable for use in behavioral prediction. In this case, the training data elements each comprise at least one ground truth behavior as ground truth information and training scene-specific information as associated training context information. The decoder then generates at least one behavior of one or more participants as output information. Both the ground truth behavior and the generated or predicted behavior of the participant(s) are often described by trajectories, which, in addition to location and time information, can also include state information such as orientation, speed, and acceleration, etc.

[0029] Another advantageous application of the invention lies in generating digital images as training data for a machine learning (ML) component. In this case, the training data elements each comprise at least one ground truth image, such as an image of a traffic situation at an intersection in sunshine, and image attributes, such as the textual description and information "intersection in sunshine" captured by appropriate sensors, as associated training context information.

[0030] The decoder then generates at least one image as output information.

[0031] In addition to the claimed training method, the inference and corresponding systems are also claimed, specifically for the two use cases: predicting the behavior of at least one participant in a traffic scene and generating digital images as training data for a machine learning (ML) component. For each of these purposes, a model with a CVAE structure is used, which, according to the invention, has been trained with a set of training data elements for the respective use case.

[0032] The claimed computer-implemented method for predicting the behavior of at least one participant in a traffic scene therefore comprises the following steps: a. Aggregating scene-specific information about a traffic scene, b. Using the pre-trained prior encoder γ to transform the scene-specific information into parameters of a multivariate context probability distribution p in the form of a Gaussian mixture distribution p with at least two components N(μpi,∑pi), c. Generate at least one context sample Z γ i the context probability distribution p and d. Using the pre-trained decoder θ to generate at least one behavioral prediction Y i based on at least one context sample Z γ i .

[0033] Basically, the context samples can Z γ i simply generated from the context probability distribution p, without sampling the individual components during this "sampling". N(μpi,∑pi) to consider the Gaussian mixture distribution p. However, it proves advantageous in several respects to consider the context samples Z. γ i from selected components N(μpi,∑pi) to generate the context probability distribution p.

[0034] If the component N(μpi,∑pi) the context probability distribution p is known, from which a context sample Z is selected. γ i If the probability of occurrence of the behavioral prediction Y generated by it has been generated, then the probability of occurrence can be determined. i simply based on learned fashion probability w i this component N(μpi,∑pi) to be evaluated or weighted.

[0035] Furthermore, component-wise sampling of the context probability distribution p also allows statements about the uncertainty of a behavioral prediction Y. i. For this purpose, at least one component is required. N(μpi,∑pi) the context probability distribution p multiple context samples Z γ i and behavioral predictions based on that Y i generated based on the distribution of these behavioral predictions Y i A measure of the uncertainty of behavioral predictions for the corresponding mode can then be derived. Advantageously, a parametric description of the distribution of behavioral predictions for each mode can be used, such as a Gaussian distribution for each mode. The uncertainty of behavioral predictions for each mode can then be easily determined based on the parameters of these distributions, for example, the expected value and the standard deviation.

[0036] Context samples can also be used in inference. γ i can be generated randomly or deterministically.

[0037] Here too, it proves advantageous if the pre-trained decoder θ for behavior prediction is used in addition to the context samples Z. γ i uses the relevant aggregated scene-specific information from the traffic scene.

[0038] The claimed computer-implemented method for generating digital images as training data for a machine learning (ML) component comprises the following steps: a. Providing image attributes, especially in the form of a textual description of image content, b. Using the pre-trained prior encoder y to transform the image attributes into parameters of a multivariate context probability distribution p in the form of a Gaussian mixture distribution p with at least two components N(μpi,∑pi), c. Generate at least one context sample Z γ ithe context probability distribution p, d. Using the pre-trained decoder θ to generate at least one digital image Y i based on at least one context sample Z γ i .

[0039] This method can be used to generate at least one new image based on a changed textual description, e.g., "intersection at dusk". Drawings

[0040] Exemplary embodiments and advantageous further developments of the invention are explained in more detail below in conjunction with the figures. Fig. Figure 1 shows a state-of-the-art training setup of a CVAE for predicting the trajectories of participants in a traffic scene. Fig. Figure 2 illustrates the training according to the invention and the resulting configuration according to the invention of a CVAE model for the prediction of trajectories of participants in a traffic scene. Description of exemplary implementations

[0041] In Fig. Figure 1 shows a training setup of a CVAE structure comprising a label encoder ϕ (referenced with 1), a prior encoder γ (referenced with 2), and a decoder θ (referenced with 5). A set of training data elements is provided for training this CVAE structure, each containing at least one ground truth information Y* 11 with associated training context information X* 12. Fig. Figure 1 shows such a training data element, which serves as an example to illustrate the training procedure. Since the CVAE structure in the described embodiment is to be trained for the behavioral prediction of participants in a traffic scene, trajectory data of a vehicle 10 is used here as ground truth information Y* 11. Scene-specific information describing the associated training traffic scene 20 is used as training context information X* 12. This scene is a four-way intersection. Vehicle 10 approaches this four-way intersection from the left and essentially has three behavior options: "Go straight," "Turn right," or "Turn left." A trajectory of the "Go straight" behavior option is represented as ground truth information Y*.

[0042] Using the label encoder ϕ 1, the training data element, i.e., the ground truth information Y* 11 together with the training context information X* 12, is converted into parameters of a multivariate label probability distribution N(µ). ϕ , Σ ϕ ) 3 in the latent space. Independently of this, only the training context information X* 12 of the training data element is transformed into parameters of a multivariate context probability distribution N(µ) using the prior encoder γ 2. γ ,Σ γ ) 4 transformed in latent space. The label probability distribution N(µ ϕ ,Σ ϕ ) 3 and the context probability distribution N(µ γ ,Σ γ ) 4 are modeled as standard normal distributions in the course of the training procedure described here, which are characterized by the parameters, expected value vector µ ϕ or µ γ and covariance matrix Σ ϕ or Σ γThis can be described. For this purpose, a distance is calculated between the label probability distribution N(µ). ϕ , Σ ϕ ) 3 and the associated context probability distribution N(µ γ ,Σ γ ) 4 is determined and compared with a first optimization criterion. A Kullback-Leibier (KL) divergence is used as the distance measure here.

[0043] Based on the label probability distribution N(µ ϕ , Σ ϕ ) 3 or their parameters, a label sample designated with 13 Z will be used. ϕ generated. In addition, N(µ) is generated based on the context probability distribution. γ , Σ γ ) 4 or their parameters, a context sample designated with 14 Z γ generated. Samples 13 and 14 can be drawn either randomly or deterministically, e.g. as with the Unscented Kalman Filter.

[0044] Based on the label sample Z ϕ13. Decoder θ 5 then generates a trajectory as output information Y, which is denoted here by 15. In the embodiment described here, decoder θ 5 also uses the training context information X* 12 for this purpose. Additionally, decoder θ could also use the context sample Z. γ Use 14. If multiple label samples are used, Z ϕ from the label probability distribution N(µ ϕ ,Σ ϕ ) 3 is drawn, then the decoder θ5 generates a family of trajectories, the distribution of which is denoted here by 6.

[0045] Finally, a distance between the ground truth information Y* 11 and the generated output information Y 15, i.e., between the ground truth trajectory 11 and the output trajectory 15, is determined and compared with a predefined second optimization criterion. If the first and / or the second optimization criterion is not met, the parameters of the label encoder ϕ 1, the prior encoder γ 2, and / or the decoder θ5 are modified. The training process continues until the first and / or the second optimization criterion is met.

[0046] The in Fig. The procedure shown and described above essentially corresponds to the known training of a CVAE.

[0047] The configuration of a CVAE model according to the invention during training is described below in conjunction with Fig. 2 explained in more detail. Fig. 2 builds on the content Fig. 1 on. Therefore, in the Fig. 1 and Fig. 2. The same reference symbols are used for identical elements. For an explanation of these elements, please refer to the explanations regarding... Fig. 1 referred.

[0048] As with the preceding in connection with Fig. In the well-known training procedure described in Section 1, the label encoder ϕ 1 transforms the training data element, i.e., the ground truth trajectory Y* 11 together with the training scene-specific information as training context information X* 12, into parameters of a multivariate label probability distribution 3 of latent features. This is a Gaussian mixture distribution 3, which in the embodiment described here is only one component. N(μq,∑q) encompasses, since each Groundtruth trajectory Y* 11 only realizes one of the three possible behavior options of the vehicle 10.

[0049] In contrast to the known training method, the prior encoder γ 2 transforms the training context information X* (12) according to the invention into parameters of a Gaussian mixture distribution 30 with at least two components N(μpi,∑pi). In the embodiment described here, three components were used. N(μpi,∑pi) 31, 32 and 33 are specified for the Gaussian mixture distribution 30, as this corresponds to the number of possible and meaningful behavior options of the vehicle 10 in the training traffic scene 20. Accordingly, the multimodal distribution of the behavior options is taken into account according to the invention by using a Gaussian mixture distribution as the context probability distribution 30.

[0050] As with the known training procedure, an initial distance is determined between the label probability distribution 3 and the context probability distribution 30. For this purpose, the distances between the label probability distribution and the context probability distribution are first calculated. N(μq,∑q) 3 and the individual components N(μpi,∑pi) The partial distances of values ​​31, 32, and 33 of the context probability distribution 30 are determined based on the KL divergence. These partial distances are then weighted and summed, and compared with a predefined first optimization criterion.

[0051] Weighting allows for considering only a selection of partial distances when determining the first distance. In this case, the first distance can be determined as the distance between the label probability distribution 3 and the "most fitting" component 31 of the context probability distribution 30. The first optimization criterion serves to align the label probability distribution 3 and the "most fitting" component 31 of the context probability distribution 30 so that this component can then be assigned as closely as possible to the behavior option or behavior mode "Drive straight ahead".

[0052] As with the known training method, at least one label sample Z is used. ϕ The label probability distribution 3 is generated, which can happen randomly or deterministically. Based on the label sample Z ϕ 13. The decoder θ5 then generates at least one output trajectory Y 15.

[0053] Then a second distance between the Groundtruth trajectory Y* 11 and the generated output trajectory Y 15 is determined and compared with a predefined second optimization criterion.

[0054] If the first and / or the second optimization criterion is not met, at least one parameter of the label encoder ϕ 1, the prior encoder γ 2, and / or the decoder θ 5 is modified. The training process continues until a predefined degree of similarity is reached between the label probability distribution 3 and the "best-fitting" component 31 of the context probability distribution 30, and / or a predefined degree of similarity exists between the ground truth trajectory and the output trajectory.

[0055] As explained in detail in the introductory description, for each component N(μpi,∑pi) 31, 32, 33 of the context probability distribution p 30 a fashion probability w i learned by training the context probability distribution p 30 as a Gaussian mixture distribution p=∑i=1NwiN(μpi,∑pi).

[0056] Thus, the CVAE model trained according to the invention is not only able to generate predictions for different behavioral modes, but also to estimate the probability of occurrence of the individual predictions depending on the behavioral mode.

[0057] Fig. Figure 2 also illustrates the claimed computer-implemented system for predicting the behavior of at least one participant in a traffic scene and a corresponding method.

[0058] Such a system comprises a perception layer for aggregating scene-specific information about a traffic scene 20 from various information sources. These sources can include vehicle-integrated sensors, such as lidar sensors, radar sensors, and / or RGB cameras installed on the ego vehicle, or external sensors, such as lidar sensors, radar sensors, and / or RGB cameras installed in or on infrastructure elements. Other potential information sources include stored map information, as well as queryable weather and road condition information, traffic situation information, etc. The perception layer aggregates the information from the various sources and preprocesses it into contextual information.

[0059] The system further comprises a prior encoder γ 2 and a decoder θ 5 of a CVAE model, which, as above, in conjunction with Fig. 2 has been trained. Using the prior encoder γ 2, the scene-specific context information is mapped into the latent space by translating it into parameters of a Gaussian mixture distribution 30 with three components. N(μpi,∑pi) be transformed.

[0060] Furthermore, an evaluation and processing unit is provided, which is trained to process latent representations 14 of the traffic scene 20 in the form of context samples Z. γ To generate 14 of the Gaussian mixture distribution 30. Advantageously, the context samples Z γ i 14 each specifically from a selected component N(μpi,∑pi) The context probability distribution 30 is generated. The context samples Z can be used in this process. γ i 14 can be generated randomly or deterministically.

[0061] The context samples Z γ i14 serve as input for the decoder θ 5, which uses it to generate behavioral predictions in the form of trajectories Y. i generated for at least one participant in the traffic scene and based on the respective fashion probability w i also the probability of occurrence of the respective predicted trajectory Y i estimates. In doing so, the decoder θ 5 can additionally use the aggregated scene-specific information of traffic scene 20.

[0062] Advantageously, during inference, i.e., during prediction using the trained CVAE model, several context samples Z are used. γ i 14 for the individual components N(μpi,∑pi) 31, 32, 33 of the context probability distribution 30 and the trajectory predictions Y based on it i generated. The trajectory sets thus generated, or rather their distributions 61, 62 and 63, are in Fig.2 shown. If the individual components N(μpi,∑pi) If the context probability distributions 31, 32, and 33 can be assigned to 30 different behavioral options or behavioral modes, then the corresponding behavioral predictions Y can be determined based on the distributions 61, 62, and 63. i a measure of the uncertainty of predicted trajectories based on this component N(μpi,∑pi) 31, 32, 33 of the context probability distribution 30 can be derived. For this purpose, for example, the parameters of a parametric description of the distributions 61, 62 and 63, such as expected value and standard deviation, can be used. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Ghosh et al., “From variational to deterministic autoencoder,” https: / / arxiv.org / abs / 1903.12436

[0001]

Claims

[1] Computer-implemented method for training a model comprising a Conditional Variational Auto Encoder (CVAE) structure with a label encoder ϕ (1), a prior encoder γ (2) and a decoder θ (5), wherein a set of training data elements is provided for the training method, each comprising at least one ground truth information Y* (11) with associated training context information X* (12), and where the following steps are performed for each training data item: - Use of the label encoder ϕ (1) to transform the training data element into characteristics of a multivariate label probability distribution (3) of latent features, - Use of the prior encoder γ (2) to transform the training context information X* (12) into parameters of a multivariate context probability distribution (30) of latent features, - determining a first distance between the label probability distribution (3) and the context probability distribution (30) and comparing the first distance with a predetermined first optimization criterion, - Generate at least one label sample Z ϕ (13) the multivariate label probability distribution (3), - Use of the decoder θ (5) to generate at least one output information Y (15) based on the label sample Z ϕ (13), - determining a second distance between the ground truth information Y* (11) and the generated output information Y (15) and comparing the second distance with a predetermined second optimization criterion, - modifying at least one parameter of the label encoder ϕ (1), the prior encoder γ (2) and / or the decoder θ (5) if the first and / or the second optimization criterion is not met; characterized by , that the training data elements are in parameters of a Gaussian mixture distribution q (3) with at least one component N(μq,∑q) be transformed and that the training context information X* (12) is converted into parameters of a Gaussian mixture distribution p (30) with at least two components N(μpi,∑pi) (31, 32, 33) can be transformed. [2] Method according to claim 1, characterized by that the given number N of components (31, 32, 33) of the context probability distribution p (30) is greater than the given number M of components of the label probability distribution q (3). [3] Method according to one of claims 1 or 2, characterized by that the first distance between the label probability distribution q (3) and the context probability distribution p (30) is determined based on the Kullback-Leibler divergence or the Wasserstein distance. [4] Method according to one of claims 1 to 3, characterized by that for the determination of the first distance between the label probability distribution q (3) and the context probability distribution p (30) a partial distance between the individual components N(μq,∑q) the label probability distribution q (3) and the individual components N(μpi,∑pi) (31, 32, 33) of the context probability distribution p (30) is determined and the partial distances are then added up, in particular weighted. [5] Method according to one of claims 1 to 4, characterized by that for each component N(μpi,∑pi) (31, 32, 33) of the context probability distribution p (30) a mode probability w i is learned by training the context probability distribution p (30) as a Gaussian mixture distribution p=∑i=1NwiN(μpi,∑pi). [6] Method according to claim 5, characterized by that to learn the mode probabilities w i For each training data item, the following steps are performed: - Generate at least one context sample Z γ i (14) for each component N(μpi,∑pi) (31, 32, 33) of the context probability distribution p (30), - Use of the decoder θ (5) to generate at least one output information Y i for each context sample Z generated in this way γ i (14), - Determining a third distance between the output information Y generated in this way i and the ground truth information Y* (11) and comparing the third distance with a given third optimization criterion and - Modifying at least one parameter of the prior encoder γ (2) if the third optimization criterion is not met. [7] Method according to one of claims 1 to 6, characterized by that the label samples Z ϕ (13) and / or the context samples Z γ i (14) are generated randomly and / or deterministically. [8] Method according to one of claims 1 to 7, characterized by that the decoder θ (5) uses training context information X* (12) to generate at least one output information Y (15). [9] Method according to one of claims 1 to 8 for training a model for predicting a behavior of at least one participant in a traffic scene, in particular for predicting trajectories of at least one participant in a traffic scene, characterized bythat the training data elements each comprise at least one ground truth behavior as ground truth information and training scene-specific information as associated training context information and that the decoder θ (5) generates at least one behavior prediction as output information. [10] Method according to one of claims 1 to 8 for training a model for generating digital images as training data for a Machine Learning (ML) component, characterized by that the training data elements each comprise at least one ground truth image and image attributes as associated training context information and that the decoder generates at least one image as output information. [11] Computer-implemented method for predicting the behavior of at least one participant in a traffic scene using a model trained according to claim 9, comprising the following steps: a. Aggregating scene-specific information of a traffic scene (20), b. Using the pre-trained prior encoder γ (2) to transform the scene-specific information into parameters of a multivariate context probability distribution p (30) in the form of a Gaussian mixture distribution p (4) with at least two components N(μpi,∑pi) (31, 32, 33), c. Generate at least one context sample Z γ i (14) the context probability distribution p (30), d. Using the pre-trained decoder θ (5) to generate at least one behavioral prediction Y i based on at least one context sample Z γ i (14). [12] Method according to claim 11, characterized by that at least one context sample Z γ i (14) from at least one selected component N(μpi,∑pi) (31, 32, 33) of the context probability distribution p (30) is generated. [13] Method according to one of claims 12, characterized by that the behavioral prediction Y i weighted based on a mode probability w i that component N(μpi,∑pi) (31, 32, 33) of the context probability distribution p (30), from which the underlying context sample Z γ i (14) has been generated. [14] Method according to one of claims 12 or 13, characterized by that for at least one component N(μpi,∑pi) (31, 32, 33) the context probability distribution p (30) several context samples Z γ i (14) and behavioral predictions based on it Y i generated, and that based on the distribution of these behavioral predictions Y ia measure of the uncertainty of behavioral predictions based on this component N(μpi,∑pi) (31, 32, 33) of the context probability distribution p (30). [15] Method according to one of claims 11 to 14, characterized by that at least one context sample Z γ i (14) is randomly generated or is deterministically generated. [16] Method according to one of claims 11 to 15, characterized by that the pre-trained decoder θ (5) additionally uses the aggregated scene-specific information of the traffic scene (20) to generate a behavior for at least one participant of the traffic scene. [17] A computer-implemented method for generating digital images as training data for a machine learning (ML) component, wherein the digital images are generated using a model trained according to claim 10 and the method comprises at least the following steps: a. Providing image attributes, in particular in the form of a textual description of image content, b. Using the pre-trained prior encoder γ to transform the image attributes into parameters of a multivariate context probability distribution p in the form of a Gaussian mixture distribution p with at least two components N(μpi,∑pi) c. Generate at least one context sample Z γ i the context probability distribution p, d. Using the pre-trained decoder θ to generate at least one digital image Y i based on at least one context sample Z γi . [18] Computer-implemented system for predicting the behavior of at least one participant in a traffic scene, the system comprising at least the following components: a. a perception layer for aggregating scene-specific information of a traffic scene, b. a model trained according to claim 9 i. with a prior encoder γ (2) for transforming the scene-specific information into parameters of a multivariate context probability distribution p (30) in the form of a Gaussian mixture distribution p (30) with at least two components N(μpi,∑pi) (31, 32, 33) and ii. a decoder θ (5) for generating a behavior prediction Y i for at least one participant in the traffic scene; c. an evaluation and computing unit designed to process at least one context sample Z γ i(14) from the context probability distribution p (30) as input for the decoder θ (5). [19] Computer-implemented system for generating digital images as training data for a machine learning (ML) component, the system comprising at least the following components: a provisioning facility for image attributes, b. a model trained according to claim 10 i. with a prior encoder γ for transforming the image attributes into parameters of a multivariate context probability distribution p in the form of a Gaussian mixture distribution p with at least two components N(μpi,∑pi) and ii. with a decoder θ for generating at least one digital image Y i ; c. an evaluation and computing unit designed to process at least one context sample Z γ ifrom the context probability distribution p as input for the decoder θ.