Methods for training an AI-based analysis module to analyze a traffic scene and methods for using this module

Abstract

According to various embodiments, a method for training an AI-based analysis module to analyze a traffic scene, as well as the associated analysis procedure, are described. The training procedure comprises: (a) encoding each training data element of a plurality of training data elements representing a respective training traffic scene; (b) for each representation in a latent space, the representations are clustered into at least one cluster; (c) using information about whether the respective training traffic scenes are critical or non-critical, each cluster is assigned a measure of the knowledge of the extent to which the cluster's representations represent critical traffic scenes, non-critical traffic scenes, or both critical and non-critical traffic scenes.The analysis procedure consists of: (a) coding a traffic scene into a representation using the previously trained ONN; (b) evaluating the criticality of the traffic scene by weighted combination of the measures assigned in the training, with each measure weighted according to the distance of the representation of the traffic scene to be evaluated from the respective cluster.

Description

[0001] The present disclosure relates to methods for training a CI-based analysis module for analyzing a traffic scene and methods for using such a module.

[0002] When controlling an at least partially automated vehicle in road traffic, it may happen that the vehicle finds itself in a critical situation (i.e., traffic scene) that was not anticipated at the design time of the respective control software. Human users typically have a gut feeling based on their experience, which leads them to drive particularly carefully in such situations, for example. To provide a similar mechanism for a vehicle control system, a machine learning model can be provided that evaluates a current traffic scene for its criticality. For this purpose, effective approaches that ensure a high level of control safety are desirable.

[0003] According to various embodiments, a method for training an AI-based analysis module for analyzing a traffic scene is provided, comprising • Encoding each training data item of a plurality of training data items representing a respective training traffic scene to a respective representation in a latent space, • Clustering the representations into at least one cluster and • Assigning, to each cluster, a measure of the knowledge of the extent to which the representations of the cluster represent critical traffic scenes, non-critical traffic scenes, or both critical and non-critical traffic scenes, using, for each training traffic scene, information as to whether the training traffic scene is critical or non-critical.

[0004] The method described above enables training of an analysis module for an effective criticality assessment of traffic scenes, whereby it delivers good results even if critical and non-critical traffic scenes cannot be (completely) separated in the latent state space (i.e., a cluster can contain both representations of critical and non-critical traffic scenes).

[0005] According to various embodiments, the criticality assessment by means of an analysis module trained according to the above method (e.g. online during control) of a respective control device enables three essential questions regarding the criticality assessment of the current traffic scene to be answered: • How close is the controlled system (e.g. vehicle and its environment) to a known non-critical traffic scene? • How close is the controlled system to a known critical traffic scene? • How well known is the current traffic scene overall?

[0006] The control system can then react appropriately, e.g. in cases of high uncertainty or high criticality, give priority to safe actions over unsafe actions (e.g. slow down, change to the right lane, stop a robot arm, etc.).

[0007] Various examples of implementation are given below.

[0008] Embodiment 1 is a method for training a Kl-based analysis module for analyzing a traffic scene as described above.

[0009] Embodiment 2 is the method according to embodiment 1, where the measure is a subjective logic opinion.

[0010] This makes it possible to take into account uncertainty regarding the knowledge as to whether the cluster's representations represent critical traffic scenes, non-critical traffic scenes, or both critical and non-critical traffic scenes.

[0011] Embodiment 3 is the method according to embodiment 1 or 2, wherein the training data elements are encoded by means of the encoder of an autoencoder structure.

[0012] The autoencoder structure is, for example, a VAE (variational auto encoder) or another type of autoencoder. The encoder can be trained so that the autoencoder's decoder can reconstruct the respective traffic scene. This ensures that the representation of the respective traffic scene adequately represents the traffic scene.

[0013] Embodiment 4 is the method according to any one of embodiments 1 to 3, wherein the plurality of training data elements represent both critical and non-critical traffic scenes and one or more of the clusters comprise representations of both critical and non-critical training traffic scenes.

[0014] The training traffic scenes therefore do not need to be strictly separated into critical and non-critical traffic scenes (in the latent space of representations). A criticality assessment is possible even if the representations of the training traffic scenes in the latent space are not separated by clustering.

[0015] Embodiment 5 is a method for analyzing a traffic scene, comprising encoding a traffic scene to be evaluated into a representation, evaluating the criticality of the traffic scene to be evaluated by weighted combination of the measures assigned according to embodiment 1, 2 or 4, wherein each measure is weighted according to the distance of the representation of the traffic scene to be evaluated to the respective cluster

[0016] Embodiment 6 is the method according to embodiment 5, wherein the determined measures are combined by the weighted combination to form a subjective logic opinion for the criticality of the traffic scene to be evaluated.

[0017] This allows not only the probability that a traffic scene is critical to be assessed (and, in response, for example, to drive carefully) but also the uncertainty about whether the traffic scene is critical or non-critical (and, for example, to drive carefully even in cases of high uncertainty).

[0018] Embodiment 7 is the method according to embodiment 6, wherein the determined measures are combined using the weighted average belief fusion operator.

[0019] This enables an effective weighted combination of subjective logic opinions.

[0020] Embodiment 8 is the method according to embodiment 5, wherein the measure for each training traffic scene is a Dempster-Shafer belief mass and the Dempster-Shafer belief masses are combined into a Dempster-Shafer belief mass for evaluating the criticality of the traffic scene to be evaluated.

[0021] Dempster-Shafer represents an alternative to the use of Subjective Logic, which can also provide measures to achieve appropriate (e.g. safe) control.

[0022] Embodiment 9 is the method according to one of the embodiments 5 to 8, wherein the coding of the training data elements and the traffic scene to be evaluated comprises the coding of respective input data containing sensor data relating to the respective traffic scene and / or an environment model for the respective traffic scene.

[0023] For example, camera images etc. can be used to describe the traffic scenes, or derived information such as object detections or trajectory predictions can be used as input data.

[0024] Embodiment 10 is the method according to one of the embodiments 1 to 9, wherein the training data elements and the traffic scene to be evaluated are encoded by means of the encoder of an autoencoder structure.

[0025] As explained above, the autoencoder structure can be, for example, a VAE (variational auto encoder) or another type of autoencoder. The encoder can be trained so that the autoencoder's decoder can reconstruct the respective traffic scene. This ensures that the representation of the respective traffic scene adequately represents the traffic scene.

[0026] Embodiment 11 is a method for planning the behavior of an at least partially automated vehicle, comprising detecting a traffic scene in which the vehicle is located, evaluating the traffic scene according to one of embodiments 5 to 10, and determining one or more control actions for the vehicle in the traffic scene depending on the evaluation result.

[0027] Embodiment 12 is a data processing system configured to perform a method according to any one of embodiments 1 to 11.

[0028] Embodiment 13 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 11.

[0029] Embodiment 14 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 11.

[0030] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Fig. 1 shows a vehicle. Fig. Figure 2 illustrates the training of a machine learning model and its use to evaluate a current traffic scene. Fig. 3 shows a flowchart illustrating a method for training a KI-based analysis module to analyze a traffic scene according to one embodiment.

[0031] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0032] Various examples are described in more detail below.

[0033] Fig. 1 shows a vehicle 101.

[0034] In the example of Fig. 1, a vehicle 101, for example a motor vehicle such as a car or truck, is provided with a vehicle control device (e.g. an Electronic Control Unit (ECU)) 102.

[0035] The vehicle control device 102 has data processing components, e.g., a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software 107 according to which the vehicle control device 102 operates, and data processed by the processor 103. The processor 103 executes the control software 107.

[0036] For example, the stored control software (computer program) includes instructions which, when executed by the processor, cause the processor 103 to perform driver assistance functions or even to control the vehicle autonomously.

[0037] The control software 107 is transferred, for example, from a computer system 105, e.g., via a network 106 (or also with the aid of a storage medium such as a memory card), to the vehicle 101. This can also occur during operation (or at least when the vehicle 101 is with the user), since the control software 107 is updated to new versions over time, for example.

[0038] The control software 107 can be trained, for example, using machine learning (ML). This means that the control software 107 implements one or more ML models 108 (or "machine learning model") that are trained based on training data, in this example from the computer system 105. The computer system 105 thus implements an ML training algorithm for training the one or more ML models 108.

[0039] The control software 107 determines control actions for the vehicle (such as steering actions, braking actions, etc.) from input data 109 that are available to it and that contain information about the environment or from which it derives information about the environment (such as by detecting other road users, e.g., other vehicles). This information includes, for example, sensor data such as from a vehicle camera or information obtained through communication with other vehicles or roadside devices.

[0040] In road traffic, it frequently happens that a vehicle finds itself in a critical (i.e., dangerous) traffic scene that was not anticipated during the design time of the control software, and especially during the training time of one or more ML models. Human drivers typically have a gut feeling based on their experience, which causes them to drive particularly carefully in such situations. Automated vehicles (i.e., control software), on the other hand, typically lack such a mechanism.

[0041] According to various embodiments, a mechanism is provided by which automated vehicles (or other agents, in particular other types of robotic devices) can react anticipatorily to unfamiliar situations. For example, the mechanism allows an automated vehicle to react to unknown or critical traffic scenes with a particularly cautious driving style. This can mitigate some unknown, critical traffic scenes (e.g., as defined by ISO 21448 Safety of the Intended Functionality), enabling safety that goes beyond the safety defined by ISO 21212 Functional Safety.

[0042] For this purpose, according to various embodiments, the information available to the vehicle (or other agent), ie the input data 109 for the control software 107 but also data derived therefrom (information about the environment), such as trajectory estimates for other road users or detected objects, ie perception information, a digital map, etc., are fed to an analysis module (of the control software 107), which evaluates the criticality of the respective traffic scene (e.g. traffic scene in the case of a vehicle) by means of a machine learning model trained for this purpose (e.g. the machine learning model 108), e.g. a neural network.

[0043] The result is passed on to a planning module (of the control software 107) where it influences the driving behavior so that the automated vehicle drives more carefully (and thus more proactively) the more critical the respective traffic scene is.

[0044] Multiple analysis modules can be provided in parallel, which prepare the perception information (also referred to as perception information) for the planning module. The perception information can be preprocessed for the analysis module. For example, lane detection can support the consistency check between map and vehicle perception. The planning module can have several additional inputs in addition to the determined criticality.

[0045] Fig. 2 illustrates the training of a machine learning model (here in particular comprising the encoder 201 of a VAE (Variational Auto-Encoder) 202 and a subsequent clustering) and its use (e.g., “online” during control) for evaluating a current traffic scene 204.

[0046] The basis for the training is a training data set 203, which has non-critical traffic scenes 205 (e.g. traffic scenes without an accident) and critical traffic scenes 206 (e.g. traffic scenes with an accident) as well as a respective label (“critical” or “non-critical”) for each traffic scene.

[0047] The encoder 201 of the variational autoencoder 202 (or another neural network for coding) is used to create a compact representation in a latent state space 207 for each of the traffic scenes 205, 206. This neural network (here the encoder 201) is also referred to below as an operational neural network (ONN). For the determined representations (represented as points in the latent state space 207), clustering is then carried out, e.g., using the DBSCAN method. The result is several clusters 208, each of which contains both representations of non-critical traffic scenes 205 (in Fig. 2 shown as filled circles) as well as critical traffic scenes 206 (in Fig. 2 (shown as unfilled circles) (in particular, the critical / non-critical label is neglected during clustering). If a variational auto encoder is used, the Mahalanobis distance, for example, can be used as a distance measure for clustering.

[0048] For each cluster 208, a subjective logic opinion is then calculated based on the labels (critical / non-critical) of the traffic scenes that the cluster contains, regarding the criticality of traffic scenes whose representations are located in the cluster.

[0049] At runtime (i.e., for inference), information about the current traffic scene 204 (e.g., a current environment model together with a history of the movements of objects in the environment of an ego vehicle) is used as input for the encoder (generally the ONN) 201. The encoder 201 transforms this input into a representation 209 in the latent state space 207 for the current traffic scene. Using a suitable distance measure, e.g., the Mahalanobis distance, in the latent space, the similarity of the current situation to the clusters 208 determined for the training traffic scenes 205, 206 is then determined (i.e., in each case as the distance of the representation 209 for the current traffic scene to the respective cluster 208). This similarity measure is then used as the respective weight in the Weighted Average Belief Fusion Operator from Subjective Logic to combine the individual Subjective Logic Opinions of the clusters into an overall Subject Logic Opinion ω tfor the representation 209 of the current traffic scene 204. The overall subject logic opinion is used (e.g., by the planning module) as a criticality measure, which evaluates both how familiar the current traffic scene is and how critical the current traffic scene is assessed to be. For example, depending on this criticality, the automated vehicle (e.g., the control device 102) adapts its driving behavior and, for example, drives more cautiously, e.g., more conservatively, the more critical the current situation appears.

[0050] As mentioned above, the ONN 201 can be the encoder of a VAE or the encoder of another auto-encoder. The input data indicating a respective traffic scene 204, 205, 206 can, for example, be encoded (at least partially) in the form of images and then processed using convolutional layers comprised by the ONN 201. The input data indicating a respective traffic scene 204, 205, 206 can also be represented, at least partially, in the form of a respective graph, and the ONN 201 in this case is or comprises a GNN (Graph Neural Network). Clustering in the latent space 207 can be performed using any clustering algorithm, such as DBSCAN.

[0051] As described above, a subjective logic opinion, in particular a binomial subjective logic opinion, can be used as the criticality measure assigned to the clusters 208 and generated by the analysis module for the current traffic scene 204. This offers the advantage that, even based on only a few data points in a respective cluster 208, it provides both a statement about the criticality and the statistical uncertainty within the criticality. Furthermore, various fusion operators, such as the weighted belief fusion operator, allow the subjective logic opinions of the clusters 208 to be combined into an overall criticality (i.e., an overall subject logic opinion for the current traffic scene 204). The weighting is carried out via the distance (e.g. Mahalanobis distance) of the online obtained data point, ie the representation 209 of the current traffic scene 204 in the latent space 207, to the clusters 208.In addition, prior knowledge or expert knowledge can be incorporated into the modeling of the prior of subjective logic opinions. As an alternative to subjective logic, belief masses from Dempster-Shafer theory or classical probabilities can also be used.

[0052] The Subjective Logic Opinion ω(b, d, u, a) of a cluster 208 can be calculated, for example, as follows: b=Nuncritical(Ci)N(Ci)+W, d=Ncritical(Ci)N(Ci)+W, u=WN(Ci)+W, a[aprior,1−aprior], where N unkritisch (C i ) the number of non-critical scenarios in cluster C i Accordingly, N kritisch (C i ) the number of critical scenarios in C i The constant known in Subjective Logic as W is set to 2, since there are two elementary events. In the simplest case, a uniform distribution is assumed for the statistical prior a.

[0053] The overall subjective logic opinion for the representation 209 of the current traffic scene 204 can then be calculated as follows: ωA ⊕˜cA⋅cBωB={bA⊕˜B=bAcAuB+bBcB″AcB″A+cA″BuA⊕˜ B=(cA+cB)′​′A″BcB″A+cA″BaA⊕˜B=aAcA+aBcBcA+cB

[0054] Here, ω A =(b A , d A , etc., a A ) and ω B =(b B , d B , and B , a B ) the Subjective Logic Opinions of the two clusters and c A and c B the respective weighting, e.g. the inverse of the Mahalanobis distance (or another distance measure) of the representation 209 of the current traffic scene 204 to cluster A or to cluster B.

[0055] If there are more clusters, the sum can be continued (i.e., the subjective logic opinions can be successively added to the result of the above formula using the operator ⊕̃). The weights can be adjusted appropriately, but all weights can also be calculated in advance.

[0056] When modeling the subjective logic opinion, more complex models are possible with additional knowledge. For example, the number of critical or non-critical traffic scenes in a cluster can also be weighted into the subjective logic opinion. This can be useful, for example, if correlations between traffic scenes are known, the distorting effect of which should be compensated. In addition, the a priori distributions (i.e., a prior ) for the clusters can be adapted using appropriate prior knowledge (if necessary, individually).

[0057] In summary, according to various embodiments, a method is provided as described in Fig. 3 shown.

[0058] Fig. 3 shows a flowchart 300 illustrating a method for training an AI-based analysis module to analyze a traffic scene according to one embodiment.

[0059] In 301, each training data item of a plurality of training data items representing a respective training traffic scene is encoded into a respective representation in a latent space.

[0060] In 302, the representations (of the traffic scenes from 301 represented by the training data elements) are clustered (i.e., grouped) into at least one cluster.

[0061] In 303, each cluster is assigned a measure of the knowledge of the extent to which the cluster's representations represent critical traffic scenes, non-critical traffic scenes, or both critical and non-critical traffic scenes. This is done using, for each training traffic scene, information indicating whether the training traffic scene is critical or non-critical.

[0062] The procedure of Fig.3 can be performed by one or more computers having one or more data processing units. The term “data processing unit” can be understood as any type of entity that enables the processing of data or signals. The data or signals can, for example, be treated according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit can comprise or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA) integrated circuit, or any combination thereof.Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0063] According to various embodiments, the method is therefore particularly computer-implemented.

[0064] Various embodiments may receive and use sensor signals from various sensors such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc. as the information about an environment or as a basis for the information about the environment, for example, to obtain sensor data regarding states of the respective system (vehicle and object or objects in the environment) and configurations and scenarios (i.e., about the control situations). The sensor data may be processed to determine the information about an environment. This may include classifying the sensor data or performing semantic segmentation on the sensor data, for example, to detect the presence of objects (in the environment in which the sensor data was obtained). Embodiments may be used to train a machine learning system and control a robot, e.g.autonomously by robot manipulators to achieve various manipulation tasks under different scenarios. In particular, embodiments are applicable to controlling and monitoring the execution of manipulation tasks, e.g., in assembly lines.

Claims

[1] A method for training a Kl-based analysis module for analyzing a traffic scene, comprising: encoding each training data item (205, 206) of a plurality of training data items representing a respective training traffic scene to a respective representation in a latent space (207); and Clustering the representations into at least one cluster (208), Assigning, to each cluster (208), a measure of the knowledge of the extent to which the representations of the cluster (208) represent critical traffic scenes, non-critical traffic scenes or both critical and non-critical traffic scenes, using, for each training traffic scene (205, 206), information as to whether the training traffic scene (205, 206) is critical or non-critical; [2] The method of claim 1, wherein the measure is a subjective logic opinion. [3] Method according to claim 1 or 2, wherein the training data elements (205, 206) are encoded by means of the encoder (201) of an autoencoder structure (202). [4] The method of any one of claims 1 to 3, wherein the plurality of training data elements (205, 206) represent both critical and non-critical traffic scenes and one or more of the clusters (208) comprise representations of both critical and non-critical training traffic scenes. [5] A method for analyzing a traffic scene, comprising: coding a traffic scene (204) to be evaluated into a representation; Assessing the criticality of the traffic scene (204) to be assessed by weighted combination of the measures assigned according to claim 1, 2 or 4, wherein each measure is weighted according to the distance of the representation of the traffic scene (204) to be assessed to the respective cluster (208) [6] Method according to claim 5, wherein the determined measures are combined by the weighted combination to form a subjective logic opinion for the criticality of the traffic scene (204) to be evaluated. [7] Method according to claim 6, wherein the determined measures are combined by means of the weighted average belief fusion operator. [8] The method of claim 5, wherein the measure for each training traffic scene (205, 206) is a Dempster-Shafer belief mass, and the Dempster-Shafer belief masses are combined into a Dempster-Shafer belief mass for evaluating the criticality of the traffic scene (204) to be evaluated. [9] Method according to one of claims 5 to 8, wherein the coding of the training data elements (205, 206) and the traffic scene (204) to be evaluated comprises coding respective input data containing sensor data relating to the respective traffic scene (204, 205, 206) and / or an environment model for the respective traffic scene (204, 205, 206). [10] Method according to one of claims 5 to 9, wherein the training data elements (205, 206) and the traffic scene to be evaluated (204) are encoded by means of the encoder (201) of an autoencoder structure (202). [11] Method for planning the behavior of an at least partially automated vehicle, comprising: detecting a traffic scene (204) in which the vehicle is located; Evaluating the traffic scene (204) according to one of claims 5 to 10; and Determining one or more control actions for the vehicle (101) in the traffic scene (204) depending on the evaluation result. [12] Data processing system (102) configured to carry out a method according to any one of claims 1 to 11. [13] A computer program comprising instructions which, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 11. [14] A computer-readable medium storing instructions that, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 11.

Images