Perception-based behavior of autonomous systems

Abstract

A first aspect of the present disclosure relates to a computer-implemented method for training a machine learning model to evaluate behavior of an autonomous system configured to address perceptual tasks, comprising: receiving, by a perceptual system of the autonomous system, sensor data; receiving metadata, wherein an impact on the resolvability of the perceptual task is encoded by the metadata; determining an error probability based on the received sensor data; providing a machine learning model, the machine learning model being designed to map the sensor data and the metadata to an error probability to resolve the perceptual task; a machine learning model is trained based on training dataset elements including the received sensor data, the received metadata, and the determined error probability. A second aspect of the present disclosure relates to a computer-implemented method for evaluating behavior of an autonomous system configured to address perceptual tasks.

Description

Background Art

[0001] An autonomous system based on at least one perception task may, for example, be a fully or partially automated vehicle and / or robot. The at least one perception task to be solved may include, for example, object recognition. Alternatively or additionally, the at least one perception task to be solved may include, for example, semantic segmentation and / or free space recognition. The sensor data received by the perception system of the autonomous system may include, for example, image data (video). Alternatively or additionally, the sensor data received by the perception system of the autonomous system may include radar data, LiDAR data, ultrasonic data, motion data, acoustic data and / or thermal data.

[0002] With the goal of providing reliable and safe autonomous systems, it is desirable to be able to evaluate the behavior of autonomous systems as best as possible already during development. However, since autonomous systems are precisely also supposed to be used in open environments, such evaluation remains a major challenge. In addition to test runs (of prototypes or previous development projects), simulations are increasingly being used to address this problem. Summary of the invention

[0003] A first general aspect of the present disclosure relates to a computer-implemented method for training a machine learning model to evaluate the behavior of an autonomous system, the autonomous system being configured to solve a perception task, which perception task includes, for example, object recognition - in particular, being configured to determine (before evaluating the behavior of the autonomous system) an error probability in solving the perception task (e.g., object recognition).

[0004] The method includes receiving sensor data by a perception system (eg, simulated or real) of an autonomous system.

[0005] The method also includes receiving metadata, wherein an impact on the solvability of the perception task (eg, an impact on the identifiability of the object) is encoded by the metadata.

[0006] The method also includes determining an error probability for solving a perception task (eg, object recognition) based on the received sensor data.

[0007] The method also includes providing a machine learning model designed to map (at least) sensor data and metadata to (at least) error probabilities for solving a perception task (e.g., object recognition).

[0008] The method also includes training a machine learning model based on training dataset elements, the training dataset elements including the received sensor data, the received metadata, and the determined error probabilities.

[0009] A second general aspect of the present disclosure relates to a computer-implemented method for evaluating the behavior of an autonomous system configured to solve a perception task, including, for example, object recognition.

[0010] The method includes simulating a perception system of an autonomous system.

[0011] The method includes simulating an environment of a perception system.

[0012] The method includes providing a machine learning model designed and trained to map (at least) sensor data and metadata to (at least) error probabilities in solving a perception task (e.g., object recognition), wherein effects on the solvability of the perception task (e.g., effects on the identifiability of the object) are encoded by the metadata.

[0013] The method also includes receiving sensor data and metadata.

[0014] The method also includes applying a machine learning model to the received sensor data and metadata, wherein an error probability is determined.

[0015] The method also includes evaluating behavior of the autonomous system based on the error probability, wherein the evaluation is generated.

[0016] A third general aspect of the present disclosure relates to a computer system designed to perform a computer-implemented method for training a machine learning model to evaluate the behavior of an autonomous system according to the first general aspect (or its embodiments), wherein the autonomous system is configured to solve a perception task, which perception task includes, for example, object recognition, and / or to perform a computer-implemented method for evaluating the behavior of an autonomous system according to the second general aspect (or its embodiments), wherein the autonomous system is configured to solve a perception task, which perception task includes, for example, object recognition.

[0017] A fourth general aspect of the present disclosure relates to a computer program designed to perform a computer-implemented method for training a machine learning model to evaluate the behavior of an autonomous system according to the first general aspect (or its embodiments), wherein the autonomous system is configured to solve a perception task, which perception task includes, for example, object recognition, and / or to perform a computer-implemented method for evaluating the behavior of an autonomous system according to the second general aspect (or its embodiments), wherein the autonomous system is configured to solve a perception task, which perception task includes, for example, object recognition.

[0018] A fifth general aspect of the present disclosure relates to a computer-readable medium or a signal storing and / or containing the computer program according to the fourth general aspect.

[0019] The methods proposed in the present disclosure according to the first or second general aspects (or embodiments thereof) are intended to enable a better evaluation of the behavior of an autonomous system, particularly already during development. In particular, these methods are used to assess the influence of errors in perception on autonomous (driving) functions. If necessary and based on this, the autonomous system can be improved, particularly already during development.

[0020] The method proposed in the present disclosure is particularly intended to provide an error model for a single perception system of an autonomous system. To this end, the error probability is calculated at least based on the sensor data and metadata, respectively. The metadata includes, for example, the degree of concealment of the object to be identified, the brightness value and / or the distance value of the object to be identified, and thus encodes the influence on the solvability of the perception task (for example, the recognizability of the object). In particular, the error probability therefore depends not only on the metadata, but also on the sensor data. As a result, the scene recorded by the perception system can be better distinguished and evaluated.

[0021] For example, in the case of object recognition, it can be the recognition of pedestrians. Such recognition is usually difficult due to a high degree of concealment. However, in some cases, if the characteristic parts of the pedestrian (such as arms and / or hands) are still visible to the perception system despite the high degree of concealment, the pedestrian can be correctly recognized. In this case, it is not necessary to output a high error probability just because of the high degree of concealment (which may be the case if the error probability depends only on metadata), but the error probability can be smaller due to the characteristic parts of the pedestrian that are still well enough to be recognized. The difficulty of solving a given perception task can thus be better quantified. A high error probability can thus be output, especially when there are real problems during the solution of the perception task. Thus, real problem cases can be identified faster and better when solving the perception task, and then the autonomous system can be improved in a targeted manner (for example, by adding additional sensors with other perspectives, other sensor types, adding sensor data from the immediately preceding moment, etc.). Otherwise, (unnecessary) adaptation of the autonomous system due to false problems in which the perception task can still be satisfactorily solved can be avoided.

[0022] The method proposed in the present disclosure according to the second general aspect (or its embodiments) enables a closed-loop simulation, which includes a simulation of the autonomous system, including its perception system, and a simulation of the environment of the perception system (e.g. including objects to be recognized). For example, such a closed-loop simulation can show which scenes (also called sequences) that have not yet been labeled have a significant impact on the overall system. Thereby, sensor data relevant to the system behavior can be selected from a large amount of sensor data and, for example, labeled. In addition, as described above, the autonomous system can be improved.

[0023] The goal of the closed-loop simulation can be to determine the impact of the errors of the perception system on the behavior of the entire system. However, the perception system should not focus on the errors of the specific implementation of the algorithm for solving the perception task, but on the errors caused by the specific task and the environment to be perceived and the associated sensor data. In one embodiment of the method according to the second aspect, this can be achieved, for example, in particular by using a large number of additional machine learning models, each of which is designed and trained to map sensor data to a solution to the perception task, in particular to object detection.

[0024] In closed-loop simulations, temporally coupled errors are of particular interest. In the method proposed in the present disclosure, temporal coupling via sensor data is also considered. Since the error model specifically observes the sensor content, the error coupling of the perception model can also be well mapped by the temporal coupling of the sensor data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1a -d schematically illustrates a computer-implemented method for training a machine learning model to evaluate the behavior of an autonomous system configured to solve a perception task (the perception task particularly includes object recognition).

[0026] Figure 2 A computer-implemented method for evaluating the behavior of an autonomous system configured to solve a perception task, which in particular comprises object recognition, is schematically illustrated. DETAILED DESCRIPTION

[0027] The method proposed in the present disclosure aims to enable better evaluation of the behavior of autonomous systems, especially already during development. First, a method for training a machine learning model is disclosed, and then a method for applying the machine learning model is disclosed.

[0028] First published Figure 1a A computer-implemented method 100 schematically shown in Figure 1 is for training a machine learning model to evaluate the behavior of an autonomous system configured to solve a perception task. The perception task may in particular include a task of object recognition, i.e. the perception task or a part thereof may be solved, for example, by recognizing an object such as a pedestrian. Alternatively, the method 100 may be designed to determine the error probability of solving the perception task (in particular object recognition) before evaluating the behavior of the autonomous system.

[0029] The method 100 includes receiving 130 sensor data by a perception system of an autonomous system. The autonomous system may be, for example, a vehicle that is fully or partially autonomously driven. Alternatively or additionally, the autonomous system may be a robot. The perception system may be simulated or real. In the case of a real perception system, receiving 130 sensor data may include measuring sensor data. The perception system may include one or more sensor systems, each sensor system including one or more sensors (e.g., LIDAR, RADAR, camera, sound, ultrasound, etc.).

[0030] The method 100 also includes receiving 140 metadata, wherein the influence on the solvability of the perception task is encoded by the metadata. For example, in the case of object recognition, the metadata can encode the recognizability of the object to be recognized. The metadata can be based on super knowledge, in particular on the simulation parts 110 and 120 (see below), that is, on simulator knowledge. Alternatively or additionally, the metadata can be determined from an upstream process, for example based on sensor data. The metadata does not necessarily include information about the solvability of the perception task to be solved. Instead, the metadata includes factors that influence the solvability of the perception task. In method 200, the solvability is quantified in the form of an error probability output by the machine learning model.

[0031] The metadata may include, for example, the degree of concealment of the object to be identified. Alternatively or additionally, the metadata may include, for example, a brightness value. Alternatively or additionally, the metadata may include, for example, a distance value of the object to be identified. Depending on the perception system and the perception task to be solved, other parameters may also be considered as metadata.

[0032] The method 100 further comprises determining 150 an error probability of solving a perception task (eg object recognition) based on the sensor data received 130. The error probability may be viewed as a measure of the difficulty (solvability) of the perception task to be solved.

[0033] The method 100 further comprises providing 160 (or receiving) a machine learning model designed to map at least the sensor data and metadata to at least an error probability for solving a perception task, in particular to an error probability for object recognition. The machine learning model may, but need not, be trained in advance. The machine learning model may comprise or be, for example, an artificial neural network, in particular a deep artificial neural network.

[0034] The method 100 also includes training 170 a machine learning model based on training dataset elements including the received 130 sensor data, the received 140 metadata, and the determined 150 error probabilities.

[0035] and Figure 1aUnlike what is schematically shown in FIG. 1 , steps 130 , 140 , and 150 may be performed in any order.

[0036] The method 100 may include simulating 110 a perception system of an autonomous system, such as Figure 1b -d is shown schematically.

[0037] Alternatively or additionally, Figure 1b As schematically shown in Figure 1-d, the method 100 may include simulating 120 an environment of the perception system, in particular an object to be identified in the environment.

[0038] Steps 110 and / or 120 are not necessary in method 100, for example, because real measured sensor data, ie, for example, from a test drive, are already available. Such measured sensor data can be provided with metadata, for example, by (automatic) labeling.

[0039] In method 100, sensor data that has been measured in whole or in part by a perception system may be received 130. Associated metadata may be generated mechanically (eg, via a fourth trained machine learning model) and / or manually based on the sensor data.

[0040] Determining 150 the error probability based on the received 130 sensor data may be as follows: Figure 1c As schematically shown in , a large number of additional machine learning models are included in application 151, each of which is designed and trained to map sensor data into a solution to a perception task, in particular to object detection, wherein each application 151 leads to a perception result, in particular an object detection result, in particular a perception success or a perception failure. The large number of additional machine learning models can be replaced by another machine learning model here. However, alternatively, a large number of additional machine learning models have proven to be advantageous because the effects of the individual additional machine learning models on the error probability can be averaged out. To this end, the additional machine learning models can be different. In particular, they may be so different that no two additional machine learning models are the same.

[0041] Then, also Figure 1c Schematically, determining 150 an error probability based on the sensor data received 130 may include determining 152 an error probability based on a perception result, in particular an object detection result, in particular based on a perception success and a perception failure. For example, the error probability may be determined based on a ratio of the number of perception successes to the total number of perception successes and perception failures.

[0042] In such Figure 1dIn other embodiments schematically illustrated, determining 150 an error probability based on sensor data received 130 may include applying 153 a third machine learning model designed and trained to map the sensor data and optional metadata to an error probability for solving a perception task, wherein the error probability is determined 150.

[0043] like Figure 1b As schematically shown, method 100 may include training 171 a machine learning model based on a training data set including a large number of training data set elements. Each training data set element may be created according to the steps of method 100. The number of training data set elements should be as large as possible to obtain a convincing trained machine learning model.

[0044] Also disclosed is a computer-implemented method 200, such as Figure 2 Schematically illustrated in FIG. 1 is a diagram for evaluating the behavior of an autonomous system configured to solve a perception task. The perception task may in particular include a task of recognizing an object, i.e. the perception task or a part thereof may be solved, for example, by recognizing an object such as a pedestrian.

[0045] Method 200 includes simulating 210 a perception system of an autonomous system.

[0046] The method 200 also includes simulating 220 an environment of the perception system.

[0047] Due to the simulation steps 210 and 220, the method can be performed frequently and is particularly suitable for Monte Carlo analysis.

[0048] The method 200 further comprises providing 230 (or receiving) a machine learning model designed and trained to map at least the sensor data and the metadata at least to an error probability for solving the perception task, in particular to an error probability for object recognition. As already described in connection with the method 100, the influence on the solvability of the perception task, in particular the influence on the identifiability of the object, is also encoded by the metadata. The machine learning model may have been trained 170, 171 according to the method 100.

[0049] The method 200 further comprises receiving 240 sensor data and metadata. The metadata may be based, for example, on super knowledge, in particular on the simulation parts 210 and 220, i.e. on simulator knowledge. Alternatively or additionally, the metadata may be determined from an upstream process, for example, based on the sensor data. The metadata does not necessarily include information about the solvability of the perception task to be solved. Rather, the metadata includes factors that influence the solvability of the perception task. In the method 200, the solvability is only quantified in step 240 in the form of an error probability output by the machine learning model.

[0050] As already described, the metadata may include, for example, the degree of concealment of the object to be identified. Alternatively or additionally, the metadata may include, for example, a brightness value. Alternatively or additionally, the metadata may include, for example, a distance value of the object to be identified. Depending on the perception system and the perception task to be solved, other parameters may also be considered as metadata.

[0051] The method 200 also includes applying 240 a machine learning model to at least the received 240 sensor data and metadata, wherein at least one error probability is determined.

[0052] Method 200 also includes evaluating 260 the behavior of the autonomous system based on the error probability, wherein the evaluation is generated. Evaluation 260 may be performed over a period of time.

[0053] and Figure 2 Unlike the schematic illustration in FIG. 2 , steps 230 and 240 may be performed in any order or simultaneously. Steps 210 and 220 may, but need not, be performed simultaneously.

[0054] Evaluating 260 the behavior of the autonomous system may include rolling a dice for at least one event based on the error probability, wherein the at least one event is identified as an error event or a non-error event. For example, rolling the dice may be implemented by forming an empirical distribution function based on the error probability and evaluating the inverse of the empirical distribution function based on a (pseudo) random number on a true interval [0, 1].

[0055] Evaluating 260 the behavior of the autonomous system may specifically include (eg, as in a Monte Carlo simulation) rolling dice for a large number of events based on error probabilities.

[0056] Evaluating 260 the behavior of the autonomous system may be based on at least two events at different points in time. Evaluating 260 the behavior of the autonomous system may be based on a large number of events.

[0057] The method 200 may include adapting 270 a perception system of the autonomous system based on the evaluation. The perception system and / or the autonomous system may thereby be improved, in particular the reliability and safety of the autonomous system may be increased.

[0058] The perception task (more precisely: solving the perception task) may comprise, for example, object recognition in both methods 100, 200. Alternatively or additionally, the perception task (more precisely: solving the perception task) may comprise semantic segmentation. Alternatively or additionally, the perception task (more precisely: solving the perception task) may comprise free space recognition. Further perception tasks or combinations thereof are conceivable.

[0059] In both methods 100, 200, the machine learning model may consist of one or more sub-models. For example, the machine learning model may include a sub-model for near objects and a sub-model for far objects. In this case, the corresponding sub-model that should be trained 170, 171 or should be applied may be first selected, for example, based on metadata and / or based on further input variables to calculate 250 the error probability.

[0060] Furthermore, a computer system is disclosed, which is designed to perform a computer-implemented method 100 for training a machine learning model to evaluate the behavior of an autonomous system, the autonomous system being configured to solve a perception task (the perception task particularly comprising object recognition). Alternatively or additionally, the computer system may be designed to perform a computer-implemented method 200 for evaluating the behavior of an autonomous system, the autonomous system being configured to solve a perception task (the perception task particularly comprising object recognition). The computer system may include a processor and / or a working memory.

[0061] Also disclosed is a computer program designed to perform a computer-implemented method 100 for training a machine learning model to evaluate the behavior of an autonomous system, the autonomous system being configured to solve a perception task (the perception task particularly comprising object recognition). Alternatively or additionally, the computer program may be designed to perform a computer-implemented method 200 for evaluating the behavior of an autonomous system, the autonomous system being configured to solve a perception task (the perception task particularly comprising object recognition). The computer program may, for example, be present interpretably or in compiled form. The computer program may be loaded (also in parts) into the RAM of a computer for execution, for example as a sequence of bits or bytes.

[0062] A computer readable medium or signal storing and / or containing a computer program is also disclosed. The medium may include, for example, one of RAM, ROM, EPROM, HDD, SDD, ... on which / in which the signal is stored.

Claims

1. A computer-implemented method (100) for training a machine learning model to evaluate the behavior of an autonomous system, the autonomous system being configured to solve a perception task, the perception task particularly comprising object recognition, the method comprising: - receiving (130) sensor data via a perception system of the autonomous system; - receiving (140) metadata, wherein an influence on the solvability of the perception task, in particular on the recognizability of the object, is encoded by the metadata; - determining (150) an error probability for solving the perception task, such as an error probability for object recognition, based on the received (130) sensor data; - providing (160) the machine learning model, the machine learning model being designed to map sensor data and metadata to error probabilities for solving the perception task, in particular to error probabilities for object recognition; -Training (170) the machine learning model based on training data set elements, the training data set elements including the received (130) sensor data, the received (140) metadata and the determined (150) error probabilities.

2. The method (100) according to claim 1, comprising: - simulating (110) a perception system of the autonomous system; - Simulating (120) the environment of the perception system, in particular the objects to be recognized in the environment.

3. The method (100) of claim 1, wherein the sensor data is measured by the perception system; and The metadata is generated mechanically and / or manually based on the sensor data.

4. The method (100) according to any one of the preceding claims, wherein determining (150) an error probability based on the received (130) sensor data comprises: - applying (151) a plurality of further machine learning models, each further machine learning model being designed and trained to map the sensor data into a solution to the perception task, in particular into object detection, wherein each application (151) results in a perception result, in particular an object detection result, in particular a perception success or a perception failure; - determining (152) the error probability based on the perception result, in particular based on the object detection result, in particular based on perception success and perception failure.

5. The method (100) of claim 4, wherein the additional machine learning model is different.

6. The method (100) according to any one of claims 1 to 3, wherein determining (150) an error probability based on the received (130) sensor data comprises: - Applying (153) a third machine learning model designed and trained to map sensor data and optional metadata to error probabilities for solving the perception task, wherein the error probabilities are determined (150).

7. The method (100) according to any one of the preceding claims, comprising: -Training (171) the machine learning model based on a training dataset comprising a large number of training dataset elements.

8. A computer-implemented method (200) for evaluating the behavior of an autonomous system, the autonomous system being configured to solve a perception task, the perception task comprising, in particular, object recognition; the method comprising: - simulating (210) a perception system of the autonomous system; - simulating (220) the environment of the perception system; - providing (230) a machine learning model designed and trained to map sensor data and metadata to error probabilities for solving the perception task, in particular to error probabilities for object recognition, wherein the impact on the solvability of the perception task, in particular the impact on the identifiability of the object, is encoded by the metadata; - receiving (240) sensor data and metadata; - applying (250) the machine learning model to the received (240) sensor data and metadata, wherein error probabilities are determined; - evaluating (260) a behavior of the autonomous system based on the error probability, wherein an evaluation is caused.

9. The method (200) of claim 8, wherein evaluating (260) the behavior of the autonomous system comprises rolling a dice for at least one event based on an error probability, wherein the at least one event is identified as an error event or a non-error event.

10. The method (200) of claim 9, wherein evaluating (260) the behavior of the autonomous system is based on at least two events at different points in time.

11. The method (200) of claim 10, wherein evaluating (260) the behavior of the autonomous system is based on a plurality of events.

12. The method (200) according to any one of claims 8 to 11, comprising: - adapting (270) a perception system of the autonomous system based on the evaluation.

13. The method (200) according to any one of claims 8 to 12, wherein the machine learning model (170, 171) has been trained according to the method (100) according to any one of claims 1 to 7.

14. The method (100, 200) according to any one of the preceding claims, wherein the metadata comprises a concealment degree of the object to be identified, a brightness value and / or a distance value of the object to be identified.

15. The method (100, 200) according to any of the preceding claims, wherein the perception task comprises object recognition, semantic segmentation and / or free space recognition.

16. A computer system designed to perform: - a computer-implemented method (100) for training a machine learning model to evaluate the behavior of an autonomous system according to any one of claims 1 to 7 or 14 to 15, wherein the autonomous system is configured to solve a perception task, wherein the perception task in particular comprises object recognition; and / or - A computer-implemented method (200) for evaluating the behavior of an autonomous system according to any one of claims 8 to 15, the autonomous system being configured to solve a perception task, the perception task comprising in particular object recognition.

17. A computer program designed to perform: - a computer-implemented method (100) for training a machine learning model to evaluate the behavior of an autonomous system according to any one of claims 1 to 7 or 14 to 15, wherein the autonomous system is configured to solve a perception task, wherein the perception task in particular comprises object recognition; and / or - A computer-implemented method (200) for evaluating the behavior of an autonomous system according to any one of claims 8 to 15, the autonomous system being configured to solve a perception task, the perception task comprising in particular object recognition.

18. A computer readable medium or signal storing and / or containing a computer program according to claim 17.

Images