Behavior of autonomous perception-based system

Abstract

To provide, according to a first aspect of the present disclosure, a computer-implemented method for training a machine learning model for evaluating a behavior of an autonomous system that is configured to solve a perception task.SOLUTION: This method includes: receiving sensor data by a perception system of an autonomous system; receiving metadata that encodes an influence on a solvability of a perception task; ascertaining an error probability based on the received sensor data; providing a machine learning model that is configured to map sensor data and metadata to the error probability for solving the perception task; and training the machine learning model based on a training data set element that includes the received sensor data, the received metadata and the ascertained error probability. A second aspect of the present disclosure relates to a computer-implemented method for evaluating a behavior of an autonomous system that is configured to solve a perception task.SELECTED DRAWING: Figure 1a

Description

[Technical field]

[0001] Prior Art The autonomous system based on the at least one perception task may be, for example, a vehicle and / or a robot running fully autonomously or partially autonomously. The at least one perception task to be solved may include, for example, object recognition. Alternatively or additionally, the at least one perception task to be solved may include, for example, semantic segmentation and / or free space perception. The sensor data received by the perception system of the autonomous system may include, for example, image data (video). Alternatively or additionally, the sensor data received by the perception system of the autonomous system may include radar data, LiDAR data, ultrasonic data, motion data, acoustic data, and / or thermal data. Summary of the Invention [Problem to be solved by the invention]

[0002] In view of the goal of providing reliable and reliable autonomous systems, it is desirable to be able to evaluate the behavior of an autonomous system as best as possible already during development. However, since autonomous systems are also to be used in open contexts, such evaluation remains a major challenge. Besides test runs (of prototypes or advanced development projects), simulation is increasingly being used to solve such problems. [Means for solving the problem]

[0003] Disclosure of the Invention A first general aspect of the present disclosure relates to a computer-implemented method for training a machine learning model for evaluating behavior of an autonomous system, the machine learning model being configured to solve a perception task including, for example, object recognition - and in particular to identify an error probability for solving the perception task (prior to evaluating the behavior of the autonomous system), for example, object recognition.

[0004] The method includes receiving sensor data by a perception system (eg, simulated or real) of the autonomous system.

[0005] The method further includes receiving metadata encoding an impact on solvability of the perceptual task, for example an impact on recognizability of the object.

[0006] The method further includes determining an error probability for solving a perception task, e.g., for object recognition, based on the received sensor data.

[0007] The method further includes providing a machine learning model configured to map (at least) the sensor data and metadata to (at least one) error probability for solving a perception task, e.g., for object recognition.

[0008] The method further includes training a machine learning model based on the training dataset members including the received sensor data, the received metadata, and the determined error probability.

[0009] A second general aspect of the present disclosure relates to a computer-implemented method for evaluating the behavior of an autonomous system configured to solve a perception task including, for example, object recognition.

[0010] The method includes simulating a perception system of the autonomous system.

[0011] The method includes simulating an environment surrounding the sensory system.

[0012] The method includes providing a machine learning model configured and trained to map (at least) sensor data and metadata to (at least one) error probability for solving a perception task, e.g. for object recognition, where the metadata encodes an impact on the solvability of the perception task, e.g. an impact on the recognizability of the object.

[0013] The method further includes receiving the sensor data and the metadata.

[0014] The method further includes applying a machine learning model to the received sensor data and metadata, where an error probability is determined.

[0015] The method further includes evaluating a behavior of the autonomous system based on the error probability, whereby an evaluation is obtained as a result.

[0016] A third general aspect of the present disclosure relates to a computer system configured to perform a computer-implemented method for training a machine learning model for assessing behavior of an autonomous system configured to solve a perception task, e.g., including object recognition, according to the first general aspect (or an embodiment thereof) and / or a computer-implemented method for assessing behavior of an autonomous system configured to solve a perception task, e.g., including object recognition, according to the second general aspect (or an embodiment thereof).

[0017] A fourth general aspect of the present disclosure relates to a computer program configured for performing a computer-implemented method for training a machine learning model for assessing a behavior of an autonomous system configured to solve a perception task, e.g. including object recognition, according to the first general aspect (or an embodiment thereof), and / or for performing a computer-implemented method for assessing a behavior of an autonomous system configured to solve a perception task, e.g. including object recognition, according to the second general aspect (or an embodiment thereof).

[0018] A fifth general aspect of the present disclosure relates to a computer-readable medium or signal storing and / or including a computer program according to the fourth general aspect.

[0019] The method according to the first or second general aspect (or an embodiment thereof) proposed in the present disclosure is aimed at making it possible to better evaluate the behavior of an autonomous system, in particular already during development. In particular, the method proposed in the present disclosure is used to assess the impact of errors in perception on the autonomous (driving) function. Depending on this, it is possible in some cases to improve the autonomous system, in particular already during development.

[0020] The method proposed in this disclosure aims to provide an error model, in particular for an individual perception system of an autonomous system. For this purpose, an error probability is calculated starting from at least sensor data and metadata, respectively. The metadata includes, for example, the degree of obscuration of the object to be recognized, the brightness value of the object to be recognized and / or the distance value of the object to be recognized, thereby encoding the influence on the solvability of the perception task, for example on the recognizability of the object. In particular, the error probability therefore depends not only on the metadata but also on the sensor data. This allows a better decomposition and evaluation of the scenarios recorded by the perception system.

[0021] In the case of object recognition, it may for example concern recognizing pedestrians. Such recognition of pedestrians is fundamentally made difficult by high obscuration. Nevertheless, in some circumstances, it is possible to accurately recognize pedestrians if their characteristic parts (e.g. arms and / or hands) are still visible to the perception system despite high obscuration. In such cases, it is not necessary to output a high error probability just because of high obscuration (as would be the case if the error probability only depended on metadata), but rather the error probability may be lower thanks to characteristic parts of the pedestrian that can still be recognized sufficiently well. This allows a better quantification of the difficulty of solving a given perception task. This allows a high error probability to be output, especially when there is a real problem in the solvability of the perception task. This allows a faster and better identification of the real problem in solving the perception task, and then allows the autonomous system to be improved as intended (e.g. by adding more sensors with other gaze angles, adding other types of sensors, adding sensor data from previous time points, etc.). Besides, it makes it possible to avoid (unnecessary) adaptations of the autonomous system caused by spurious problems, where the perception task could still be solved satisfactorily.

[0022] The method according to the second general aspect (or an embodiment thereof) proposed in the present disclosure allows a closed-loop simulation, including a simulation of the autonomous system, including its perception system, and a simulation of the surrounding environment of the perception system, including, for example, the object to be recognized. For example, such a closed-loop simulation can show which as yet unlabeled scenarios (also called sequences) have a significant impact on the entire system. This allows the selection and, for example, labeling of sensor data relevant to the system's behavior from a large number of sensor data. Furthermore, the autonomous system can be improved, as already explained above.

[0023] The goal of the closed-loop simulation can be considered to be the identification of the impact of errors of the perception system on the behavior of the whole system. However, in this case, the perception system should not focus on errors of a concrete implementation of an algorithm for solving a perception task, but on errors that arise based on the concrete task, the surrounding environment to be perceived, and the corresponding sensor data. In one embodiment of the method according to the second aspect, this can be achieved, for example, by using multiple further machine learning models, each configured and trained in particular to map sensor data to the solving of the perception task, in particular to the detection of objects.

[0024] In closed-loop simulation, the temporal coupling of errors is particularly important. In the method proposed in this disclosure, the temporal coupling via sensor data is taken into account together. Thanks to an error model that specifically considers the sensor content, the temporal coupling of sensor data also allows a good mapping of the error coupling of the perception model. [Brief description of the drawings]

[0025] [Figure 1a] FIG. 1 is a schematic diagram of a computer-implemented method for training a machine learning model for evaluating the behavior of an autonomous system configured to solve a perception task, particularly one that includes object recognition. [Figure 1b] FIG. 1 is a schematic diagram of a computer-implemented method for training a machine learning model for evaluating the behavior of an autonomous system configured to solve a perception task, particularly one that includes object recognition. [Figure 1c] FIG. 1 is a schematic diagram of a computer-implemented method for training a machine learning model for evaluating the behavior of an autonomous system configured to solve a perception task, particularly one that includes object recognition. [Figure 1d] FIG. 1 is a schematic diagram of a computer-implemented method for training a machine learning model for evaluating the behavior of an autonomous system configured to solve a perception task, particularly one that includes object recognition. [Diagram 2] FIG. 1 is a schematic diagram of a computer-implemented method for evaluating the behavior of an autonomous system configured to solve a perception task, particularly one that includes object recognition. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Detailed Description The method proposed in this disclosure aims to make it possible to better evaluate the behavior of autonomous systems, in particular already during development. Firstly, a method for training a machine learning model is disclosed, followed by a method for applying the machine learning model.

[0027] Firstly, a computer-implemented method 100 is disclosed, as shown in FIG. 1a, for training a machine learning model for evaluating the behavior of an autonomous system configured to solve a perception task. The perception task may in particular include a task of object recognition, i.e. an object, such as a pedestrian, may be recognized to solve the perception task or a part thereof. Alternatively, the method 100 may aim to determine an error probability for solving the perception task (in particular for object recognition) prior to evaluating the behavior of the autonomous system.

[0028] The method 100 includes receiving 130 the sensor data by a perception system of the autonomous system. The autonomous system may be, for example, a vehicle that travels fully autonomously or partially autonomously. Alternatively or additionally, the autonomous system may be a robot. The perception system may be simulated or real. In the case of a real perception system, receiving 130 the sensor data may include measuring the sensor data. The perception system may include one or more sensor systems, each including one or more sensors (e.g., LiDAR, radar, camera, sound, ultrasonic, etc.).

[0029] The method 100 further comprises receiving 140 metadata encoding an influence on the solvability of the perception task. For example, in case of object recognition, the metadata can encode the recognizability of the object to be recognized. The metadata can for example be based on super knowledge, in particular the simulation units 110 and 120 (see below), i.e. simulator knowledge. Alternatively or additionally, the metadata can for example be determined based on sensor data from an upstream procedure. The metadata does not necessarily include information on the solvability of the perception task to be solved. Rather, the metadata includes factors that have an influence on the solvability of the perception task. The solvability is quantified for the first time in the method 200 in the form of an error probability output by the machine learning model.

[0030] The metadata may for example comprise the obscuration degree of the object to be recognized. Alternatively or additionally, the metadata may for example comprise a luminance value. Alternatively or additionally, the metadata may for example comprise a distance value of the object to be recognized. Depending on the perception system and the perception task to be solved, other parameters may also be considered as metadata.

[0031] The method 100 further includes determining 150 an error probability for solving a perception task, e.g., for object recognition, based on the received 130 sensor data. The error probability may be considered as a measure for the difficulty (solvability) of the perception task to be solved.

[0032] The method 100 further includes providing (or receiving) 160 a machine learning model configured to map at least the sensor data and metadata to at least one error probability for solving a perception task, in particular for object recognition. The machine learning model may be pre-trained, but may also be non-pre-trained. The machine learning model may include or be, for example, an artificial neural network, in particular a deep artificial neural network.

[0033] The method 100 further includes training 170 a machine learning model based on training dataset elements including the received 130 sensor data, the received 140 metadata, and the determined 150 error probability.

[0034] Unlike what is shown diagrammatically in FIG. 1a, steps 130, 140, and 150 may be performed in any arbitrary order.

[0035] The method 100 may include simulating 110 a perception system of the autonomous system, as shown generally in Figures 1b-1d.

[0036] Alternatively or additionally, the method 100 may comprise simulating 120 the surrounding environment of the perception system, in particular the object to be recognised within the surrounding environment, as also shown diagrammatically in Figures 1b to 1d.

[0037] In the method 100, steps 110 and / or 120 are not necessary, for example, if real sensor data, i.e. measured sensor data, already exists, for example from a test drive. Such measured sensor data can be provided with metadata, for example by (automatic) labeling.

[0038] In the method 100, sensor data measured fully or partially by a perception system may be received 130. Metadata pertaining to this may be mechanically generated (e.g., via a fourth trained machine learning model) based on the sensor data and / or may be manually generated.

[0039] As shown diagrammatically in FIG. 1c, determining 150 the error probability based on the received 130 sensor data may include applying 151 a plurality of further machine learning models, each configured and trained to map the sensor data to the solution of a perception task, in particular to the detection of an object, each application 151 resulting in a perception result, in particular an object detection result, in particular a perception success or a perception failure. Here, the plurality of further machine learning models can be replaced by one further machine learning model. However, alternatively, it has been found that the plurality of further machine learning models is advantageous in that the influence that each of the further machine learning models has on the error probability is averaged. To this end, the plurality of further machine learning models may be different from each other. In particular, the plurality of further machine learning models may be different from each other such that no two of the plurality of further machine learning models are identical.

[0040] Then, as also shown diagrammatically in Fig. 1c, determining 150 an error probability based on the received 130 sensor data may include determining 152 an error probability based on the perception results, in particular the object detection results, in particular the perception successes and perception failures. For example, the error probability may be determined from the ratio of the number of perception successes to the total number of perception successes and perception failures.

[0041] In another embodiment, shown generally in FIG. 1d, determining 150 an error probability based on the received 130 sensor data may include applying 153 a third machine learning model configured and trained to map the sensor data, and optionally the metadata, to an error probability for solving the perception task, whereby the error probability is determined 150.

[0042] The method 100 may include training 171 a machine learning model based on a training data set including a plurality of training data set elements, as shown generally in Figures 1b-1d. Each training data set element may be generated after the steps of the method 100. The plurality of training data set elements should be as large as possible to obtain a convincing trained machine learning model.

[0043] Further disclosed is a computer-implemented method 200, as shown diagrammatically in Fig. 2, for evaluating the behavior of an autonomous system configured to solve a perception task. The perception task may in particular include a task of object recognition, i.e. an object, such as a pedestrian, may be recognized to solve, for example, the perception task or a part thereof.

[0044] The method 200 includes simulating 210 a perception system of an autonomous system.

[0045] The method 200 further comprises simulating 220 the surroundings of the perception system. Thanks to the simulation steps 210, 220, the method can be performed very frequently, making it particularly suitable for Monte Carlo analysis.

[0046] The method 200 further comprises providing (or receiving) 230 a machine learning model configured and trained to map at least the sensor data and metadata to an error probability for solving at least the perception task, in particular for object recognition. As already described in relation to the method 100, this metadata also encodes an impact on the solvability of the perception task, in particular on the recognizability of the object. The machine learning model may have been trained 170, 171 according to the method 100.

[0047] The method 200 further comprises receiving 240 sensor data and metadata. The metadata can for example be based on super knowledge, in particular the simulation units 210 and 220, i.e. simulator knowledge. Alternatively or additionally, the metadata can for example be determined based on sensor data from an upstream procedure. The metadata does not necessarily contain information about the solvability of the perception task to be solved. Rather, the metadata comprises factors that have an influence on the solvability of the perception task. The solvability is quantified for the first time in the method 200 in the form of an error probability output by the machine learning model in step 240.

[0048] As already explained, the metadata may for example comprise the obscuration degree of the object to be recognized. Alternatively or additionally, the metadata may for example comprise a luminance value. Alternatively or additionally, the metadata may for example comprise a distance value of the object to be recognized. Depending on the perception system and the perception task to be solved, other parameters may also be considered as metadata.

[0049] The method 200 further includes applying 250 a machine learning model to at least the received 240 sensor data and metadata, whereby at least one error probability is determined.

[0050] The method 200 further includes evaluating 260 a behavior of the autonomous system based on the error probability, whereby an evaluation is obtained 260. The evaluating 260 may be performed over a predetermined period of time.

[0051] 2, steps 230 and 240 may be performed in any order or may be performed simultaneously, and steps 210 and 220 may be performed simultaneously, but not simultaneously.

[0052] Evaluating 260 the behavior of the autonomous system may include rolling a die for at least one event based on an error probability, where the at least one event is identified as an error event or a non-error event. Rolling the die may be achieved, for example, by forming an empirical distribution function based on the error probability and evaluating its inverse based on (pseudo)random numbers on the real interval [0,1].

[0053] Evaluating 260 the behavior of the autonomous system may include, among other things, rolling dice for multiple events based on error probabilities (eg, as in a Monte Carlo simulation).

[0054] Evaluating 260 the behavior of the autonomous system can be based on at least two events, each at a different time. Evaluating 260 the behavior of the autonomous system can be based on a plurality of events.

[0055] The method 200 may include adjusting 270 a perception system of the autonomous system based on the evaluation, which may improve the perception system and / or the autonomous system, and in particular may increase the reliability and authenticity of the autonomous system.

[0056] The perception task (or more precisely, solving the perception task) in both methods 100, 200 may include, for example, object recognition. Alternatively or additionally, the perception task (or more precisely, solving the perception task) may include semantic segmentation. Alternatively or additionally, the perception task (or more precisely, solving the perception task) may include free space recognition. Further perception tasks or combinations thereof are also envisaged.

[0057] In both methods 100, 200, the machine learning model may be assembled from one or more sub-models. For example, the machine learning model may include a sub-model for nearby objects and a sub-model for distant objects. In this case, first, the respective sub-model to be trained 170, 171 or applied to calculate the error probability 250 may be selected, for example, based on metadata and / or based on further input variables.

[0058] Further disclosed is a computer system configured to perform the computer-implemented method 100 for training a machine learning model for assessing behavior of an autonomous system, particularly configured to solve a perception task including object recognition. Alternatively or additionally, the computer system may be configured to perform the computer-implemented method 200 for assessing behavior of an autonomous system, particularly configured to solve a perception task including object recognition. The computer system may include a processor and / or a main memory.

[0059] Further disclosed is a computer program configured to perform the computer-implemented method 100 for training a machine learning model for assessing the behavior of an autonomous system, in particular configured to solve a perception task including object recognition. Alternatively or additionally, the computer program may be configured to perform the computer-implemented method 200 for assessing the behavior of an autonomous system, in particular configured to solve a perception task including object recognition. The computer program may exist, for example, in an interpretable or editable form. The computer program may be (even partially) loaded into the RAM of the computer to be implemented, for example, as a bit sequence or a byte sequence.

[0060] Further disclosed is a computer readable medium or signal that stores and / or includes a computer program. The medium may include, for example, one of a RAM, a ROM, an EPROM, a HDD, a SDD, etc., on which the signal is stored.

Claims

1. A computer-implemented method (100) for training a machine learning model for evaluating the behavior of an autonomous system, configured to solve a perception task, in particular including object recognition, comprising: - receiving (130) sensor data by a perception system of the autonomous system; - receiving (140) metadata encoding an impact on the solvability of the perceptual task, in particular on the recognizability of an object; - determining (150) an error probability for solving the perception task, e.g. for the object recognition, based on the received (130) sensor data; Providing a machine learning model (160) configured to map the sensor data and the metadata to an error probability for solving the perception task, in particular for the object recognition; training (170) the machine learning model based on training dataset elements including the received (130) sensor data, the received (140) metadata, and the identified (150) error probability; A method (100).

2. The method (100) comprises: - simulating (110) the perception system of the autonomous system; - simulating (120) an environment surrounding the perception system, in particular the object to be recognized within said environment; The method of claim 1 , comprising:

3. the sensor data is measured by the perception system; the metadata is mechanically generated based on the sensor data and / or is manually generated; 2. The method of claim 1 (100).

4. Determining (150) a probability of error based on the received (130) sensor data comprises: - applying (151) a number of further machine learning models, each configured and trained to map the sensor data to the solution of said perception task, in particular to the detection of an object, each application (151) resulting in a perception result, in particular an object detection result, in particular a perception success or a perception failure; - determining (152) said error probability based on said perception results, in particular on said object detection results, in particular based on said perception successes and said perception failures; The method (100) of any one of claims 1 to 3, comprising:

5. the plurality of further machine learning models are each distinct; 5. The method (100) of claim 4.

6. Determining (150) a probability of error based on the received (130) sensor data comprises: applying (153) a third machine learning model configured and trained to map the sensor data, and optionally the metadata, to an error probability for solving the perception task, whereby the error probability is determined (150); The method (100) of any one of claims 1 to 3, comprising:

7. The method (100) comprises: Training the machine learning model based on a training data set including a plurality of training data set elements (171). The method (100) of any one of claims 1 to 6, comprising:

8. A computer-implemented method (200) for evaluating the behavior of an autonomous system configured to solve a perception task, in particular involving object recognition, said method (200) comprising: - simulating (210) a perception system of said autonomous system; - simulating (220) an environment surrounding the perception system; Providing a machine learning model (230) configured and trained to map sensor data and metadata to error probabilities for solving the perception task, in particular for the object recognition, the metadata encoding an impact on the solvability of the perception task, in particular on the recognizability of an object; Receiving sensor data and metadata (240); applying (250) the machine learning model to the received (240) sensor data and metadata, whereby an error probability is determined (250); - evaluating (260) a behavior of the autonomous system based on the error probability, whereby an evaluation is obtained as a result (260); The method (200).

9. Evaluating (260) the behavior of the autonomous system includes rolling a die for at least one event based on the error probability; the at least one event is identified as an error event or as a non-error event; 9. The method (200) of claim 8.

10. evaluating (260) the behavior of the autonomous system based on at least two events, each at a different time; 10. The method (200) of claim 9.

11. Evaluating (260) the behavior of the autonomous system based on a plurality of events.

11. The method (200) of claim 10.

12. The method (200) comprises: Adjusting the perception system of the autonomous system based on the evaluation (270). The method (200) of any one of claims 8 to 11, comprising:

13. The machine learning model has been trained (170, 171) according to the method (100) of any one of claims 1 to 7. The method (200) of any one of claims 8 to 12.

14. the metadata includes an obscuration value of the object to be recognized, a brightness value of the object to be recognized, and / or a distance value of the object to be recognized; The method (100, 200) of any one of claims 1 to 13.

15. The perception task includes object recognition, semantic segmentation, and / or free space perception. The method (100, 200) of any one of claims 1 to 14.

16. 1. A computer system, comprising: - configured to perform a computer-implemented method (100) for training a machine learning model for assessing the behavior of an autonomous system, in particular configured to solve a perception task including object recognition, according to any one of claims 1 to 7 or any one of claims 14 to 15, and / or - configured to perform a computer-implemented method (200) for evaluating the behavior of an autonomous system, in particular configured to solve a perception task including object recognition, according to any one of claims 8 to 15, Computer system.

17. A computer program, comprising: - configured for performing a computer-implemented method (100) for training a machine learning model for assessing the behavior of an autonomous system, in particular configured to solve a perception task including object recognition, according to any one of claims 1 to 7 or any one of claims 14 to 15, and / or - configured to perform a computer-implemented method (200) for evaluating the behavior of an autonomous system, adapted to solve a perception task, in particular including object recognition, according to any one of claims 8 to 15, Computer program.

18. 18. A computer readable medium or signal storing and / or including a computer program according to claim 17.

Images