Computer-implemented methods for generating first and second training datasets, for evaluating changes to a machine learning model, for evaluating a machine learning model, for perceiving traffic scenarios, computer program for perceiving traffic scenarios, storage medium, data carrier signal and automated driving system

Abstract

A computer-implemented method for generating an initial training dataset, wherein the initial training dataset comprises synthetically generated data in order to obtain the most diverse data possible, and a machine learning model can be trained using the initial training dataset; the method comprises the following steps: • Obtaining a basic dataset, wherein the basic dataset includes data elements and associated annotations, where the data elements include sensor data (GTV100); • Determining events that cause a machine learning model trained using a base data set to make at least partially incorrect predictions (GTV101); • Mapping the events as changed data elements (GTV102); • Establishing occupancy levels, where occupancy levels represent characteristics of changes (GTV103); • Generating synthetic data and associated annotations, wherein the synthetic data are based on data elements of the base dataset and wherein the synthetic data include changes based on the specified events (GTV104).

Description

[0001] The invention relates to computer-implemented methods for generating first and second training datasets. Furthermore, the invention relates to a computer-implemented method for evaluating changes to a machine learning model, as well as computer-implemented methods for evaluating a machine learning model. The invention also relates to a computer-implemented method for perceiving traffic scenarios, a computer program for perceiving traffic scenarios, a storage medium, a data carrier signal, and an automated driving system.

[0002] The following definitions apply to the entire disclosure content.

[0003] Artificial neural networks and other machine learning models often suffer from insufficient data for training. Ideally, a robust machine learning model requires enough data points to reflect all possible events. This is frequently impossible, as illustrated by the example of autonomous driving. In this scenario, a machine learning model would need to have encountered every possible situation at least once to function reliably. Furthermore, every possible anomaly, malfunction, or similar issue would need to be considered and covered by data. For instance, if a camera's red channel fails, the machine learning model should still be able to recognize pedestrians and other objects, rather than simply failing completely.

[0004] The present invention therefore comprises a method for generating new synthetic data based on existing real data (for example, sensor-acquired data) in order to obtain the most diverse range of new data possible; for example, the new data could be traffic scenarios. In this method, so-called data augmentation is applied differently than previously known in the art. Instead of using arbitrary random data augmentation when training a model, it is used to verify the robustness of an already trained model and subsequently to improve it in a targeted manner. The method disclosed below can be applied to any dataset and any machine learning model that is trained using data. The result is a new dataset consisting of all the originally existing data and a large number of synthetically generated data points.In contrast to the random generation of a very large number of new data points, the disclosed method ensures that, by means of an intelligent workflow, only a limited number of new data points are generated, which, through targeted selection, are sufficient to train a machine learning model and ensure high robustness against errors.

[0005] The object of the invention was to provide a method for generating an optimized training dataset for a machine learning model. In particular, it was an object of the invention to generate a training dataset that includes modifications to the original data in order to cover any possible anomaly, malfunction, or similar feature in the data.

[0006] The subject matter of the independent claim and the dependent claims each solve this problem. Further developments and advantageous embodiments will become apparent from the drawings and the description of preferred embodiments.

[0007] According to one aspect, the invention provides a computer-implemented method for generating a first training data set, wherein the first training data set comprises synthetically generated data in order to obtain the most diverse data possible and a machine learning model can be trained using the first training data set, the method comprising the steps: • Obtaining a basic dataset, wherein the basic dataset comprises data elements and associated annotations, where the data elements comprise sensor data; • Identifying events that cause a machine learning model trained using a base dataset to make at least partially incorrect predictions; • Representing events as changed data elements; • Defining assignments, where assignments represent levels of change; • Generating synthetic data and associated annotations, wherein the synthetic data is based on data elements of the base dataset and wherein the synthetic data includes changes based on the specified events.

[0008] Synthetically generated data is data that is artificially generated, for example with the help of a computer, and is not based on real measurements.

[0009] A machine learning model includes, for example, artificial neural networks. However, it can also include other machine learning models.

[0010] Using the first training dataset generated with the disclosed method, a machine learning model can be trained, for example, to perceive traffic scenarios for use in an automated driving system. Other applications are also possible.

[0011] Sensor data includes data that has been measured by sensors, but also data that has been generated by a simulation based on a sensor model.

[0012] Sensor data can include, for example, traffic scenarios. Traffic scenarios include information about road users and infrastructure. Road users include, for example, vehicles and / or pedestrians. Infrastructure includes, for example, traffic signs, traffic lights, vegetation, road surface, and / or road markings.

[0013] The present invention is not limited to applications for perceiving traffic scenarios, for example, for use in an automated driving system. The first training dataset and the disclosed training method can also be applied to other applications, such as Industry 4.0 applications, for example, for detecting component defects in production plants.

[0014] The basic dataset is identical for all disclosed methods. The basic dataset comprises data elements and associated annotations. The data elements include sensor data, for example, from environmental sensors such as image sensors and / or acoustic sensors, or other sensors.

[0015] Events that cause a machine learning model trained using a base dataset to make at least partially incorrect predictions are those that are either not covered at all, not fully covered, or at least not sufficiently covered by the base dataset. Such events can include hardware-related errors, such as a defective color channel in a camera and / or a dirty camera lens. Environmental factors, such as rain, snow, fog, or others, can also lead to incorrect predictions. Furthermore, defects that can occur on components, for example, due to poor manufacturing quality, which are rare in reality, may also be insufficiently covered by the base dataset.

[0016] Representing events as changes to data elements can involve artificially altered image files, such as those affected by fog (mentioned here as an example of a change). It could also involve artificially altered audio files or other data elements. It can be advantageous to represent an event not as a single change to a data element, but rather by representing different levels of change based on assignments. Using fog as an example of a change, assignments ranging from light to dense fog can be selected in predefined steps, thus artificially altering the data elements. The generated synthetic data contains each assignment as a change. There are various approaches to determining the assignments for each change. The simplest is a predetermined selection, e.g.,Fixed intervals within a range, or a randomized selection (i.e., determining a random assignment each time), are options. Since this method of determining assignments can be very computationally intensive, it can be advantageous to use an optimization method that optimally searches a given interval for assignments that cause varying degrees of detrimental change. Examples of such methods include gradient descent and evolutionary optimization.

[0017] The synthetic data and associated annotations generated from the first training dataset then include changes based on specific events compared to the data of the base dataset.

[0018] The following computer-implemented method can be used to train a machine learning model with a basic dataset. The method comprises the following steps: • Obtaining a basic dataset, wherein the basic dataset comprises data elements and associated annotations, where the data elements comprise sensor data; • Feeding the basic data set into the machine learning model; • Obtaining predictions from the machine learning model; • Comparing the predictions with the annotations of the base dataset; • Learning the annotations of the base dataset by fitting the model.

[0019] Furthermore, a numerical value can then be determined to obtain information about the degree of agreement between the predictions made by the machine learning model and the annotations of the underlying dataset. The determination of this numerical value can be carried out, for example, as described below: • Determining a numerical value between zero and one, where the numerical value is one if all predictions match the annotations of the base dataset and zero if none of the predictions match the annotations of the base dataset.

[0020] The calculated numerical value between zero and one can be used to assess how good the predictions of the trained model are.

[0021] The training method can be, for example, supervised learning. However, other training methods are also possible. In supervised learning, the model is adapted, for example, by gradient-based optimization of weighting factors in the machine learning model. Other training methods, such as reinforcement learning, can also be used.

[0022] The calculated numerical value between zero and one can be used to assess how good the predictions of the trained model are.

[0023] According to another aspect, the invention provides a computer-implemented method for evaluating changes to a machine learning model for verifying and improving the robustness of the machine learning model, wherein the machine learning model is pre-trained with a basic data set, the method comprising the steps: • Obtaining an initial training dataset, where the initial training dataset comprises synthetically generated data; • Feeding the first training data set into the machine learning model; • Obtaining predictions from the machine learning model; • Evaluating the predictions made by the machine learning model based on assignments, where assignments represent levels of changes; • Determining the influence of at least one change to data elements on the processing of the data elements by the machine learning model for each assignment, where the influence of each assignment is represented by a numerical value; • Average of all numerical values; • Values ​​of at least one change based on the average numerical value.

[0024] The evaluation can be carried out as follows, for example: the lower the average numerical value, the more harmful the change.

[0025] A model pre-trained with a base dataset is now trained with an initial training dataset, which includes synthetically generated data based on changes to data elements of the base dataset, and where the synthetic data includes changes based on specific events.

[0026] By determining the impact of at least one change on the machine learning model for each assignment, it is possible, for example, to ascertain the magnitude of the change's influence on the model. The individual assignments for each change can be stored in memory.

[0027] Averaging all numerical values ​​yields a single numerical value for each change, encompassing all considered values ​​for that change. For example, an evaluation method might reveal that the lower the numerical value, the more detrimental the change is to the model's predictions. Subsequently, the changes can be evaluated. This means determining which change is the most detrimental—that is, which change results in the model making the least accurate predictions compared to other changes. For instance, a hypothetical result could be that fog is more detrimental than rain.

[0028] The machine learning model evaluated using the disclosed method can, for example, be used to perceive traffic scenarios for use in an automated driving system. However, other applications are also possible.

[0029] According to another aspect, the invention presents a computer-implemented method for generating at least one second training data set based on the changes deemed most harmful to n, where n is a natural number, wherein the second training data set comprises synthetically generated data and a machine learning model can be trained using the second training data set, the method comprising the steps: • Receiving evidence of the most n-damaging change; • Determining the influence of occupancy on the change; • Generating data elements and associated annotations, wherein the number of generated data elements increases with the influence of the assignment on the change, wherein the total number of generated data elements corresponds to the total number of data elements of a basic dataset, wherein the basic dataset comprises data elements and associated annotations, wherein the data elements comprise sensor data; wherein, in particular, traffic scenarios for use in an automated driving system can be perceived with the second training dataset generated in this way.

[0030] The base dataset, as described above, is the base dataset used for all disclosed methods. Using the disclosed method, a second training dataset can be generated for each change. The number of possible changes is n, where n is a natural number. The more detrimental the effect of an occupancy on the change, the more synthetically generated data elements the dataset contains based on that occupancy. In a hypothetical example of fog as a change, dense fog as an occupancy might have a greater effect on the change, i.e., be more detrimental to the model, than light fog. Consequently, more data elements with dense fog would be generated than with light fog.

[0031] It can be advantageous to start with the most damaging change. After generating a data set for the most damaging change, the second most damaging change can be used and the procedure carried out. This procedure can be performed for each possible change as needed.

[0032] A second training dataset generated in this way is suitable for perceiving traffic scenarios for use in an automated driving system, but a second training dataset generated in this way is also suitable for perceiving other instances and for other purposes.

[0033] An automated driving system comprises hardware components, such as environmental sensors (e.g., imaging sensors and / or radiation sensors), control units, actuators, drive and chassis components, and software components, such as software for evaluating raw data from the environmental sensors, determining environmental models, or planning a trajectory or mission. The driving system can be, for example, a car, truck, bus, people or cargo mover, shuttle, transport system, or robot.

[0034] The computer-implemented method described below allows a machine learning model, which has already been pre-trained with a basic dataset, to be trained using the generated second training dataset. The method comprises the following steps: • Obtaining a second training dataset; • Feeding the second training dataset into the machine learning model; • Obtaining predictions from the machine learning model; • Compare the predictions with the annotations of the second training dataset; • Learning the annotations of the second training dataset by adapting the model.

[0035] The second training dataset, based on the most damaging change, can initially be used to train the machine learning model. The machine learning model can then be evaluated as described below. Training with the second training dataset, based on further changes, and evaluating the machine learning model can be repeated iteratively until all changes have been processed or until the remaining changes cause minimal damage.

[0036] According to another aspect, the invention presents a computer-implemented method for evaluating a machine learning model for verifying and improving the robustness of the machine learning model, using a first training data set, wherein the machine learning model is trained with at least one second training data set, wherein the at least one second training data set is generated based on the changes rated as most damaging to the n-scale, the method comprising the steps: • Obtaining an initial training dataset as described above; • Feeding the first training data set into the machine learning model; • Obtaining predictions from the machine learning model; • Evaluating the predictions made by the machine learning model based on assignments, where assignments represent levels of changes; • Determining the influence of at least one change to data elements on the processing of the data elements by the machine learning model for each assignment, where the influence of each assignment is represented by a numerical value; • Average of all numerical values; • Comparison of the averaged numerical value with the previously described averaged numerical value, where the averaged numerical value is higher than the previously described averaged numerical value.

[0037] The averaged value calculated here is based on the machine learning model being trained on the second training dataset and evaluated on the first training dataset, whereas the previously described averaged value is based on the previously evaluated machine learning model being trained only on the base dataset. Therefore, the averaged value calculated here is, as expected, higher than the previously described averaged value.

[0038] According to the disclosed method, a machine learning model trained with a second training dataset is evaluated using a first training dataset. This provides information about the impact of changes on a machine learning model that has already been trained with a dataset based on at least one of the most damaging changes. By comparing the numerical values ​​of the impact on the model trained with the second training dataset and the model trained only with the original dataset, it is shown to what extent the model trained with the second training dataset has become less sensitive to changes.

[0039] A machine learning model trained on at least one other training dataset can be evaluated using a base dataset via the following computer-implemented procedure. The procedure comprises the following steps: • Obtaining a basic dataset, where the basic dataset includes data elements and associated annotations; • Feeding the basic data set into the machine learning model; • Obtaining predictions from the machine learning model; • Comparing the predictions with the annotations of the base dataset; • Determining a numerical value between zero and one, where the numerical value is one if all predictions match the annotations of the base dataset and zero if none of the predictions match the annotations of the base dataset.

[0040] According to the disclosed method, a machine learning model trained with a second training dataset is evaluated using the base dataset. The numerical value determined here provides information about how the quality of the model, which has already been trained with a dataset based on at least one of the n most damaging changes, has changed with respect to the base dataset compared to a model trained exclusively with the base dataset.

[0041] According to another aspect, the invention provides a computer-implemented method for perceiving traffic scenarios, the method comprising the following steps: • Receiving sensor data from at least one traffic scenario; • Processing the sensor data by a machine learning model trained with a second training dataset and a base dataset, the machine learning model being trained in particular to perceive traffic scenarios; • Generating a signal that indicates the traffic scenarios determined using the machine learning model; • Outputting the signal to a control unit for regulating and / or controlling longitudinal and / or lateral guidance of an automated driving system or to a driver.

[0042] The methods described above can be used to train a machine learning model that can be used to perceive traffic scenarios, especially for use in an automated driving system.

[0043] According to another aspect, the invention provides a computer program for perceiving traffic scenarios, the computer program comprising program instructions that cause a computer to execute the steps of a method for perceiving traffic scenarios when the computer program is loaded onto the computer or executed.

[0044] The instructions of the computer program according to the invention comprise machine instructions, source code, or object code written in assembly language, an object-oriented programming language, for example C++, or in a procedural programming language, for example C. The computer program is executed wholly or partially on an integrated circuit or in a remote system comprising a cloud. The integrated circuit can, for example, be a graphics processing unit (GPU).

[0045] According to another aspect, the invention provides a storage medium on which the computer program for perceiving traffic scenarios is stored. This storage medium can be, for example, a non-volatile, permanent storage medium. A storage medium could be, for instance, a USB flash drive, a hard drive, a memory card, or a CD-ROM. However, the storage medium could also be, for example, a cloud service in which the computer program is stored.

[0046] According to another aspect, the invention provides a data carrier signal that transmits the computer program for perceiving traffic scenarios. This data carrier signal allows the computer program for perceiving traffic scenarios to be transmitted, for example, from a cloud to a control unit of an automated driving system, for instance via an over-the-air update.

[0047] According to another aspect, the invention provides an automated driving system comprising at least one environment perception unit and at least one control unit for longitudinal and / or lateral guidance, wherein the environment perception unit executes a machine learning model trained with the second training data set and the basic data set, and the control unit regulates and / or controls the driving system based on a perception of traffic scenarios by the machine learning model.

[0048] The environmental perception unit can, for example, comprise one or more imaging sensors, in particular cameras. It can also, for example, comprise one or more beam sensors, in particular lidar and / or radar sensors. Likewise, the environmental perception unit can comprise one or more acoustic sensors or other sensors.

[0049] The control unit for longitudinal and / or lateral guidance can, for example, send signals to a brake and / or acceleration and / or steering unit.

[0050] All disclosed independent claims realize the inventive idea of ​​optimally training a machine learning model with the most diverse dataset possible, in order to subsequently use the machine learning model in real-world applications, particularly in an automated driving system for perceiving traffic scenarios. The present invention therefore comprises a method for generating new synthetic data based on existing real data (for example, sensor-acquired data) in order to provide the most diverse dataset possible. Advantageously, the method disclosed in claim 1 for generating a first training dataset produces synthetic data that do not occur, or at least only very rarely, in metrologically acquired sensor data. The synthetic data are based on a base dataset and include changes based on specific events.To subsequently determine how the synthetically generated data, and thus the changes, affect a machine learning model, a method according to claim 2 for evaluating changes to a machine learning model is disclosed. This allows for the targeted determination of which change is most detrimental to the machine learning model. Based on the most detrimental change, further synthetic data can then be selectively generated, as disclosed in claim 3. The machine learning model is then trained with the newly generated dataset. Subsequently, as disclosed in claim 4, it can be evaluated how the dataset from claim 1 now affects the machine learning model.Iteratively, further second training datasets can be synthetically generated based on detrimental changes until the first training dataset from claim 1, which can include all possible changes, has no or at least only a very minor influence on the trained machine learning model. It is also possible to feed the base dataset into the machine learning model trained in this way to obtain information about how the machine learning model behaves with respect to the base data and how the quality changes during the training process according to the disclosed methods.

[0051] A machine learning model trained in this way can then be used to perceive traffic scenarios, provided it has been trained specifically for perceiving traffic scenarios. Such a model can then be used in an automated driving system.

[0052] The invention is explained by way of example using the following embodiments. They show: Fig. 1. A flowchart of a computer-implemented procedure for generating an initial training dataset, Fig. 2. A flowchart of a computer-implemented procedure for evaluating changes on a machine learning model, Fig. 3. A flowchart of a computer-implemented procedure for generating a second training dataset, Fig. 4. A flowchart of a computer-implemented procedure for evaluating a machine learning model, Fig. 5. A flowchart of a computer-implemented procedure for the perception of traffic scenarios, Fig. 6 a schematic representation of an exemplary implementation of a computer program, Fig. 7 a schematic representation of an exemplary embodiment of a storage medium, Fig. 8 a schematic representation of an exemplary embodiment of a data carrier signal, Fig. 9 a schematic representation of an exemplary embodiment of an automatically operated vehicle.

[0053] In the figures, identical reference symbols denote identical or functionally similar objects. To avoid repetition, only the relevant objects are marked with reference symbols in each figure.

[0054] The Fig. The flowchart shown in Figure 1 of a computer-implemented procedure for generating an initial training dataset includes a step GTV100 of obtaining a base dataset, wherein the base dataset comprises data elements and associated annotations, where the data elements comprise sensor data. Furthermore, the flowchart shown in Figure 1 includes... Fig. The flowchart shown in Figure 1 comprises a step GTV101 of determining events that cause a machine learning model trained using a base dataset to make at least partially incorrect predictions, and a step GTV102 of mapping the events to changed data elements. Furthermore, the procedure according to Fig. 1 a step GTV103 of defining assignments, wherein assignments represent manifestations of changes, and a step GTV104 of generating synthetic data and associated annotations, wherein the synthetic data are based on data elements of the base dataset and wherein the synthetic data include changes based on the specified events.

[0055] The Fig. The flowchart shown in Figure 2 of a computer-implemented procedure for evaluating changes on a machine learning model, where the machine learning model is pre-trained with a base dataset, includes a step WV100 of obtaining an initial training dataset. The initial training dataset is, according to the [reference to diagram / reference], [reference to diagram / reference]. Fig. The procedure shown in 1 is generated. Furthermore, it includes the method shown in Fig. The two methods shown comprise a step WV101 of feeding the first training data set into the machine learning model and a step WV102 of obtaining predictions from the machine learning model. Furthermore, the method according to Fig. 2. A step WV103 of evaluating the predictions made by the machine learning model based on assignments, and a step WV104 of determining the influence of at least one change to data elements on the processing of the data elements by the machine learning model for each assignment, where the influence of each assignment is represented by a numerical value. This in Fig. The procedure shown also includes a step WV105 of averaging all numerical values ​​and a step WV106 of evaluating at least one change based on the averaged numerical value.

[0056] The Fig. 3 Flowchart shown of a computer-implemented procedure for generating a second training dataset based on the changes deemed most harmful to n according to Fig. 2, wherein the second training dataset comprises synthetically generated data and a machine learning model can be trained using the second training dataset, includes a step GTN100 of obtaining assignments of the most n-damaging change according to Fig. 2. Furthermore, this includes Fig. 3 methods shown comprise a step GTN101 of determining the influence of the occupancy on the change and a step GTN 102 of generating data elements and associated annotations, wherein the number of generated data elements increases with the influence of the occupancy on the change, wherein the total number of generated data elements corresponds to the total number of data elements of the basic data set, wherein the basic data set comprises data elements and associated annotations, and wherein the data elements comprise sensor data.

[0057] The Fig. 4 Flowchart shown of a computer-implemented procedure for evaluating a machine learning model using a first training dataset, wherein the machine learning model is equipped with at least one second training dataset, which according to Fig. 3 is generated, is trained, includes a step ETV100 of obtaining an initial training dataset, which according to Fig. 1 is generated. Furthermore, this includes in Fig. The four methods shown comprise a step ETV101 of feeding the first training data set into the machine learning model and a step ETV102 of obtaining predictions from the machine learning model. The procedure shown in Fig. The method shown in section 4 also includes a step ETV103 of evaluating the predictions made by the machine learning model based on assignments, where assignments represent values ​​of changes, and a step ETV104 of determining the influence of at least one change on the machine learning model for each assignment, where the influence of each assignment is represented by a numerical value, with the smallest numerical value representing the largest influence and the largest numerical value representing the smallest influence. Furthermore, the method shown in Fig. The four methods shown involve a step ETV105 of averaging all numerical values ​​and a step ETV106 of comparing the averaged numerical value with the one in Fig. 2 averaged numerical value, where the average numerical value is higher than the one in Fig. 2 is the average numerical value.

[0058] The Fig. The flowchart shown in Figure 5 of a computer-implemented method for perceiving traffic scenarios includes a step CV100 for obtaining sensor data for at least one traffic scenario. Furthermore, the flowchart includes... Fig. The 5 methods shown represent step CV101 of processing the sensor data by a machine learning model that is equipped with a according to Fig. The machine learning model is trained on a second training dataset generated in 3 and a base dataset, and is specifically trained to perceive traffic scenarios and includes step CV102 of generating a signal that indicates the traffic scenarios determined by the machine learning model. Furthermore, the system comprises... Fig. The 5 methods shown include step CV103 of outputting the signal to a control unit 7 for regulating and / or controlling a longitudinal and / or lateral guidance of an automated driving system 5 or to a driver.

[0059] The in Fig. Figure 6, a schematic representation of an embodiment of a computer program 1, shows an example of a computer program 1 for perceiving traffic scenarios, wherein the computer program 1 is designed according to Fig. 6 is executed by a computer 2.

[0060] The in Fig. Figure 7 shows a schematic representation of an embodiment of a storage medium 3, illustrating by way of example a USB stick as storage medium 3, onto which the computer program 1 has been loaded and which can transfer the computer program 1 to a computer 2.

[0061] The in Fig. Figure 8, a schematic representation of an embodiment of a data carrier signal 4, shows an example of a data carrier signal 4 that transmits a computer program 1. According to Fig. 8. As an example, the computer program 1 is transferred from a cloud using the data carrier signal 4.

[0062] The in Fig. Figure 9, a schematic representation of an embodiment of an automated vehicle 5, shows a passenger car as an example of an automated vehicle 5. According to Fig. 9 The automated vehicle 5 comprises an environment perception unit 6 and a control unit 7. The environment perception unit 6 is located according to Fig. 9. The vehicle interior is located in the area behind the windshield. The environmental perception unit could therefore include a sensor, for example a camera, that can capture data from the vehicle's surroundings, especially traffic scenarios. Reference sign GTV100 - GTV104 Procedure steps WV100 - WV106 Procedure steps GTN100 - GTN102 Process Steps ETV100 - ETV106 Procedure steps CV100 - CV103 Procedure steps 1 computer program 2 computers 3 Storage medium 4 Data carrier signal 5 Automated drive system 6 Environmental perception unit 7 Control unit

Claims

[1] Computer-implemented method for generating a first training data set, wherein the first training data set comprises synthetically generated data in order to obtain the most diverse data possible and a machine learning model can be trained using the first training data set, the method comprising the steps: • Obtaining a base dataset, the base dataset comprising data elements and associated annotations, the data elements comprising sensor data (GTV100); • Determining events that cause a machine learning model trained on a base dataset to make at least partially incorrect predictions (GTV101); • Mapping the events as changed data elements (GTV102); • Determination of assignments, whereby assignments reflect characteristics of changes (GTV103); • Generating synthetic data and associated annotations, where the synthetic data is based on data elements of the base dataset and where the synthetic data includes changes based on the determined events (GTV104). [2] A computer-implemented method for evaluating changes to a machine learning model for testing and improving the robustness of the machine learning model, wherein the machine learning model is pre-trained with a base data set, the method comprising the steps of: • Obtaining a first training data set, wherein the first training data set is generated according to claim 1 (WV100); • Feeding the first training dataset into the machine learning model (WV101); • Obtaining predictions from the machine learning model (WV102); • Evaluating the predictions made by the machine learning model based on occupancies, where occupancies reflect the extent of changes (W\ / 103); • Determining an influence of at least one change of data elements on the processing of the data elements by the machine learning model for each occupancy, whereby the influence of each occupancy is represented by a numerical value (WV104); • Averaging all numerical values ​​(W\ / 105); • Values ​​of at least one change based on the averaged numerical value (WV106). [3] A computer-implemented method for generating at least one second training data set based on the n-most harmful changes according to claim 2, wherein n is a natural number, wherein the second training data set comprises synthetically generated data and a machine learning model can be trained using the second training data set, the method comprising the steps: • Obtaining assignments of the n-most harmful change according to claim 2 (GTN100); • Determining the influence of the occupancies on the change according to claim 2 (GTN101); • Generating data elements and associated annotations, wherein the number of generated data elements increases with the influence of the occupancy on the change, wherein the total number of generated data elements corresponds to the total number of data elements of a basic data set, wherein the basic data set comprises data elements and associated annotations, wherein the data elements comprise sensor data (GTN102), wherein with the second training data set generated in this way, in particular traffic scenarios for use in an automated driving system (5) can be perceived. [4] A computer-implemented method for evaluating a machine learning model for checking and improving the robustness of the machine learning model, using a first training data set, wherein the machine learning model is trained with at least one second training data set, wherein the at least one second training data set is generated based on the n-most harmful changes according to claim 3, the method comprising the steps: • Obtaining the first training data set, wherein the first training data set is generated according to claim 1 (ETV100); • Feeding the first training dataset into the machine learning model (ETV101); • Obtaining predictions from the machine learning model (ETV102); • Evaluate the predictions made by the machine learning model based on occupancies, where occupancies reflect the extent of changes (ETV103); • Determining an influence of at least one change of data elements on the processing of the data elements by the machine learning model for each occupancy, whereby the influence of each occupancy is represented by a numerical value (ETV104); • Averaging of all numerical values ​​(ETV105); • comparing the averaged numerical value with the numerical value averaged in claim 2, wherein the averaged numerical value is higher than the numerical value averaged in claim 2 (ETV106). [5] Computer-implemented method for perceiving traffic scenarios, the method comprising the steps: • Obtaining sensor data from at least one traffic scenario (CV100); • Processing the sensor data by a machine learning model which is trained with a second training data set generated according to claim 3 and a base data set, wherein the machine learning model is trained in particular to perceive traffic scenarios (CV101); • Generating a signal indicating the traffic scenarios determined by the machine learning model (CV102); • Outputting the signal to a control unit (7) for regulating and / or controlling a longitudinal and / or transverse guidance of an automated driving system (5) or to a driver (CV103). [6] Computer program (1) for perceiving traffic scenarios, the computer program (1) comprising program instructions which cause a computer (2) to carry out the steps of a method according to claim 5 when the computer program (1) is loaded onto the computer (2) or executed. [7] Storage medium (3) on which the computer program (1) according to claim 6 is stored. [8] Data carrier signal (4) transmitting the computer program (1) according to claim 7. [9] Automated driving system (5) comprising at least one environment perception unit (6) and at least one control unit (7) for longitudinal and / or lateral guidance, wherein the environment perception unit (6) executes a machine learning model trained according to the method according to claim 5 and the control unit (7) regulates and / or controls the driving system based on a perception of the machine learning model of traffic scenarios.

Images