Methods and generators for generating perturbed input data for neural networks

Abstract

The invention relates to a method for generating perturbed input data (9) for a neural network (11) for analyzing sensor data, in particular digital images, of a driver assistance system, wherein a first measure is defined which indicates how the degree of change of the sensor data is measured, a second measure is defined which indicates what the perturbation from the sensor data is directed against, an optimization problem is produced from the combination of the first measure and the second measure, the optimization problem is solved by means of at least one solution algorithm, wherein the solution indicates a target perturbation of the input data (9), and the perturbed input data (9) is generated for the neural network (11) from the sensor data by means of the target perturbation. The invention also relates to a generator (10) for carrying out the method and a method for checking the robustness of a neural network (11) which employs the method for generating perturbed input data (9).

Description

TECHNICAL FIELD

[0001] The present invention relates to a method for generating disturbed input data for a neural network for analyzing sensor data of a driver assistance system, in particular for analyzing digital images. The present invention also relates to a method for checking the robustness of such a neural network and a method for improving a parameter set of such a neural network. The present invention also relates to a generator for generating disturbed input data for a neural network for analyzing sensor data of a driver assistance system, in particular for analyzing digital images. BACKGROUND

[0002] Modern vehicles comprise driver assistance systems which support the driver in controlling the vehicle or take over the driving task partially or completely. By using such driver assistance systems, different degrees of automation of the vehicle control can be achieved. In the case of a low degree of automation, only information and warnings are output to the driver. In the case of a high degree of automation, the driver assistance system actively intervenes in the control of the vehicle. For example, there is an intervention in the steering or positive or negative acceleration of the vehicle. In the case of a higher degree of automation, the intervention in the devices of the vehicle is such that certain types of movement of the vehicle can be carried out automatically, for example straight-ahead travel. In the case of the highest degree of automation, the vehicle can be driven automatically.

[0003] In such driver assistance systems, the analysis of digital images recorded by the surroundings of the vehicle during driving is of great importance. Only if the digital images are analyzed correctly, the driver assistance system can reliably control the vehicle. Machine learning has great potential in the analysis of digital images of driver assistance systems. Sensor raw data generated for example by a camera, a radar sensor or a lidar sensor of the vehicle are processed by means of deep neural networks. The neural networks generate output data from which the driver assistance system derives relevant information about partially or fully automated driving. For example, the type and position of objects in the environment of the vehicle and their behavior are determined. Furthermore, the road geometry and the road topology can be determined by means of neural networks. In particular, special convolutional networks (English: convolutional neuronal networks) are suitable for processing digital images.

[0004] For use in a driver assistance system, such a deep neural network is trained. Here, the parameters of the neural network can be adjusted appropriately by the input data without human expert intervention. For a given parameterization, here the difference between the output of the neural network and the ground truth is measured. This difference is also called "loss". Here, a so-called loss function is chosen in such a way that the parameters are differentiable with respect to it. In the course of gradient descent, the parameters of the neural network are then adjusted in each training step according to the derivative of the difference determined on the basis of more examples. These training steps are often repeated until the difference, i.e. the loss, no longer decreases.

[0005] With this approach, the parameters are determined without the need for human expert evaluation or semantically driven modeling. The result of this for the neural networks is that these are often largely opaque to people and their calculations cannot be explained. This leads to the fact that, in particular, deep neural networks cannot usually be systematically tested or formally verified.

[0006] In addition, there is the problem that deep neural networks are susceptible to harmful interference (English: adversial perturbations). Small manipulations of the input data that are hardly or not at all perceptible to people, or manipulations that do not change the situation assessment, can lead to the output data being significantly different from the output data that would have been produced without the manipulation. Such manipulations can both be intentionally induced changes to the sensor data due to sensor noise, weather influences or certain colors and contrasts, but also randomly occurring image changes.

[0007] It cannot be foreseen here which input features the neural network will react so sensitively to that the output data will also change significantly in the event of a slight change in the input data. This has the consequence that synthetic data cannot be successfully used to train neural networks used in such driver assistance systems. It has been found that neural networks trained on simulations or with other synthetic data have very poor performance when used in a driver assistance system that utilizes real sensor data. It has also been found that the implementation of driver assistance systems with neural networks in other domains also greatly reduces the quality of the function. It can happen, for example, that a driver assistance system with a neural network trained in the summer can not be suitable for implementation in the winter. Therefore, the development and approval of neural networks for driver assistance systems based on simulations is problematic.

[0008] Therefore, there is a need to develop neural networks for driver assistance systems that are robust against interference. Even when the input data is interfered with, the neural network should generate usable output data for the driver assistance system.

[0009] In order to achieve this, it is known to generate disturbed input data for a neural network by means of known disturbances and to test how the output data of the neural network reacts to these disturbed input data. A set of disturbances in the input data by means of which the robustness of the neural network to such disturbances can be tested exists. However, this leads to the problem that disturbed input data can only be generated to a limited extent by means of known disturbances. There is therefore a need for a neural network for analyzing sensor data, in particular digital images, of a driver assistance system, in order to test and improve a neural network, to generate disturbed input data for the neural network. SUMMARY

[0010] It is therefore the task of the present application to propose a method and a generator for generating disturbed input data for a neural network for analyzing sensor data, in particular digital images, of a driver assistance system, with which new disturbed input data can be generated for a neural network by means of known disturbances in a simple manner.

[0011] This task is achieved by the method and the generator according to the present application. Furthermore, a method for checking the robustness of a neural network for analyzing sensor data, in particular digital images, with respect to disturbed input data and a method for improving a parameter set of such a neural network can thereby be proposed.

[0012] In the case of a method according to the present application for generating disturbed input data for a neural network for analyzing sensor data of a driver assistance system, a first measure is defined, which indicates how the degree of change of a digital image is measured and quantified, wherein the first measure compares the image distance between two digital images and outputs a value for the image distance, and a second measure is defined, which indicates what the disturbance of the input data of a digital image is directed against, wherein the second measure is directed against a change in a certain class of objects, wherein the objects are recognized and classified. An optimization problem results from the combination of the first measure and the second measure, wherein the optimization problem comprises a loss function of the neural network, which contains the disturbance parameters as parameters and the images generated by the disturbance according to the second measure, and for which the minimum of the disturbance parameters is sought under the condition that the degree of change of the generated images with respect to the initial images according to the first measure is below a certain value. The optimization problem is solved by means of at least one solution algorithm, wherein the solution indicates the target disturbance of the input data, and disturbed input data for the neural network are generated from the sensor data by means of the target disturbance.

[0013] The sensor data is in particular a digital image. In this case, the target disturbance thus generates a disturbed, i.e. changed, digital image, which forms the input data for the neural network, which analyzes the digital image.

[0014] In the method according to the application, possible harmful disturbances of a neural network for analyzing sensor data are observed on a structural level. Disturbances are considered as a combination of different elements for which different measures are defined. Surprisingly, it is thereby possible to generate a large number of new harmful target disturbances based on known disturbances by analyzing the structure of the known disturbances in terms of the measures, instead of using random combinations of disturbances only.

[0015] In the method according to the application, an optimization problem is advantageously generated from two measures, which measure the change of the sensor data, in particular of a digital image. For such optimization problems, there are a large number of known solution algorithms. These solution algorithms can thus be employed to solve the optimization problem. A target disturbance of the input data is thereby generated. The disturbed input data for the neural network can then be generated from the sensor data with the aid of the target disturbance. The neural network can then be tested and trained on the basis of the disturbed input data. Advantageously, the method according to the application enables new disturbances to be generated very quickly and in a simple manner.

[0016] The first measure used in the method according to the application indicates how the degree of change of the sensor data is to be measured. If the sensor data is a digital image of a camera, the disturbance for testing the neural network should generally be as small as possible. The first measure indicates how the degree of change of the digital image is to be quantified. For example, a digital image can be changed by moving, rotating or mirroring the pixels of the image. The first measure indicates the degree of change of such a transformation. According to the first measure, a rotation or a translation of a digital image can be defined by a fixed point and a rotation angle or a translation distance in the horizontal and vertical direction. Furthermore, the first measure can determine the image distance for each pixel of the image by determining the sum of the differences of all pixel values. The pixel values can be, for example, grey values or color values. For each pixel, the difference of the pixel values of the original image and the disturbed image is formed. This difference is determined for each pixel and the differences are then added up. As a result, an image distance is generated which indicates the difference of the two images according to the first measure.

[0017] Furthermore, according to the first measure, altered image regions can be observed. An image region can be defined by a starting point and a range in the horizontal and vertical direction or by a list of pixels. An image distance can be determined for these image regions according to the first measure.

[0018] Furthermore, the first measure can indicate the degree of change of a digital image in relation to image features, such as brightness, contrast and / or structure values or any combination thereof.

[0019] In defining the first measure, restrictions can also be included, for example, that only certain image regions are considered in the first measure in which, for example, certain image features are present. For example, only those regions can be observed in which the contrast exceeds a certain threshold.

[0020] According to an embodiment of the method according to the application, the first measure is selected from first measures that measure a potentially naturally occurring disturbance, since disturbances determined by these measures can actually occur in the field. Such natural disturbances are, for example, changes in the sensor data due to weather influences, for example, fog or snow, sensor noise or due to camera dirt. Furthermore, naturally occurring disturbances are objects that naturally occur in the vehicle surroundings, for example, printed posters or stickers on objects. If, for example, the disturbance of the second measure is directed at making an object of a certain class disappear, a printed poster, a sticker, fog or a texture on an object can be added to the digital image. By means of such disturbances according to the second measure for a specific influence in the sensor data, it is advantageously possible to generate for the neural network such disturbed input data which are particularly relevant for use in a driver assistance system.

[0021] According to an embodiment of the method according to the application, the second measure is directed at a change in the class of an object. The second measure is in particular a measure of the difference between the real model output and the expected erroneous model output, i.e., the goal of the adversarial disturbance. For example, in the case of a digital image, a small image region or a small number of pixels can be disturbed in such a way that the object of the digital image is no longer recognized as a traffic participant, for example, a pedestrian, but as an object of another class, for example, a road region. Furthermore, the disturbance can aim to ensure that every time a region is recognized as a street, this street is always recognized as an empty street without other traffic participants.

[0022] According to a further embodiment of the method according to the application, the second measure can be directed at the disappearance of an object. The purpose of the disturbance is, for example, that the recognized object undergoes a change that makes it disappear. The second measure can also here be related only to certain image regions. For example, the purpose of the disturbance described by the second measure can be directed at the fact that a certain class of object cannot occur in a specific image region.

[0023] According to a further embodiment of the method according to the application, the second measure is directed at a change in an object of a specific class. For example, an object can be recognized and classified. For example, an image region can be assigned to a traffic participant. The goal of the second measure is then, for example, directed at the fact that this object is displayed larger or smaller or at another position. For example, an object that is classified as a pedestrian can be displayed smaller or larger. In this case, the enlargement is defined, for example, by the absolute amount of pixels by which the object is enlarged or reduced to the left, to the right, above and below as a result of the disturbance.

[0024] A large number of possible disturbances can be described by the second measure. Any change can be made to the sensor data in order to change the sensor data such that, when the sensor data in the driver assistance system is analyzed, it is no longer possible to obtain correct results, in particular with regard to safety. For example, a pattern or grid can be applied to the sensor data in order to make a specific class of objects, for example pedestrians, disappear in the digital image, but other objects continue to be correctly classified. For the application of the method according to the application in a driver assistance system, those second measures are of particular relevance here which measure disturbances which occur naturally: the model output looks plausible, but deviates from reality in certain safety-relevant details.

[0025] According to a design of the method according to the application, the disturbances described by the first measure and / or the second measure are disturbances which occur naturally. In order to use in a driver assistance system, the possible disturbances described by the first measure and / or the second measure are thus selected, which are of particular relevance for checking and improving the neural network which is applied in the driver assistance system.

[0026] According to a further design of the method according to the application, the first measure and / or the second measure are stored in a database. A data set on disturbances which occur naturally is then loaded from the database, which are measured with the first measure and / or the second measure with the disturbances which occur naturally. For example, the measure of possible disturbances at the input data (first measure) and the measure of possible changes in the model output (second measure) can be stored in the database. According to a design of the method according to the application, a data set of disturbances which occur naturally (measured with the first measure) and possible targets (adversarial changes in the model output - for example, ignoring all pedestrians - measured with the second measure) is then loaded from the database.

[0027] According to a further design of the method according to the application, a third measure is defined which indicates to which sensor data a third disturbance is applied. For example, the disturbance is applied to all data, only one data point or data with a specific condition, for example all data with a multi-lane road. An optimization problem is then generated from a combination of at least two of the first measure, the second measure and the third measure. The optimization problem is in particular produced from a combination of the first measure, the second measure and the third measure. The sensor data is in particular digital images. These digital images are in particular analyzed by a neural network in the driver assistance system.

[0028] The third measure can in particular relate to all sensor data, for example all digital images. For example, in all digital images, the disturbance can cause a certain class of objects to disappear.

[0029] Furthermore, the third measure can only influence a subset of the sensor data, in particular of the digital images. The disturbance can for example only describe some digital images which contain objects of a specific class, for example objects classified as pedestrians. Furthermore, the third measure can describe digital images recorded on a snowy or rainy day. Thus, when used in a driver assistance system, the disturbed input data of the neural network can for example lead to a different evaluation of special traffic situations or environmental conditions.

[0030] According to a further design of the method according to the application, the third measure only describes sensor data containing specific objects. Alternatively or additionally, the third measure can only select specific digital images.

[0031] The optimization problem generated on the basis of the measures can for example be formulated as follows: For a given maximum change of a digital image, for example by rotating a certain image region, the number of pixels classified as a person should be minimized and, precisely, the images in which a person appears as many as possible.

[0032] In another example, the number of pixels classified as a person should be minimized with a minimum change in the input image in regions of high contrast and, precisely, the images in which a person appears as many as possible.

[0033] In the method according to the application, a solution algorithm is specified for this optimization problem. According to a design of the method according to the application, the solution algorithm comprises an iterative method using the gradient of the neural network to determine the direction of change. Furthermore, iterative methods using sampling, evaluation and combinations thereof can be used.

[0034] According to a further design of the method according to the application, a Monte Carlo method is used as a solution algorithm, in which for example noise is generated for a digital image and the result is checked. According to a further design, a genetic algorithm can be used to solve the optimization problem.

[0035] The solution of the optimization problem can for example be a disturbed digital image or a disturbance which can be used to disturb the sensor data in order to generate disturbed input data for the neural network. The disturbed sensor data or the disturbed digital image then represents the input data of the neural network to be checked. By combining at pixel level, for example by summation, the disturbance can also be applied to a set of input data.

[0036] A further aspect of the application relates to a method for generating disturbed input data for a neural network for analyzing sensor data, in particular digital images, of a driver assistance system, wherein a first set of first metrics is defined, a second set of second metrics is defined, the first metrics each indicate differently how a degree of change of the sensor data is measured and quantified, wherein the first metrics compare an image distance between two digital images and output a value for the image distance, the second metrics each indicate differently what the disturbance of the sensor data is directed against, wherein the second metrics are directed against a change of objects of a certain class and the objects are recognized and classified. An arbitrary combination of a first metric of the first set and a second metric of the second set is selected, an optimization problem is generated from the selected combination of the first metrics and the second metrics, wherein the optimization problem comprises a loss function of the neural network, which comprises as parameters a disturbance parameter and an image generated by the disturbance according to the second metrics, and for the optimization problem a minimum of the disturbance parameter is sought under the condition that a degree of change of the generated image with respect to an initial image according to the first metrics is below a certain value. The optimization problem is solved by means of at least one solution algorithm, wherein the solution indicates a target disturbance of the input data, and the disturbed input data for the neural network is generated from the sensor data by means of the target disturbance.

[0037] The advantage of this method is that any first metric of the first set and any second metric of the second set can be used in order to reach a target disturbance by solving the optimization problem. The more metrics the first set and the second set comprise, the more different target disturbances can be generated by the method. Thus, a very large number of target disturbances can be produced.

[0038] According to a further design of the method according to the application, the first set comprises at least two, in particular at least five, different first metrics. However, the first metrics can also comprise more than 10, 20 or more than 100 metrics.

[0039] According to a further design of the method according to the application, the second set comprises at least two, in particular at least five, different second metrics. However, the second metrics can also comprise more than 10, 20 or more than 100 metrics.

[0040] The first metrics and / or the second metrics of the first set or the second set can in particular individually or in combination have the features as described above.

[0041] According to a further design of the method according to the application, a third metric is defined, which indicates to which type of sensor data the disturbance is applied, and an arbitrary combination of a first metric of the first set, a second metric of the second set and the third metric is selected. An optimization problem is then generated from the selected combination of the first metric, the second metric and the third metric.

[0042] The third metric can in particular have the features as described above, individually or in combination.

[0043] According to a further design of the method according to the application, a set of solution algorithms is defined which comprises a plurality of solution algorithms which each differently solve the optimization problem in order to generate different target disturbances of the input data. Then, an arbitrary solution algorithm of the set of solution algorithms is selected in order to generate the disturbed input data for the neural network from the sensor data. In this way, an even greater number of target disturbances can be produced, since the solution algorithms can also be varied, wherein each solution algorithm achieves a different target disturbance.

[0044] The solution algorithms of the set of solution algorithms can comprise an iterative method using the gradient of the neural network to determine the direction of change, and a sampling-based method, a gradient-based method, a gradient-based method with momentum and / or a proxy-model-based method.

[0045] The application also relates to a method for checking the robustness of a neural network for analyzing sensor data, in particular digital images, with respect to disturbed input data, wherein the following steps are carried out: providing a neural network with an associated parameter set; generating training data by means of an example-sensor data set; generating a first analysis of the example-sensor data set on the basis of the training data by means of the neural network; generating disturbed input data by means of the above-mentioned method as training data for the example-sensor data set for generating disturbed input data for the neural network; generating a second analysis of the example-sensor data set on the basis of the disturbed input data by means of the neural network; comparing the first analysis and the second analysis; and determining a robustness value depending on the comparison of the first analysis and the second analysis.

[0046] The application also relates to a method for improving a parameter set of a neural network for analyzing sensor data, in particular digital images, with respect to disturbed input data. In this method the following steps are carried out:

[0047] a. providing a neural network with an associated parameter set;

[0048] b. generating training data by means of an example-sensor data set;

[0049] c. generating a first analysis of the example data set on the basis of the training data by means of the neural network;

[0050] d. generating disturbed input data by means of the above-mentioned method as training data for the example-sensor data set for generating disturbed input data for the neural network;

[0051] e. generating a second analysis of the example-sensor data set on the basis of the disturbed input data by means of the neural network;

[0052] f. comparing the first analysis and the second analysis; and

[0053] g. determining a robustness value depending on the comparison of the first analysis and the second analysis.

[0054] Example - Example image of a sensor data set, in particular a digital image.

[0055] By means of this method, the method described at the outset can be used for generating the disturbed input data in order to check the robustness of the neural network for analyzing the sensor data with respect to the disturbed input data. If the neural network is applied in a method for analyzing sensor data of a driver assistance system, it is important for the safety during the operation of the vehicle on which the driver assistance system acts that the neural network is robust with respect to disturbed input data of the neural network. If the deviation in the first analysis and the second analysis is very small, the neural network is robust with respect to such disturbed input data. The disturbed input data then has a very small influence on the output of the neural network. If, however, the disturbed input data, even if the disturbance of the sensor data is very small, leads to a very large deviation of the second analysis of the example-sensor data set from the first analysis, the neural network is not robust with respect to the disturbance of the input data.

[0056] If the sensor data is a digital image, the first analysis and the second analysis can comprise a semantic segmentation of the digital image, a recognition of objects of the digital image, a classification of the digital image objects or a recognition of the position of the objects in the digital image. Furthermore, it can be recognized by means of the analysis how the objects in the digital image change. These analyses are particularly relevant when using a neural network in a driver assistance system, it is therefore important that the neural network is robust with respect to disturbances occurring in such analyses, so that only slight changes occur in the analysis if disturbed input data is used.

[0057] The application also relates to a method for improving a parameter set of a neural network for analyzing sensor data, in particular digital images, with respect to disturbed input data. The method comprises the steps a. to f. as described above. In step h., an improved parameter set of the neural network is then generated on the basis of the comparison of the first analysis and the second analysis.

[0058] The improved parameter set is obtained by training the neural network. The training is carried out with respect to disturbed and undisturbed sensor data, i.e. in particular digital images. The improved parameter set is then obtained, for example, by gradient descent (adversarial training).

[0059] The application also relates to a generator for generating disturbed input data for a neural network for analyzing sensor data, in particular digital images, of a driver assistance system, the generator having: a first metric unit with a first metric, which indicates how a degree of change in the sensor data is measured and quantified; a second metric unit with a second metric, which indicates what the disturbance of the input data from the sensor data is directed against, wherein the second metric is directed against a change in a certain class of objects, and the objects are recognized and classified; a processing unit, which is connected to the first and second metric units and is designed to generate an optimization problem from the first and second metrics, wherein the optimization problem comprises a loss function of the neural network, which contains a disturbance parameter and an image produced by the disturbance according to the second metric as parameters, and for which the minimum of the disturbance parameter is sought under the condition that the degree of change of the generated image relative to the initial image according to the first metric is below a certain value; a solving unit, which is connected to the processing unit and is designed to solve the optimization problem by means of at least one solving algorithm, wherein the solution indicates the target disturbance of the input data from the sensor data; and a generating unit, which is connected to the solving unit and is designed to generate the disturbed input data for the neural network from the sensor data by means of the target disturbance.

[0060] The generator according to the application is in particular designed to carry out the above-mentioned method for generating disturbed input data. The generator therefore also has the same advantages as the method.

[0061] According to one design of the generator according to the application, the generator also comprises a third metric unit with a third metric, which indicates which sensor data the disturbance is applied to. In this case, the processing unit is also connected to the third metric unit and is designed to generate the optimization problem from at least two of the first, second and third metrics.

[0062] The application also relates to a device for generating a parameter set for a neural network for analyzing sensor data of a driver assistance system, the device having: a first analysis unit for generating a first analysis by means of the neural network on the basis of training data of an example-sensor data set; the above-mentioned generator for generating disturbed input data, which is used to generate the disturbed input data as training data of the example-sensor data set; a second analysis unit for generating a second analysis of the example-sensor data set by means of the neural network on the basis of the disturbed input data; a comparison unit, which is connected to the first and second analysis units and is designed to compare the first and second analyses; and a generating unit, which is connected to the comparison unit and is designed to generate an improved parameter set for the neural network on the basis of the comparison of the first and second analyses.

[0063] The apparatus for generating a parameter set is specifically designed to carry out the above-mentioned method for improving a parameter set of a neural network. The apparatus therefore also has the same advantages as the method. BRIEF DESCRIPTION OF DRAWINGS

[0064] The application will now be explained by means of embodiments with reference to the accompanying drawings.

[0065] Figure 1 The structure of an embodiment of a generator according to the application is shown schematically;

[0066] Figure 2 The flow of an embodiment of a method for generating disturbed input data according to the application is shown schematically;

[0067] Figure 3 The structure of an embodiment of an apparatus for generating a parameter set according to the application is shown schematically;

[0068] Figure 4 The flow of an embodiment of a method for checking the robustness of a neural network according to the application is shown;

[0069] Figure 5 The flow of an embodiment of a method for improving a parameter set of a neural network according to the application is shown;

[0070] Fig. 6 shows an example of a disturbance. DETAILED DESCRIPTION

[0071] In embodiments of the application, sensor data is analyzed by a neural network, or disturbed input data is generated for a neural network from these sensor data. In these embodiments, the sensor data is raw data from vehicle sensors. The sensors can be video cameras, radar sensors, lidar sensors or any other sensors which generate sensor data which is further processed in a driver assistance system. In the following, it is assumed for example that the sensor data is digital images taken by a video camera of the vehicle. However, the application can also be applied in the same way to other sensor data.

[0072] Reference is first made to Figure 1 An embodiment of a generator 10 for generating disturbed input data for a neural network for analyzing digital images of a driver assistance system is described.

[0073] The generator 10 comprises a first metric unit 1, a second metric unit 2 and a third metric unit 3. The first metric unit 1 comprises a first metric which indicates how the degree of change of a digital image is measured. The first metric unit 1 defines how the degree of change of a digital image is measured. The definition of the first metric can be input into the first metric unit 1. However, the first metric unit 1 can also access a database 16 via an interface in which data is stored with a plurality of possible definitions of metrics which measure the degree of change of a digital image. For example, the first metric can compare the image distance between two digital images and output a value for the image distance. The image distance can be defined, for example, by the sum of the differences of all pixel values of the digital images to be compared.

[0074] In this embodiment, the first metric unit 1 selects from the database 16 the disturbance which is as natural as possible. A natural disturbance is understood to mean a disturbance which affects the digital image of the surroundings of the vehicle as would also occur as a result of phenomena which occur naturally in the surroundings of the vehicle. The change in the digital image as a result of a natural disturbance corresponds, for example, to a change in the digital image as a result of a weather phenomenon, for example, as a result of fog, snow or rain occurring. Furthermore, a natural disturbance is understood to mean an image change in which an object is inserted into the image or disappears from the image, which can also occur in the surroundings of the vehicle. For example, a poster or a sticker can be inserted in the surroundings of the vehicle on an object. Other, non-natural disturbances, such as can also be contained in the database 16, are not taken into account by the second metric unit 1, since these are less relevant for testing the neural network used in the driver assistance system.

[0075] The second metric unit 2 comprises a second metric which indicates what the input data of the digital image is disturbed by, i.e. the second metric defines what the digital image is disturbed by. The definition of the second metric can be transmitted to the second metric unit 2 by input. Likewise, the second metric unit 2 can also be connected to a database 16 in which data is stored about a large number of disturbances which are directed at a specific change in the digital image. This can here be a set of these disturbances.

[0076] In the embodiment, the second metric unit 2 selects from the database 16 the disturbance which is as credible as possible. A credible disturbance is understood to mean a disturbance which apparently leads to a true model output, but differs from it in relevant details. In the case of a credible disturbance, for example, the correct segmentation occurs, but in which the lane marking is always moving. Other, non-credible disturbances, such as can also be contained in the database 16, are not taken into account by the second metric unit 2, since these are less relevant for testing the neural network used in the driver assistance system. A severely non-credible model output can thus be easily detected.

[0077] The second measure, for example, can aim to increase the size of all objects assigned to a particular class, for example the class of pedestrians. The disturbance thus generates a digital image in which the objects of the initial image that are classified as pedestrians are iteratively enlarged in all four directions, wherein the resulting segmentation of the disturbed digital image is again combined with one another. The result is a digital image in which all objects that do not belong to the class of pedestrians remain unchanged, but the objects that belong to the class of pedestrians are displayed enlarged. The other objects change only within the scope of the change brought about by the enlargement of the objects of the class of pedestrians.

[0078] The third measure unit 3 comprises a third measure which indicates to which digital images the disturbance is applied. For example, it can be defined by the measure that the disturbance is applied only to digital images which indicate other traffic participants, i.e. for example pedestrians, cyclists and other vehicles.

[0079] The three measure units 1 to 3 are connected to a processing unit 4. The processing unit 4 is designed to generate an optimization problem from the three measures of the first to third measure units 1 to 3. For example, the optimization problem comprises a loss function of a neural network which contains the disturbance parameters and the images generated by the disturbance as parameters (second measure) as parameters. For the optimization problem, a minimum of the disturbance parameters should be found and, in particular, with respect to the digital images defined according to the third measure, and with the condition that the degree of change of the generated images with respect to the initial images according to the first measure is below a certain value.

[0080] The processing unit 4 transmits the optimization problem to a solution unit 5 as a data set. The solution unit 5 is connected to a database 6 in which at least one solution algorithm, preferably a plurality of solution algorithms, for the optimization problem is stored. Such solution algorithms are known per se. For example, a Monte Carlo method, a genetic algorithm and / or a gradient-based method can be stored in the database 6, which can be called by the solution unit 5. By means of these solution algorithms, the solution unit 5 can generate a target disturbance of the input data of the digital images as a solution of the optimization problem. The target disturbance thus generates a disturbed digital image which can be used as input data for a neural network for analyzing the digital images. The neural network is designed, in particular, for analyzing digital images of a driver assistance system.

[0081] The solution unit 5 transmits the target-disturbance to a generation unit 7. The generation unit 7 is also connected to a database 8 in which a plurality of digital images is stored. By means of the target-disturbance, the generation unit 7 can disturb the digital images in the database 8 so that disturbed input data 9 of the digital images for the neural network are generated. The disturbed input data 9 are then output by the generation unit 7. With these disturbed input data 9, the neural network can then be tested, trained or the parameter set of the neural network can be improved.

[0082] ReferenceFigure 2 An embodiment of the method according to the present invention will be explained for generating distorted input data 9:

[0083] In step S1, a first metric is defined, which indicates how to measure the degree of change in a digital image. The first metric, or the data set describing the first metric, is stored in the first metric unit 1.

[0084] In step S2, a second metric is defined, which indicates what the interference in the digital image is targeting. This second metric, or the data set describing the second metric, is also stored in the second metric unit 2.

[0085] Finally, in step S3, a third metric is defined, indicating which digital images the interference applies to. This third metric, or the data set describing it, is stored in the third metric unit 3.

[0086] In step S4, the data set describing the three metrics is transmitted to processing unit 4.

[0087] In step S5, processing unit 4 generates an optimization problem based on the combination of three metrics. In step S6, processing unit 4 transmits the data set describing the generated optimization problem to solution unit 5.

[0088] In step S7, the solving unit 5 solves the optimization problem using at least one solving algorithm, for example, by accessing the database 6 which has already transmitted the solving algorithm to the solving unit 5. The solution is the target interference of the digital image.

[0089] In step S8, the data set of the target interference is transmitted to the generation unit 7.

[0090] In step S9, generation unit 7 generates distorted digital images by accessing database 8 as input data 9 for the neural network. In step S10, these distorted input data 9 are output.

[0091] The following uses an example of enlarging pedestrian-type objects to illustrate the method according to the present invention in detail:

[0092] The following reference Figures 6A to 6C The method according to the present invention will be described in detail using an example of an enlarged pedestrian category object:

[0093] Given a model M, there exists an input x for this model. This input x is a digital image, such as... Figure 6A As shown. Furthermore, the output M(x) = y is defined. The disturbance is represented by Δ, thus obtaining the modified input x' = x + Δ. The modified output is then y' = M(x + Δ). The target output is represented by y''.

[0094] The output y of model M is Figure 6BThe digital image x has been segmented, that is, the pixels of the digital image x have been classified, as shown in Figure 6B The following classification is produced here:

[0095] K1 : sky;

[0096] K2 : nature;

[0097] K3 : building;

[0098] K4 : pedestrian;

[0099] K5 : traffic sign;

[0100] K6 : street;

[0101] K7 : marker.

[0102] The target output y" to be produced from the perturbation Δ is shown in Figure 6C The target of the perturbation Δ is to amplify the display of the pedestrian. The definition of the target perturbation is that each pixel is allowed to be displaced by a value of at most 3. The target data consists of the specific image x.

[0103] The first metric is then defined as follows:

[0104]

[0105] The size of the perturbation is thus measured as the maximum pixel value between 0 and 255 in the perturbation Δ.

[0106] The second metric is defined as follows:

[0107]

[0108] It defines the sum of the pixel differences with the target output.

[0109] The third metric is defined as follows:

[0110]

[0111] According to this third metric, only the input image x thus has a small size. Therefore, if d3(x') < 1 is required, the attack only involves the input image x. If d3(x') < 2 is required, the focus of attention on the data to be attacked changes significantly: the attack then involves all the images.

[0112] An optimization problem is then formed from these three metrics as follows:

[0113]

[0114] According to the optimization problem, Δ should be found such that d2(Δ) is minimal, with di(Δ) < 3 on x.

[0115] The optimization problem can be solved by means of known solution algorithms. Thereby, a new adversarial perturbation is derived from the known (d1, d3) and the new (d2) measures. A new adversarial perturbation is also generated by recombining the known measures (d1,.., d3) in a new way or by combining them with another solution algorithm. Thus, the method according to the application allows to construct almost any number of new adversarial perturbations in a simple way.

[0116] According to a variant of this example, the first measure can only allow a change of pixels in the image region which is classified as "tree". The following optimization problem is then obtained: one should find Δ in the image region "tree" of the digital image x such that d2(Δ) is minimal, with d1(Δ) < 3.

[0117] According to another variant of this example, one can search for a perturbation of all images with respect to the third measure, where the first measure d1 and the second measure d2 remain unchanged. The optimization problem can then be formulated as follows: one should find Δ such that d2(Δ) is minimal for all images, with d1(Δ) < 3. In other words: one should find Δ with d1(Δ) < 3 such that the model output for all input images x looks like y".

[0118] Another embodiment of the generator 10 according to the application and of the method for generating a perturbed input data 9 according to the application is explained below:

[0119] As in the first embodiment, the generator 10 of the other embodiment comprises a first measure unit 1 and a second measure unit 2. However, in this case, the first measure unit 1 comprises a first group with a plurality of first measures, which each indicate differently how a degree of change of the sensor data is to be measured. In this case, the second measure unit 2 comprises a second group with a plurality of second measures, which each indicate differently what the perturbation of the input data 9 of the sensor data is to be directed against. In this case, the processing unit 4 connected to the first measure unit 1 and the second measure unit 2 is designed to generate an optimization problem from an arbitrary combination of the first measures of the first group and the second measures of the second group.

[0120] The solution unit 5 connected to the processing unit 4 is then designed to solve the optimization problem by means of at least one solution algorithm, wherein the solution indicates a target perturbation of the input data 9 of the sensor data. Similar to the first embodiment, the generation unit 7 is also designed to generate the perturbed input data 9 for the neural network 11 from the sensor data by means of the target perturbation.

[0121] The method of the further embodiment is performed analogously to the method of the first embodiment. However, in this case, a first set comprising first metrics is defined, which each indicate differently how a degree of change of the sensor data is to be measured. Furthermore, a second set comprising second metrics is defined, which each indicate differently what the sensor data is disturbed with respect to. Then, any combination of a first metric of the first set and a second metric of the second set is selected, and an optimization problem is generated from the selected combination of the first metric and the second metric. Then, as in the method of the first embodiment, the optimization problem is solved by means of at least one solving algorithm, wherein the solution indicates the target disturbance of the input data 9. By means of the target disturbance, the disturbed input data 9 is generated for the neural network 11 from the sensor data.

[0122] Reference is made to Figure 3 Embodiments of an apparatus for generating a parameter set for a neural network will be described:

[0123] The apparatus comprises a database 8 with digital images. Reference is made to Figure 1 The described generator 10 is connected to this database 8. The neural network 11 is connected to the database 8 and the generator 10. The output of the neural network 11 is connected to a first analysis unit 12 and a second analysis unit 13. The first analysis unit 12 produces a first analysis by means of the neural network 11 on the basis of digital images supplied as input data to the neural network 11 from the database 8. The second analysis unit 13 produces a second analysis on the basis of the disturbed input data 9 supplied to the neural network 11 by the generator 10. For generating the disturbed input data 9, the generator 10 accesses the database 8 with digital images and applies the target disturbance generated by the generator 10 to this database.

[0124] The first analysis unit 12 and the second analysis unit 13 are connected to a comparison unit 14. The comparison unit is designed to compare the first analysis and the second analysis with one another.

[0125] The comparison unit 14 is connected to a parameter set generation unit 15. The parameter set generation unit 15 is designed to generate an improved parameter set for the neural network 11 on the basis of the comparison result of the first analysis and the second analysis transmitted from the comparison unit 14. The parameter set of the neural network 11 is generated by the parameter set generation unit 15 in such a way that the disturbed input data 9 generated by the generator 10 has little influence on the analysis of these input data by means of the neural network 11. In particular, the improved parameter set is generated in such a way that the influence of the disturbed input data 9 on the semantic segmentation of the digital images by means of the neural network 11 with respect to the disturbed input data does not lead to objects relevant to safety for a driver assistance system being incorrectly classified, disappearing or being displayed altered. The neural network 11 can thus be trained by means of the disturbed input data 9 produced by the generator 10.

[0126] Reference is made to Figure 4 An embodiment of a method according to the application will be described for checking the robustness of a neural network:

[0127] In step R1, a neural network with an associated parameter set is provided. This neural network is to be checked.

[0128] In step R2, training data is generated by means of a large number of digital images.

[0129] In step R3, the neural network is trained in a manner known per se using the training data, and a first analysis of the digital images is produced on the basis of the training data by means of the neural network.

[0130] In step R4, the first analysis is compared with a reference Figure 2 The method explained produces the disturbed input data as training data for the digital images.

[0131] In step R5, a second analysis of the digital images is generated on the basis of the disturbed input data, i.e. on the basis of the digital images to which the target disturbance has been applied, by means of the neural network.

[0132] In step R6, the first analysis and the second analysis are compared with one another.

[0133] In step R7, a robustness value is finally determined as a function of the comparison of the first analysis and the second analysis. The robustness value is high when the second analysis deviates less from the first analysis, in particular taking into account deviations which are relevant to the operation of the driver assistance system, in particular safety-critical deviations.

[0134] Reference is made to Figure 5 , a method for improving a parameter set of a neural network is described:

[0135] Steps R1 to R6 are first carried out as explained with reference to Figure 4 Subsequently, in step R8, an improved parameter set of the neural network is produced on the basis of the comparison and the second analysis.

[0136] List of reference signs

[0137] 1 first metric unit

[0138] 2 second metric unit

[0139] 3 third metric unit

[0140] 4 processing unit

[0141] 5 solution unit

[0142] 6 database with solution algorithms

[0143] 7 generation unit

[0144] 8 database of image data with numbers

[0145] 9 disturbed input data

[0146] 10 generator

[0147] 11 neural network

[0148] 12 first analysis unit

[0149] 13 second analysis unit

[0150] 14 comparison unit

[0151] 15 parameter set generation unit

[0152] 16 database with disturbances

Claims

1. A method for generating disturbed input data (9) for a neural network (11) for analyzing sensor data of a driver assistance system, wherein the sensor data are digital images, in which method: a first measure is defined, which indicates how the degree of change of the sensor data is measured and quantified, wherein the first measure compares the image distance between two digital images and outputs a value for the image distance, a second measure is defined, which indicates what the disturbance from the sensor data is directed against, wherein the second measure is directed against a change in a certain class of objects, and the objects are recognized and classified, wherein the first measure and the second measure measure the change in the sensor data of the digital images, wherein an optimization problem results from the combination of the first measure and the second measure, wherein the optimization problem comprises a loss function of the neural network, which contains the disturbance parameters and the images generated by the disturbance according to the second measure as parameters, and for which optimization problem the minimum of the disturbance parameters is sought on the condition that the degree of change of the generated images with respect to the initial images according to the first measure is below a certain value, solving the optimization problem by means of at least one solution algorithm, wherein the solution indicates the target disturbance of the input data (9), and the disturbed input data (9) is generated for the neural network (11) by means of the target disturbance, which is a changed digital image, which analyzes the digital images.

2. The method according to claim 1, characterized in that the second measure is directed against a change in the classification of the objects.

3. The method according to claim 1, characterized in that the disturbance described by the first measure and / or second measure is a naturally occurring disturbance.

4. The method according to claim 3, characterized in that the first measure and / or second measure are stored in a database (16), and a data set on the naturally occurring disturbances measured with the first measure and / or second measure is loaded from the database (16).

5. The method according to any of the preceding claims 1-4, characterized in that a third measure is defined, which indicates to which sensor data the disturbance is applied, and the optimization problem is generated from the combination of the first measure, the second measure and the third measure.

6. The method according to claim 5, characterized in that the third measure relates to all sensor data.

7. The method according to claim 5, characterized in that the third measure relates only to a subset of the sensor data.

8. The method according to claim 5, characterized in that the third measure describes only the sensor data containing a certain object.

9. The method according to any of the preceding claims 1-4, characterized in that the solution algorithm comprises an iterative method using the gradient of the neural network to determine the direction of change.

10. A method of generating perturbed input data (9) for a neural network (11) for analyzing sensor data of a driver assistance system, wherein the sensor data are digital images, in which method: a first set of first metrics is defined, which first metrics respectively and differently indicate how a degree of change of sensor data is measured and quantified, wherein a first metric of the first set compares an image distance between two digital images and outputs a value for the image distance, a second set of second metrics is defined, which second metrics respectively and differently indicate what the perturbation of sensor data is directed against, wherein one second metric of the second set is directed against a change of objects of a certain class, and the objects are recognized and classified, an arbitrary combination of first metrics of the first set and second metrics of the second set is selected, an optimization problem is generated from the selected combination of first and second metrics, wherein the optimization problem comprises a loss function of the neural network, which neural network comprises as parameters a perturbation parameter and images generated by the perturbation according to the second metrics, and for which optimization problem a minimum of the perturbation parameter is sought under the condition that a degree of change of the generated images with respect to the initial images according to the first metrics is below a certain value, solving the optimization problem by means of at least one solution algorithm, wherein the solution indicates a target perturbation of the input data (9), and by means of the target perturbation, the perturbed input data (9) is generated for the neural network (11) from the sensor data.

11. The method according to claim 10, characterized in that a third metric is defined, which third metric indicates which sensor data the perturbation is applied to, and an arbitrary combination of first metrics of the first set, second metrics of the second set and the third metric is selected, an optimization problem is generated from the selected combination of first, second and third metrics.

12. The method according to any one of claims 10 to 11, characterized in that a set of solution algorithms is defined, which comprises a plurality of solution algorithms, which respectively and differently solve the optimization problem in order to generate different target perturbations of the input data, an arbitrary solution algorithm of the set of solution algorithms is selected in order to generate the perturbed input data (9) for the neural network (11) from the sensor data.

13. A method for checking the robustness of a neural network (11) for analyzing sensor data with respect to perturbed input data (9), wherein the sensor data are digital images, in which method the following steps are performed: the neural network (11) is provided with a related set of parameters; training data is generated by means of an example-sensor data set; a first analysis of the example-sensor data set is generated by means of the neural network (11) on the basis of the training data; the perturbed input data (9) is generated by means of the method according to any one of claims 1 to 12 as training data of the example-sensor data set; a second analysis of the example-sensor data set is generated by means of the neural network (11) on the basis of the perturbed input data (9); the first analysis and the second analysis are compared; A robustness value is determined based on a comparison of the first and second analyses.

14. A method for improving a parameter set of a neural network (11) for analyzing sensor data, wherein the sensor data are digital images, in which method the following steps are performed: providing the neural network (11) with a related parameter set; generating training data by means of example-sensor data sets; generating a first analysis of the example-sensor data sets by means of the neural network (11) based on the training data; generating disturbed input data (9) by means of the method according to any one of claims 1 to 12 as training data of the example-sensor data sets; generating a second analysis of the example-sensor data sets by means of the neural network (11) based on the disturbed input data (9); comparing the first and second analyses; generating an improved parameter set for the neural network (11) based on a comparison of the first and second analyses.

15. A generator (10) for generating disturbed input data (9) for a neural network (11) for analyzing sensor data of a driver assistance system, wherein the sensor data are digital images, the generator (10) having: a first metric unit (1) with a first metric, which indicates how a degree of change of sensor data is measured and quantified, wherein the first metric compares an image distance between two digital images and outputs a value for the image distance; a second metric unit (2) with a second metric, which indicates what the disturbance of the input data (9) from the sensor data of a digital image is directed against, wherein the second metric is directed against a change of objects of a certain class, and the objects are recognized and classified, wherein the first metric and the second metric measure a change of sensor data of a digital image; a processing unit (4), which is connected to the first metric unit (1) and the second metric unit (2) and is designed to generate an optimization problem from the first metric and the second metric, wherein the optimization problem comprises a loss function of a neural network, which contains a disturbance parameter and a generated image by disturbance according to the second metric as parameters, and for which optimization problem a minimum of the disturbance parameter is sought under the condition that a degree of change of the generated image with respect to an initial image according to the first metric is below a certain value; a solving unit (5), which is connected to the processing unit (4) and is designed to solve the optimization problem by means of at least one solving algorithm, wherein the solution indicates a target disturbance of the input data (9) from the sensor data; and a generating unit (7), which is connected to the solving unit (5) and is designed to generate disturbed input data (9) for a neural network (11) by means of the target disturbance, which are changed digital images, which the neural network (11) analyzes.

16. A generator (10) for generating disturbed input data (9) for a neural network (11) for analyzing sensor data of a driver assistance system, wherein the sensor data are digital images, the generator (10) having: a first metric unit (1) with a first set of first metrics, which first metrics indicate differently each how a degree of change of sensor data is measured and quantified, wherein the first metrics compare an image distance between two digital images and output a value for the image distance; a second metric unit (2) with a second set of second metrics, which second metrics indicate differently each what the disturbance of the input data (9) from the sensor data is directed against, wherein the second metrics are directed against a change of objects of a certain class, and the objects are recognized and classified; a processing unit (4), which is connected to the first metric unit (1) and to the second metric unit (2) and is designed to generate an optimization problem from an arbitrary combination of the first metrics of the first set and the second metrics of the second set, wherein the optimization problem comprises a loss function of a neural network, which contains as parameters a disturbance parameter and an image generated by the disturbance according to the second metrics, and for which the minimum of the disturbance parameter is sought under the condition that the degree of change of the generated image with respect to an initial image according to the first metrics is below a certain value; a solving unit (5), which is connected to the processing unit (4) and is designed to solve the optimization problem by means of at least one solving algorithm, wherein the solution indicates a target disturbance of the input data (9) from the sensor data; and a generating unit (7), which is connected to the solving unit (5) and is designed to generate disturbed input data (9) for the neural network (11) from the sensor data by means of the target disturbance.

Images