METHOD AND DEVICE FOR EVALUATING AND CERTIFIING THE ROBUSTNESS OF AN AI-BASED INFORMATION PROCESSING SYSTEM

Description

[0001] The invention relates to a method and a device for evaluating and certifying the robustness of an AI-based information processing system.

[0002] Machine learning, for example based on neural networks, has great potential for application in modern driver assistance systems and automated vehicles. Functions based on deep neural networks process sensor data (for example, from cameras, radar, or lidar sensors) to derive relevant information. This information includes, for example, the type and position of objects in the vehicle's environment, the behavior of the objects, or the road geometry or topology.

[0003] A key feature in the development of AI-based information processing systems (the training) lies in purely data-driven parameter fitting without expert intervention. For example, in deep neural networks, the deviation of an output (for a given parameterization) from a ground truth is determined (the so-called loss). The loss function used is chosen in such a way that the parameters of the neural network depend on it in a differentiable manner. Within the framework of gradient descent, the parameters of the neural network are adjusted in each training step according to the derivative of the deviation (determined from several examples). These training steps are repeated very often until the loss no longer decreases.

[0004] This approach involves determining the parameters of an AI-based information processing system, particularly a neural network, without expert assessment or semantically motivated modeling. This can have significant consequences for the properties of the AI-based information processing system, especially the neural network.

[0005] In particular, deep neural networks are largely opaque to humans, and their calculations are not interpretable. This poses a significant limitation for systematic testing or formal verification.

[0006] Furthermore, deep neural networks are particularly susceptible to harmful interference, so-called adversarial perturbations: small manipulations of the input data, barely perceptible to humans or not altering their semantic content, can lead to completely different output data. Such manipulations can include both intentionally induced changes to the data ("neural hacking") and randomly occurring image changes (sensor noise, weather influences, certain colors or contrasts).

[0007] Furthermore, it is particularly unclear which input characteristics a neural network becomes sensitive to. This means that synthetically generated data, for example through simulation, can hardly be used successfully for training neural networks so far: neural networks trained in simulation or on other synthetic data exhibit surprisingly poor performance on real sensor data. Even running neural networks in a different data domain (training in summer, running in winter, etc.) sometimes drastically reduces their functional performance. This has the consequence, among others, that the possibility of developing and releasing neural networks in simulation (eliminating expensive labeling and time-consuming real-world testing), which sounds very attractive from a cost perspective, does not seem realistic.

[0008] The second point, in particular, is highly relevant to potential limitations of powerful neural networks in the area of ​​functional safety. To measure this, it is essential to assess the robustness of the network implementation against minor changes (augmentations) in the input data. Since such changes can be manifold (sensor noise, weather influences, image manipulation, semantically meaningless content changes, e.g., the wall color of background buildings), there is no single, universally accepted measure of robustness. Instead, numerous robustness values ​​must be measured against disturbances (i.e., augmentations) of varying types and intensities. Furthermore, the robustness of neural networks is not an absolute value but rather depends on the current input data.

[0009] Therefore, a method and device for the automatic evaluation and certification of the robustness of AI-based information processing systems is desirable, which can already check robustness values ​​during the development of AI-based functions and evaluate the effectiveness of robustness measures.

[0010] From DE 10 2018 218 586 A1 a method for generating robust automatic learning systems and testing trained automatic learning systems is known.

[0011] From Baidu Security X-Lab, "Tackling AI Challenges in Safety-Critical Scenarios - A Review on Robustness of Deep Learning Models and the Release of Perceptron Robustness Benchmark Tools," June 18, 2019, pages 1-8, URL: https: / / medium.com / baiduxlab / tackling-ai-challenges-in-safetycritical-scenarios-a-review-on-robustness-of-deeplearning-8e0e30ff1018, describes tools for assessing the robustness of deep learning models.

[0012] The invention is based on the objective of creating a method and a device for evaluating and certifying the robustness of an AI-based information processing system.

[0013] The problem is solved according to the invention by a method with the features of claim 1 and a device with the features of claim 10. Advantageous embodiments of the invention are set forth in the dependent claims.

[0014] In particular, a method according to claim 1 for evaluating and certifying the robustness of an AI-based information processing system is provided, wherein the AI-based information processing system provides a function for automated driving of a motor vehicle and / or for driver assistance of the motor vehicle and / or for capturing the environment of the motor vehicle and / or perceiving the environment of the motor vehicle or for another application, wherein at least one multidimensional data structure is received or generated as part of the AI-based information processing system, wherein in the at least one multidimensional data structure, specific difference values ​​between output data of the AI-based information processing system, obtained for data and augmented data, are determined at least by means of at least one difference measure, depending on at least the dimensions data set,Data augmentation definition(s) and difference measure definition(s) are stored, wherein at least one robustness of the AI-based information processing system is determined from at least one selection of the difference values ​​and compared with at least one robustness requirement, and wherein, based on a comparison result, the AI-based information processing system is either rejected, re-evaluated with a modified multidimensional data structure, or certified as robust.

[0015] Furthermore, in particular, a device according to claim 10 is provided for evaluating and certifying the robustness of an AI-based information processing system, wherein the AI-based information processing system provides a function for the automated driving of a motor vehicle and / or for driver assistance of the motor vehicle and / or for the detection of the environment of the motor vehicle and / or perception of the environment of the motor vehicle or for another application, comprising a data processing device, wherein the data processing device is configured to receive or generate at least one multidimensional data structure belonging to the AI-based information processing system, wherein in the at least one multidimensional data structure, at least by means of at least one difference measure, specific difference values ​​between output data of the AI-based information processing system,The goal is to determine at least one robustness of the AI-based information processing system from at least one selection of the difference values, depending on at least the dimensions dataset, data augmentation definition(s) and difference measure definition(s), and to compare this robustness with at least one robustness requirement based on a comparison result, and to either reject the AI-based information processing system, re-evaluate it with a modified multidimensional data structure or certify it as robust based on a comparison result.

[0016] The method and device enable the automated evaluation and certification of the robustness of an AI-based information processing system. This is achieved by receiving or generating a multidimensional data structure. Within this multidimensional data structure, at least one difference value is stored, representing at least one difference measure, of the difference values ​​between the output data of the AI-based information processing system as a function of at least the dimensions of data set, data augmentation definition(s), and difference measure definition(s). The difference values ​​for the data and augmented data are obtained.In other words, each data point of the multidimensional data structure contains a difference value that indicates the magnitude of a difference, determined using a predefined difference measure, between a result of the AI-based information processing system for the data (the dataset) and for data (the dataset) augmented using a data augmentation procedure. The data augmentation procedure may, for example, have added adversarial disturbances or noise to the data. From at least a selection, i.e., a subset, of the difference values ​​contained in the multidimensional data structure, at least one robustness value of the AI-based information processing system is determined and compared with at least one robustness requirement. Based on the comparison result, the AI-based information processing system is either rejected, re-evaluated with a modified multidimensional data structure, or certified as robust.

[0017] One advantage of the method and the device is that an AI-based information processing system can be fully automatically evaluated and / or certified. Specifically, it is intended that the corresponding multidimensional data structure is provided for an AI-based information processing system that is to be tested, evaluated, and / or certified for robustness. This multidimensional data structure may, for example, have been previously created during the development of the AI-based information processing system for various datasets, data augmentation methods, and / or difference measures.

[0018] The AI-based information processing system is intended to provide a function for automated driving of a motor vehicle and / or for driver assistance of the motor vehicle and / or for environmental sensing and / or environmental perception. In particular, it is intended that an AI-based information processing system, for example, a trained neural network, is loaded into the memory of at least one electronic control unit (ECU) and executed or applied there to evaluate and process sensor data acquired by sensors and, for example, to generate and provide control signals, such as for actuators. Additionally, at least one associated multidimensional data structure may also be stored in the memory.

[0019] Alternatively or additionally, the method and device can also be used in other applications. These could include, for example, automated fleet management, interior monitoring, driver observation, production control, video surveillance, robotics applications, automated flight, automated rail vehicles, or applications in space travel. Even in these applications, which are not explicitly covered by the claims, it is specifically intended that an AI-based information processing system is loaded into the memory of at least one control unit, and that the AI-based information processing system is executed by means of the at least one control unit, for example, to evaluate acquired sensor data and to generate and provide control signals, for example, for a production process and / or for at least one actuator.Additionally, at least one associated multidimensional data structure can be stored or already stored in the memory.

[0020] It is intended that the AI-based information processing system in an ECU will only be activated and / or executed if certification has been demonstrated, for example, by verifying the existence of a robustness certification. For this purpose, information is to be modified or stored in the ECU's memory, granting permission to activate or use the AI-based information processing system. Activation is to be effected by means of a corresponding result signal containing a certification, which is supplied to the ECU and triggers the activation within the ECU.

[0021] An AI-based information processing system is, in particular, an information processing system based on an artificial intelligence (AI) method. For example, the AI-based information processing system may be designed as a deep neural network or comprise at least one such network. In principle, however, the method described in this disclosure can also be used in other AI-based information processing systems, such as rule-based information processing systems. The at least one AI-based information processing system is, in particular, trained and / or finally parameterized. For example, the AI-based information processing system may be a trained neural network. An AI-based information processing system, in particular, comprises a structural description and parameters and / or is defined by a structural description and parameters.

[0022] A multidimensional data structure is defined, in particular, by at least the dimensions AI-based information processing system, dataset, data augmentation definition, and difference measure definition. The dimension AI-based information processing system includes, as its value range, all planned AI-based information processing systems (if more than one AI-based information processing system exists or is to be examined). If only one AI-based information processing system exists or is to be examined, the dimension AI-based information processing system can be omitted. The dimension dataset includes, as its value range, all planned datasets. The dimension data augmentation definition includes, as its value range, all planned data augmentation methods. The dimension difference measure definition includes, as its value range, all planned difference measures.Each combination of values ​​within these dimensions is assigned a data point that includes at least one difference value.

[0023] A dataset comprises, in particular, data. The data can be one-dimensional or multi-dimensional, especially two-dimensional. For example, the data could be images from a camera or a lidar sensor. In principle, however, any sensor data can be used.

[0024] A data augmentation definition specifically defines a data augmentation process or method. The data augmentation definition specifies how data within the dataset is to be modified. A wide variety of modifications are possible. Examples include: adding noise, adding one or more adversarial disturbances and / or sensor interference, changing contrast, brightness, colors, or weather conditions (e.g., adding snow or rain to a camera image captured in bright summer sunshine). A data augmentation process or method is designed and defined based on physical sensor characteristics (interference, etc.) and / or potential physical and / or technical malfunctions of the sensor and / or potential adversarial interference.

[0025] A difference measure definition, in particular, defines a difference measure. The difference measure specifies, specifically, how output data from an AI-based information processing system, generated for (non-augmented) data in the dataset, should be compared with output data from the AI-based information processing system generated for augmented data. For example, if the AI-based information processing system outputs a vector as output data, a difference measure could involve comparing the vectors, for example, by determining a difference between them. A simple example of another difference measure is as follows: If, for example, the AI-based information processing system outputs as output data how many pedestrians are present in a captured camera image, the respective numbers output for the data and the augmented data can be compared (e.g., by comparing the number of pedestrians).3 pedestrians versus 5 pedestrians, so the difference is equal to 2 pedestrians).

[0026] If a dataset contains temporally sequential data, a difference measure can also refer to temporally sequential, i.e., temporally adjacent, data. This allows a database to be created and provided for robustness assessment when processing video sequences (or other temporally sequential data) by an AI-based information processing system. For example, the robustness assessment can verify whether a pedestrian in a video sequence across multiple video frames is reliably recognized as a pedestrian by the AI-based information processing system.

[0027] The multidimensional data structure also contains, in particular, all artifacts relevant to its creation as metadata and / or headers. These artifacts include, for example: references to the software code used, references to the at least one AI-based information processing system and hyperparameters used for training, references to one or more datasets used (possibly including descriptive data), and / or initial values ​​used for random number generators ("random seeds").

[0028] Parts of the data processing equipment can be designed individually or collectively as a combination of hardware and software, for example as program code that is executed on a microcontroller or microprocessor.

[0029] In one embodiment, the at least one AI-based information processing system is a neural network and / or comprises at least one neural network. The neural network can, in particular, be a deep neural network, such as a convolutional neural network. A neural network includes, in particular, a structural description and parameters (e.g., filter parameters, weights, activation functions, etc.) of the neural network. The neural network is, in particular, a trained neural network.

[0030] In one embodiment, the AI-based information processing system is loaded into the memory of at least one electronic control unit (ECU) after certification. Specifically, a structural description and parameters of the AI-based information processing system are loaded into the memory of the ECU, enabling the ECU to subsequently execute or apply the system. The ECU is, in particular, an ECU of a motor vehicle. This allows the AI-based information processing system, certified as robust, to be used directly and / or automatically in the ECU. The ECU could, for example, be an ECU of a motor vehicle that evaluates sensor data and uses the AI-based information processing system to create an environmental perception (e.g.,performs object recognition) and in particular generates and provides control data, for example for at least one actuator.

[0031] In one embodiment, it is provided that, to provide a modified multidimensional data structure, at least one of the following dimensions is changed and / or extended: data set, data augmentation definition, difference measure definition. This allows a data basis for robustness assessment to be modified. In particular, this makes it possible to gradually increase the robustness requirements for the evaluation and certification of the AI-based information processing system.In particular, this allows the robustness of an AI-based information processing system to be increased (step by step) during its development. For example, variants of the AI-based information processing system deemed less robust can be discarded at the beginning of development, while variants deemed robust can be further evaluated and / or certified against additional robustness requirements. When modifying and / or extending the dataset, for instance, other data and / or data domains can be included. Furthermore, the dataset can be expanded to include additional data. Additional data augmentation definitions for further data augmentation methods can also be added.Alternatively or additionally, parameters of the data augmentation methods can be changed in the data augmentation definition. The difference measures can be modified or supplemented with further difference measures.

[0032] In one embodiment, the structure and / or parameters and / or training of the AI-based information processing system are modified based on the comparison result, and the modified AI-based information processing system is then re-evaluated and / or certified. This allows the robustness assessment to be used directly as feedback in the development of an AI-based information processing system, for example, via AutoML (Auto Machine Learning), where automatically modified neural networks or variants of neural networks can be generated and evaluated for robustness. The development and optimization of robust AI-based information processing systems can thus be automated.

[0033] In one embodiment, a set of several different AI-based information processing systems is evaluated, and the AI-based information processing system, particularly for loading and / or application in an electronic control unit (ECU), especially an ECU of a motor vehicle, is selected that best fulfills at least one robustness requirement with respect to at least one optimization criterion. This allows the development of robust AI-based information processing systems by generating variants of the AI-based information processing systems and subsequently selecting the most robust variant(s). For each AI-based information processing system, a multidimensional data structure corresponding to the AI-based information processing system under consideration is used.

[0034] In one embodiment, the comparison result is output for each of the at least one robustness requirement. This allows a detailed report to be generated and provided, so that the result of the assessment and / or certification of the robustness of the AI-based information processing system can be documented in detail.

[0035] In one embodiment, at least one modified AI-based information processing system is additionally evaluated, with the respective determined robustness of the AI-based information processing system and the at least one modified AI-based information processing system being compared. Based on the robustness comparison, a selection decision is made for the AI-based information processing system or the at least one modified AI-based information processing system. This allows for a stepwise adaptation of the AI-based information processing system, in which each change and / or adaptation is directly evaluated with regard to its effect on robustness and is either retained or discarded depending on the result.This allows the development of a robust AI-based information processing system to be carried out efficiently.

[0036] In one embodiment, the at least one multidimensional data structure belonging to the AI-based information processing system is provided by a control unit in whose memory the multidimensional data structure is stored, wherein the specific difference values ​​are received by querying and / or retrieving them from the control unit's memory. In particular, the specific difference values ​​are queried and / or retrieved and received depending on at least the dimensions of data set, data augmentation definition(s), and difference measure definition(s). The device can, in particular, include such a control unit or be connected to one if required.

[0037] Further features for the design of the device result from the description of embodiments of the method. The advantages of the device are the same in each case as in the embodiments of the method.

[0038] It is also proposed to use an AI-based information processing system, assessed as robust by means of the method described in this disclosure or by means of a device described in this disclosure, in particular certified, for the robust evaluation of sensor data from at least one sensor, in particular at least one sensor of a motor vehicle.

[0039] Furthermore, in particular a control unit is created, comprising a memory in which at least one AI-based information processing system, assessed as robust by means of a method according to one of the described embodiments, and in particular certified, is stored for the robust evaluation of sensor data from at least one sensor.

[0040] Furthermore, a computer program is created, in particular comprising instructions which, when the computer program is executed by a computer, cause it to execute the process steps of the method according to any of the described embodiments.

[0041] In addition, a data carrier signal is also created that transmits such a computer program.

[0042] If a multidimensional data structure for an AI-based information processing system to be evaluated and / or certified is not available or cannot be provided, the creation of a multidimensional data structure for the AI-based information processing system can be carried out in particular by means of the procedure described below for providing a database for the robustness assessment of at least one AI-based information processing system.

[0043] The generation is carried out by means of a method for providing a database for robustness assessment of at least one AI-based information processing system, wherein at least one AI-based information processing system, at least one data set, at least one data augmentation definition and at least one difference measure definition are received as input parameters, wherein a multidimensional data structure is generated based on the input parameters, wherein the dimensions and value ranges of the dimensions of the multidimensional data structure are determined by the received input parameters, and wherein each data point of the multidimensional data structure comprises a difference value determined by means of the at least one defined difference measure, which is determined by forming the at least one defined difference measure between input data.which were generated by the at least one AI-based information processing system for data of the at least one dataset and for the same data augmented by means of the at least one defined data augmentation, and wherein the generated multidimensional data structure is provided so that the robustness of the at least one AI-based information processing system can be assessed based on the difference values ​​encompassed by the multidimensional data structure.

[0044] For generation, the method is carried out, for example, by means of a device for providing a database for robustness assessment of at least one AI-based information processing system, comprising a data processing unit, wherein the data processing unit is configured to receive as input parameters at least one AI-based information processing system, at least one data set, at least one data augmentation definition, and at least one difference measure definition, and to generate a multidimensional data structure based on the input parameters, wherein the dimensions and value ranges of the dimensions of the multidimensional data structure are determined by the received input parameters, and wherein each data point of the multidimensional data structure comprises a difference value determined by means of the at least one defined difference measure, which is determined.by forming at least one defined difference measure between input data generated by the AI-based information processing system for data of the at least one dataset and for the same data augmented by means of the at least one defined data augmentation, and by providing the generated multidimensional data structure so that the robustness of the at least one AI-based information processing system can be assessed based on the difference values ​​encompassed by the multidimensional data structure.

[0045] The method and device for providing a database for robustness assessment of at least one AI-based information processing system enable the provision of such a database. For this purpose, a multidimensional data structure is created in which difference values ​​are stored as data points. The dimensions of the multidimensional data structure, which can also be referred to as a hypercube, include at least the following dimensions: AI-based information processing system, data set, data augmentation, and difference measure. Therefore, to generate the multidimensional data structure, at least one AI-based information processing system, at least one data set, at least one data augmentation definition, and at least one difference measure definition are received.For each combination of these dimensions, that is, for each possible value within the dimensions, a difference value is calculated using the respective difference measure and stored for the corresponding data point. The difference value defined by the respective difference measure definition is determined from the source data generated by the at least one AI-based information processing system for both (non-augmented) and augmented data of the dataset. The augmented data is generated using the data augmentation defined for the data point via the data augmentation definition.In other words, for each data point within the multidimensional data structure, a combination of at least the dimensions of AI-based information processing system, dataset, data augmentation definition, and difference measure definition is obtained. That is, for each data point, an AI-based information processing system, a dataset, a data augmentation (or a data augmentation process), and a difference measure are defined. Based on this, data from the dataset is augmented (e.g., distorted) for the data point using the data augmentation process, and a difference value between the non-augmented and augmented data of the dataset is determined by applying the AI-based information processing system to the data. The determined difference value is stored in the data point.This procedure is performed for all data points until a difference value has been determined and stored for each data point within the multidimensional data structure. The difference values ​​can later be retrieved from the data structure at any time by specifying the combination of at least the AI-based information processing system, the dataset, the data augmentation definition, and the difference measure definition. This allows the robustness of the at least one AI-based information processing system to be assessed based on the difference values ​​contained in the multidimensional data structure. This makes it possible to determine robustness at any time, even retrospectively, and to consider, for example, only a subset of the multidimensional data structure.This can improve the determination and evaluation of the robustness of at least one AI-based information processing system, especially with regard to flexibility.

[0046] One advantage of the method and device for providing a database for robustness assessment of at least one AI-based information processing system is that measuring and evaluating the robustness of an AI-based information processing system can be performed even without the AI-based information processing system itself, i.e., without a model description (structure, parameters, activation functions, etc.) and without the at least one data set. This is particularly advantageous if the AI-based information processing system and / or potentially sensitive data are not to be published or made available. For example, the method for providing the database enables the certification of an AI-based information processing system without the AI-based information processing system itself having to be part of the certification process.All that is required is the multidimensional data structure.

[0047] The procedure for providing a database for assessing the robustness of at least one AI-based information processing system can also be part of the procedure for evaluating and certifying the robustness of an AI-based information processing system.

[0048] During the receiving phase of the process for providing a database for robustness assessment of at least one AI-based information processing system, the structural description and parameters of the AI-based information processing system are received in particular. The AI-based information processing system is then executed on the data and the augmented data using the data processing unit.

[0049] Parts of the data processing equipment of the device for providing a database for robustness assessment of at least one AI-based information processing system can be designed individually or collectively as a combination of hardware and software, for example as program code that is executed on a microcontroller or microprocessor.

[0050] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, the provision includes an interface for the targeted retrieval of data points from the multidimensional data structure. Such an interface enables, in particular, the retrieval or querying of individual data points, ranges, or sets of data points depending on combinations of at least one AI-based information processing system, a data set, a data augmentation definition, and a difference measure definition. This allows for the flexible determination of robustness values ​​(especially robustness scores) and, for example, their averaging over multiple data points or multiple data augmentations or data augmentation methods.The procedure for evaluating and certifying the robustness of an AI-based information processing system can, in particular, work using this interface.

[0051] It can also be provided that the interface is configured to allow queries to be specifically restricted by additional parameters. Possible queries with specified parameters could include, for example, the following: Queries of specific entries (difference values) of the multidimensional data structure, queries of maximum, minimum, or average values ​​over sections of the multidimensional data structure, queries of variances, bandwidths, etc. of the difference values ​​in sections of the multidimensional data structure, queries of weighted sums (integrals) over sections of the multidimensional data structure, queries of histograms over the difference values ​​and / or the aforementioned values, queries of functional trends of the difference values ​​and / or the aforementioned values ​​under variation of section parameters, queries of heatmaps or other data visualization methods over the difference values ​​and / or the aforementioned values ​​or trends, queries of temporal trends of the difference values ​​and / or the aforementioned values ​​in the case of a temporally sequential data set.

[0052] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, the provision includes transmitting the multidimensional data structure to a certification service provider and / or a user of the AI-based information processing system and / or loading the multidimensional data structure into the memory of at least one control unit. This allows robustness to be determined even after delivery and use, or during the entire service life of the at least one AI-based information processing system in the field, for example, using new or modified robustness measures, without having to recalculate the difference values.

[0053] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, it is provided that a set of subparameters for the at least one data augmentation definition is received as input parameters. The generation of the multidimensional data structure and / or the augmentation of the data is then performed taking into account the received set of subparameters. This allows the data augmentation or the data augmentation method(s) to be further specified or parameterized. For example, in the case of brightness variations in camera images, a range of brightness values ​​can be specified within which the camera images are to be augmented, i.e., their brightness varied. The multidimensional data structure is expanded accordingly for each of the subparameters to be considered.With a brightness variation that has three sub-parameters (e.g., -20%, 0, and +20%), the data augmentation definition dimension then has three corresponding values ​​for the brightness variation. Another example is specifying different noise parameters (e.g., a target value for a signal-to-noise ratio, etc.).

[0054] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, it is provided that a set of filter criteria for individual input parameters is additionally received as input parameters, with the generation of the multidimensional data structure taking the received set of filter criteria into account. This allows for the targeted creation of a multidimensional data structure that considers the filter criteria. In particular, this makes it possible to model or prepare specific test scenarios. Filter criteria can, for example, be tags on the data that allow binning and / or resolution of a specific robustness based on the criteria. These can be, for example, context values ​​(e.g., for the contexts city, country, corner case, holiday, major nearby event, weather, etc.) or data properties (e.g.,(e.g., pollution, motion blur, blinding...).

[0055] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, it is provided that a selection of statistical distribution functions for parameter distributions for the at least one data augmentation definition is received as an additional input parameter, wherein the generation of the multidimensional data structure takes into account the selection of statistical distribution functions for parameter distributions. This allows statistical distributions to be considered during the augmentation (e.g., perturbation) of the data in the dataset. For example, for a data augmentation that includes a brightness variation in images, a distribution function can be provided from which statistical sampling is performed when generating the multidimensional data structure (e.g., 10 samples from the following distribution: [-30% to -10%] with p=0).3, [-10% to +10%] with p=0.5 and [+10% to +30%] with p=0.2).

[0056] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, it is provided that, as input parameters, a selection of distributions for combinations of parameters from data augmentation definitions and data from the dataset is received. The generation of the multidimensional data structure takes into account the received set of distributions for the parameter combinations. This allows distributions for the combinations to be considered. A distribution can, for example, be an 'exposure' to the individual input data streams or tags, which allows aggregation 'against' the distribution (i.e., summation over (the individual data) x (exposure)). In this way, a 'realistically expected robustness risk' can be calculated.

[0057] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, it is provided that a selection of relevance values ​​for individual data points is received as input parameters, with the generation of the multidimensional data structure taking the received relevance into account. This allows particularly relevant data points, i.e., particularly relevant combinations of input parameters, to be identified so that data points and difference values ​​can be generated or determined for these combinations. This is particularly advantageous if, for example, a robustness test for certain combinations of input parameters is legally required or has proven effective in robustness assessments.

[0058] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, it is provided that, for each data point, the results obtained by the at least one AI-based information processing system are additionally stored in the multidimensional data structure. This allows the results generated for the data and the augmented data to be used retrospectively. In particular, when the multidimensional data structure is extended with further data augmentations, it is not necessary to generate results again for the non-augmented data, but rather to directly access the results already stored in the multidimensional data structure. This saves computing power and processing time.

[0059] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, the multidimensional data structure is extended after provision by inserting at least one further dimension and / or by extending a value range of at least one dimension, with the extended multidimensional data structure being provided. This allows for the subsequent consideration of further or new data augmentation methods, further or new datasets, and / or further or new difference measures. In particular, the existing data points can be reused and do not need to be recalculated, since extending the multidimensional data structure is readily possible.

[0060] In one embodiment of the method and the device for providing a database for robustness assessment of at least one AI-based information processing system, it is provided that the at least one AI-based information processing system provides a function for automated driving of a motor vehicle and / or for driver assistance of the motor vehicle and / or for environmental sensing and / or environmental perception.

[0061] However, it is also possible, in principle, for the method and the device to be used in other applications. These could include, for example, automated fleet management, interior monitoring, driver observation, production control, video surveillance, robotics applications, automated flight, automated rail vehicles, or applications in space travel.

[0062] The invention is explained in more detail below with reference to preferred embodiments and the figures. These show: Fig. 1 a schematic representation of an embodiment of the device for evaluating and certifying the robustness of an AI-based information processing system; Fig. 2 a schematic representation of an embodiment of a device for providing a database for robustness assessment of at least one AI-based information processing system, with which a multidimensional data structure can be generated for the disclosed method; Fig. 3 a schematic flowchart to illustrate an embodiment of the method for providing a database for robustness assessment of at least one AI-based information processing system, with which a multidimensional data structure can be generated for the disclosed method; Fig. 4 a schematic flowchart of an embodiment of the method for evaluating and certifying the robustness of an AI-based information processing system.

[0063] InFig. 1 Figure 1 shows a schematic representation of an embodiment of the device 1 for evaluating and certifying the robustness of an AI-based information processing system. The AI-based information processing system can, for example, be a trained deep neural network. The device 1 comprises a data processing unit 2. The data processing unit 2 comprises a computing unit 3 and a memory 4. The device 1 is specifically configured to perform the method described in this disclosure for evaluating and certifying the robustness of an AI-based information processing system.

[0064] The AI-based information processing system is intended to provide a function for automated driving of a motor vehicle and / or for driver assistance of the motor vehicle and / or for environmental sensing and / or environmental perception or for another application.

[0065] The data processing unit 2 is configured to receive at least one multidimensional data structure 20 belonging to an AI-based information processing system to be reviewed, i.e., evaluated and / or certified. Specifically, a multidimensional data structure 20 is received. The received multidimensional data structure 20 is stored in memory 4 (schematically represented as a cube). The at least one multidimensional data structure 20 contains at least one difference value 22, determined by at least one difference measure, for differences between the output data of the AI-based information processing system, obtained for data and augmented data, depending on at least the dimensions of data set, data augmentation definition(s), and difference measure definition(s).

[0066] To evaluate and / or certify the AI-based information processing system, the computing facility 3 can access the difference values ​​22 stored in individual data points 21 of the multidimensional data structure 20, each of which corresponds to a combination of the aforementioned dimensions.

[0067] From at least a selection, that is, a subset, of the difference values ​​22, the data processing unit 2, in particular the computing unit 3, determines at least one robustness 25 of the AI-based information processing system. For this purpose, maximum values ​​or weighted or unweighted average values ​​can be determined, for example, based on the individual difference values ​​22. Furthermore, it is possible that the difference values ​​22 are statistically evaluated along one or more of the aforementioned dimensions, for example, by determining statistical measures of distributions or histograms, etc., and using them as a measure of the robustness 25.

[0068] The at least one specific robustness 25 is expressed in particular in the form of at least one robustness value. The at least one specific robustness 25 is compared with at least one robustness requirement 26 by means of the data processing device 2, in particular by means of the computing device 3. The at least one robustness requirement 26 is specified or supplied to the device 1 and received by the data processing device 2, in particular by the computing device 3, and stored in the memory 4.

[0069] Based on a comparison result, data processing unit 2, specifically computing unit 3, performs various actions. Either the AI-based information processing system is discarded, re-evaluated with a modified multidimensional data structure 20+, or certified as robust. In the first case, for example, a corresponding result signal 27 is output, indicating that the AI-based information processing system is being discarded or is to be discarded. This result signal 27 can, for example, be fed to a system 60 that optimizes AI-based information processing systems, such as neural networks, for robustness (e.g., using AutoML).In the second case, a corresponding result signal 27 is also output, containing information that the AI-based information processing system is to be re-evaluated with a modified multidimensional data structure 20+. Subsequently, such a modified multidimensional data structure 20+ is supplied to the device 1, and the measures described above are repeated with the modified multidimensional data structure 20+. In the third case, a corresponding result signal 27 is also output, containing information about the certification of the AI-based information processing system. This result signal 27, or the information about the certification of the AI-based information processing system, can, for example, be loaded into a memory of at least one control unit 30 in which the AI-based information processing system is to be executed.

[0070] It may be provided that the information on the certification of the AI-based information processing system includes a certificate of authenticity, for example in the form of a code or key.

[0071] It is intended that the AI-based information processing system in the control unit will only be activated or executed if certification has been demonstrated, for example, by verifying the existence of a robustness certification. For this purpose, information is to be modified or stored in the control unit's memory, granting permission to activate or use the AI-based information processing system. Activation is to be effected by means of a corresponding result signal 27, containing a certification, which is supplied to the control unit 30 and triggers the activation within the control unit.

[0072] It can also be provided that the AI-based information processing system, for example in the form of a structural description and / or parameters, is loaded into a memory of at least one control unit 30 after certification. The control unit 30 is, for example, a control unit 30 of a motor vehicle.

[0073] It may be necessary to provide a modified multidimensional data structure 20+ by changing and / or extending at least one of the following dimensions: data set, data augmentation definition, difference measure definition.

[0074] It may be provided that a structure and / or parameters and / or training of the AI-based information processing system is changed based on the comparison result, whereby the changed AI-based information processing system is re-evaluated and / or certified.

[0075] It may be provided that a set of several different AI-based information processing systems is evaluated, and that the AI-based information processing system, in particular for loading and / or application in an electronic control unit (ECU), especially an ECU of a motor vehicle, is selected that best fulfills at least one robustness requirement 26 with regard to at least one optimization criterion 28. It may be provided that a multidimensional data structure 20, generated for the several different AI-based information processing systems, is supplied to the device 1. This data structure additionally includes a dimension "AI-based information processing system" which has a range of values ​​corresponding to a number of the AI-based information processing systems. The optimization criterion 28 is also supplied to the device 1.An example of an optimization criterion 28 is, for example, the greatest possible distance between a comparison result 29 and at least one robustness requirement 26.

[0076] It may be provided that the comparison result 29 is output for each of the at least one robustness requirement 26.

[0077] It may be provided that at least one modified AI-based information processing system is additionally evaluated, wherein the respective determined robustnesses 25 of the AI-based information processing system and the at least one modified AI-based information processing system are compared with each other, and wherein, based on a comparison result of the robustnesses, a selection decision is made for the AI-based information processing system or the at least one modified AI-based information processing system. For this purpose, the provided multidimensional data structure 20 supplied to the device 1 comprises difference values ​​22 for both the AI-based information processing system and the at least one modified AI-based information processing system. In particular, certain robustness values ​​are compared with each other to compare the robustnesses 25.If, for example, increased robustness 25 is expressed in a larger robustness value, then the AI-based information processing system with the larger robustness value will be selected.

[0078] If a multidimensional data structure for an AI-based information processing system to be evaluated and / or certified is not available or cannot be provided, then the creation of a multidimensional data structure 20 for the AI-based information processing system can alternatively be carried out, in particular by means of the following with reference to the Fig. 2 described device and that with reference to the Fig. 3 The described procedures will be carried out.

[0079] In Fig. 2 Figure 1 shows a schematic representation of an embodiment of a device 1000 for providing a database for robustness assessment of at least one AI-based information processing system 10, which is configured, for example, as a trained neural network. In principle, however, the device 1000 can also be used for other AI-based information processing systems 10. It is intended that the AI-based information processing system 10 provides a function for automated driving of a motor vehicle and / or for driver assistance of the motor vehicle and / or for environmental sensing and / or environmental perception, or another application.

[0080] The device 1000 comprises a data processing unit 2. The data processing unit 2 comprises a computing unit 3 and a storage unit 4.

[0081] The data processing unit 2 receives as input parameters 9 at least one AI-based information processing system 10 (i.e., at least one neural network in the example), at least one data set 11, at least one data augmentation definition 12, and at least one difference measure definition 13; these are received by the data processing unit 2.

[0082] An AI-based information processing system 10 includes, in particular, a structural description as well as parameters (weightings, activation functions, filter parameters, etc.) of the AI-based information processing system 10. The data set 11 includes data, for example, two-dimensional camera images and / or other one- or multi-dimensional sensor data from at least one sensor (camera, lidar, radar, ultrasound, etc.). The at least one data augmentation definition 12 includes, in particular, a description of at least one data augmentation method, that is, a description of how data are to be augmented (e.g., distorted) within the framework of the method described in this disclosure.The at least one difference measure definition includes in particular a description of at least one difference measure, that is, a description of how the non-augmented data are to be compared with the augmented data within the framework of the procedure described in this disclosure or in what way a difference value 22 is to be determined.

[0083] Based on the input parameters 9, a multidimensional data structure 20 is generated by the data processing unit 2, in particular by the computing unit 3, and stored in memory 4, which is located in the Fig. 2 The multidimensional data structure 20 is schematically represented as a cube. The dimensions and value ranges of the dimensions are determined by the received input parameters 10, 11, 12, and 13 (in the described example, the multidimensional data structure 20 comprises four dimensions). To generate the multidimensional data structure 20, a data point 21 is created or determined for each combination of the input parameters 10, 11, 12, and 13. Each data point 21 of the multidimensional data structure 20 comprises a difference value 22 determined by the difference measure defined via the at least one difference measure definition 13.The difference value 22 is determined by the data processing unit 2, in particular by the computing unit 3, by forming the at least one defined difference measure between input data generated by the at least one AI-based information processing system 10, both for data of the at least one data record 11 and for the same data augmented by the data augmentation defined by the at least one data augmentation definition 12. For this purpose, the data processing unit 2, in particular the computing unit 3, executes the AI-based information processing system 10 on both the data and the augmented data. In a simple example, where the AI-based information processing system 10 provides, for instance, a number as input data, a difference measure could be, for example, a difference between the provided numbers.If vectors are output, differences between the vectors can be calculated, for example using a scalar product.

[0084] Once the difference values ​​22 for all data points 21 of the multidimensional data structure 20 have been determined, the generated multidimensional data structure 20 is provided. In particular, the multidimensional data structure 20 is output. Using the provided multidimensional data structure 20, the robustness of the at least one AI-based information processing system 10 can be assessed based on the difference values ​​22 encompassed by the multidimensional data structure 20 by means of the evaluation and certification procedure described in this disclosure.

[0085] With the help of the in the Fig. 2 device 1000 shown and of the one in the Fig. 3 The methods shown can be used to determine difference values ​​22 for different datasets 11, different data augmentation methods, and different difference measures. Furthermore, it is possible to compare different AI-based information processing systems 10 with one another. Each possible combination of the input parameters 9 corresponds to a data point 21 for which a difference value 22 is determined. The device 1000 and the associated method therefore allow the provision of a comprehensive and flexibly expandable database for the robustness assessment of at least one AI-based information processing system 10.

[0086] It may be provided that the provision includes the provision of an interface 5 for the targeted retrieval of data points 21 of the multidimensional data structure 20. The interface 5 can be implemented as hardware and / or as software.

[0087] Provisioning may involve transmitting the multidimensional data structure 20 to a certification service provider and / or a user of the AI-based information processing system 10 and / or loading the multidimensional data structure 20 into the memory of at least one control unit 30. This allows for subsequent assessment and / or certification of robustness.

[0088] It can be provided that, in addition to input parameter 9, a set of subparameters 14 is received for the at least one data augmentation definition 12, whereby the generation of the multidimensional data structure 20 and / or the augmentation of the data takes into account the received set of subparameters 14. The subparameters 14 include, for example, value ranges as input parameter 9 for a data augmentation function.

[0089] It may be provided that, as input parameter 9, an additional set of filter criteria 15 is received for individual input parameters 9, whereby the generation of the multidimensional data structure 20 takes into account the received set of filter criteria 15.

[0090] It may be provided that, as input parameter 9, an additional selection of statistical distribution functions 16 for parameter distributions is received for at least one data augmentation definition 12, whereby the generation of the multidimensional data structure 20 takes into account the selection of statistical distribution functions 16 for parameter distributions.

[0091] It may be provided that, as input parameter 9, an additional selection of distributions 17 for combinations of parameters of data augmentation definitions 12 and data of the data set 11 is received, whereby the generation of the multidimensional data structure 20 takes into account the received set of distributions 17 for the combinations of the parameters.

[0092] It can be provided that a selection for a relevance 18 of individual data points 21 is received as input parameter 9, whereby the generation of the multidimensional data structure 20 takes into account the received relevance 18.

[0093] It may be provided that, in addition, for each data point 21, the results 23 generated by the at least one AI-based information processing system 10 are stored in the multidimensional data structure 20.

[0094] It may be provided that the multidimensional data structure 20 is extended after provision by inserting at least one further dimension and / or by extending a value range of at least one dimension, whereby the extended multidimensional data structure 20+ is provided.

[0095] It is specifically intended that at least one AI-based information processing system 10 provides a function for automated driving of a motor vehicle and / or for driver assistance of the motor vehicle and / or for environmental detection and / or environmental perception.

[0096] In Fig. 3 A schematic flowchart is shown to illustrate an embodiment of the method for providing a database for the robustness assessment of at least one AI-based information processing system 10.

[0097] As input parameters 9, at least one AI-based information processing system 10, for example a trained neural network, at least one data set 11, at least one data augmentation definition 12, and at least one difference measure definition 13 are specified. Based on this, at least one AI-based information processing system 10a, data 11a from at least one data set 11, at least one data augmentation method 12a, and at least one difference measure 13a are specified.

[0098] In a process step 100, the data 11a are augmented using at least one data augmentation method 12a, e.g., the data 11a are disturbed by adding noise or at least one adversarial disturbance. It may be provided that parameters for the data augmentation are randomly selected in a preceding process step 90.

[0099] In process step 101, the at least one AI-based information processing system 10 is applied to the non-augmented data 11a and the respective augmented data. The results obtained are then compared date by date in process step 102, and a difference value is determined for each date using the at least one difference measure 13a. For multiple AI-based information processing systems 10, multiple datasets 11, multiple data augmentation methods 12a, and / or multiple difference measures 13a, this is performed for each possible combination, so that a difference value is determined for each combination. A multidimensional data structure 20 is generated from the difference values, with each difference value being assigned to individual data points, each of which is associated with a possible combination of the input parameters 9.

[0100] The multidimensional data structure 20 generated in this way is provided, for example in the form of an interface 5, by means of which the difference values ​​for any combination of the input parameters 9 can be queried or retrieved, so that the robustness of the at least one AI-based information processing system 10 can be assessed based on the difference values ​​encompassed by the multidimensional data structure 20.

[0101] The determination of robustness is schematically illustrated in process steps 200 to 202. In process step 200, difference values ​​are retrieved according to various filter criteria, which are derived from a predefined robustness measure. The retrieved difference values ​​are aggregated in process step 201, for example, by statistically or otherwise evaluating them, so that (statistical) key performance indicators (e.g., in the form of Key Performance Indicators 40, or aggregated difference measures, e.g., "average error" = 0.5 or 0.4, etc.) can be calculated and provided (distribution parameters, etc.). This aggregation is performed, for example, across one or more dimension axes of the multidimensional data structure 20, e.g., across a complete dataset 11 and / or across all data augmentation methods 12a and / or across all difference measures 13a.For example, maximum values ​​and / or unweighted or weighted average values ​​can be calculated and provided.

[0102] In a process step 202, the key figures and / or aggregated values ​​generated in this way can be visualized, for example, by displaying the results as a graph 41, creating histograms 43, displaying the scalar values ​​44, generating heatmaps 45, and / or displaying a pie chart 46. This allows for an improved evaluation and certification of the robustness 25 of the at least one AI-based information processing system 10. Furthermore, a report 42 can be generated and output containing the results of a robustness evaluation and / or certification.

[0103] In particular, it is possible that providing the multidimensional data structure 20 includes providing an interface 5 for the targeted retrieval of data points from the multidimensional data structure 20. A query may include, in particular, the following parameters: desired filter criteria 50, desired axis or dimension selection 51, desired aggregation method(s) 52, and desired visualization method 53.

[0104] In this way, in particular a metric generator 300 can be created with which AI-based information processing systems 10, for example trained neural networks, can be evaluated and certified in a comparable and repeatable manner with regard to robustness 25.

[0105] Providing the multidimensional data structure 20 makes it possible, in particular, to assess the robustness 25 of an AI-based information processing system 10 even without the data 11a and without the AI-based information processing system 10 itself. This is especially advantageous for sensitive data.

[0106] In Fig. 4 Figure 1 shows a schematic flowchart of an embodiment of the method for evaluating and certifying the robustness 25 of an AI-based information processing system 10, for example, a (trained) neural network. The method is embedded here as an example in a method for the automated generation, evaluation, and / or certification of AI-based information processing systems 10.

[0107] The starting point is an AI-based information processing system 10, for example, a (trained) neural network that was created and trained in a generation process 70 ("Model Factory"). The creation and training of the AI-based information processing system 10 can be automated (e.g., using AutoML) or "manually" by a development team.

[0108] Furthermore, verification requirements 80 are specified. These requirements define, in particular, how the AI-based information processing system 10 is to be evaluated, that is, which data 11a, data augmentation methods 12a, and / or which difference measures 13a are to be considered, and, via a report definition 81, which results of the evaluation are to be included in an output or report. The verification requirements 80 also include robustness requirements 26, that is, for example, robustness thresholds 25 that must not be exceeded if the AI-based information processing system 10 is to be certified as robust.

[0109] The AI-based information processing system 10, the data 11a, the data augmentation procedure(s) 12a and / or the difference measure(s) 13a and the report definition 81 are referred to with reference to the Fig. 3 The previously described metric generator 300 is passed to the metric generator. The metric generator 300 delivers as a result at least a robustness 25 of the AI-based information processing system 10 (e.g., in the form of the in Fig. 3 Key Performance Indicators shown (40), by using the multidimensional data structure (20) Figuren 1 , 2 and 3 The difference values ​​contained in 22 will be evaluated.

[0110] In process step 400, the determined minimum robustness 25 is compared with the minimum robustness requirement 26. Based on the comparison result, the AI-based information processing system 10 is either rejected, re-evaluated with a modified multidimensional data structure by the metric generator 300, or certified as robust. As a result of process step 400, a result signal 27 is returned to the generation process 70 so that the AI-based information processing system 10 is either rejected, re-evaluated, or certified.

[0111] It may be provided, in particular, that the AI-based information processing system 10 is loaded into the memory of at least one control unit after certification. Additionally, it may be provided that the multidimensional data structure is loaded into the memory of the control unit.

[0112] The method described in this disclosure for evaluating and certifying the robustness of an AI-based information processing system 10 enables the fully automated development, evaluation, and / or certification and deployment of AI-based information processing systems 10, such as neural networks. This automation saves time, effort, and costs. Furthermore, the use of the multidimensional data structure during evaluation improves the reproducibility, documentation, and comparability of the evaluation and certification process. Reference symbol list

[0113] 1 Device 2 Data processing device 3 Computing device 4 Memory 5 Interface 9 Input parameter 10 AI-based information processing system 11 Data record 11a Data 12 Data augmentation definition 12a Data augmentation method 13 Difference measure definition 13a Difference measure 14 Subparameter 15 Filter criterion 16 Distribution function 17 Distributions 18 Relevance 20 Multidimensional data structure 20 + Modified / extended multidimensional data structure 21 Data point 22 Difference value 23 Inferred result 25 Robustness 26 Robustness requirement 27 Result signal 28 Optimization criterion 29 Comparison result 30 Control unit 40 Key Performance Indicator 41 Graph 42 Report 43 Histogram 44 Scalar value 45 Heatmap 46 Pie chart 50 Desired filter criterion 51 Desired axis or...Dimension selection 52 Desired aggregation method 53 Desired visualization method 60 System 70 Generation process 80 Verification requirements 81 Report definition 90 Process step 100-102 Process steps 200-202 Process steps 300 Metric generator 400 Process step 1000 Device.

Claims

1. Computer-implemented method for assessing and certifying the robustness (25) of an Al-based information processing system (10), wherein the Al-based information processing system (10) provides a function for automated driving of a motor vehicle and / or for driver assistance of the motor vehicle and / or for capturing the environment of the motor vehicle and / or perceiving the environment of the motor vehicle or for another application for evaluating captured sensor data of at least one sensor of the motor vehicle and for generating control signals, wherein at least one multidimensional data structure (20) is received or generated that belongs to the Al-based information processing system (10), wherein in the at least one multidimensional data structure (20), at least by means of at least one difference measure (13a), determined difference values (22) between source data of the Al-based information processing system (10) obtained for non-augmented data (11a) and output data of the Al-based information processing system (10) obtained for augmented data are stored on the basis of at least the dimensions data set (11), data augmentation definition(s) (12) and difference measure definition(s) (13), wherein at least the robustness (25) of the Al-based information processing system (10) is determined from at least a selection of the difference values (22) and compared with at least one robustness requirement (26), and wherein, on the basis of a comparison result (29), the Al-based information processing system (10) is either discarded, reassessed with a modified multidimensional data structure (20+) or certified as robust, wherein the Al-based information processing system (10) is only activated and / or executed in a control unit (30) of the motor vehicle if certification has been demonstrated, wherein, for this purpose, information is modified or stored in the memory of the control unit (30) which includes permission to activate or use the Al-based information processing system (10), and wherein the activation is effected by means of a result signal (27), containing the certification, which is supplied to the control unit (30) and which causes the activation in the control unit (30).

2. Method according to claim 1, characterized in that the Al-based information processing system (10) is loaded into a memory of at least one control unit (30) after certification.

3. Method according to claim 1 or 2, characterized in that in order to provide a modified multidimensional data structure (20+), at least one of the following dimensions is modified and / or extended: data set (11), data augmentation definition (12), difference measure definition (13).

4. Method according to any of the preceding claims, characterized in that a structure and / or parameters and / or training of the Al-based information processing system (10) is modified on the basis of the comparison result (29), the modified Al-based information processing system (10) being reassessed and / or re-certified.

5. Method according to any of the preceding claims, characterized in that a set of multiple different Al-based information processing systems (10) is assessed, the Al-based information processing system (10) that best meets the at least one robustness requirement (26) with respect to at least one optimization criterion (28) being selected.

6. Method according to any of the preceding claims, characterized in that the comparison result (29) is issued for each of the at least one robustness requirements (26).

7. Method according to any of the preceding claims, characterized in that additionally, at least one modified Al-based information processing system (10) is assessed, the respective determined robustnesses (25) of the Al-based information processing system (10) and of the at least one modified Al-based information processing system (10) being compared, and, on the basis of a comparison result of the robustnesses (25), a selection decision being made for the Al-based information processing system (10) or the at least one modified Al-based information processing system (10).

8. Method according to any of claims 1 to 7, characterized in that the Al-based information processing system (10) is a neural network and / or comprises at least one neural network.

9. Method according to any of claims 1 to 8, characterized in that the at least one multidimensional data structure (20) belonging to the Al-based information processing system (10) is provided by means of a control unit, in the memory of which the multidimensional data structure (20) is stored, the determined difference values (22) being received by querying and / or retrieving the determined difference values (22) from the memory of the control unit.

10. Device (1) for assessing and certifying the robustness (25) of an Al-based information processing system (10), wherein the Al-based information processing system (10) provides a function for automated driving of a motor vehicle and / or for driver assistance of the motor vehicle and / or for capturing the environment of the motor vehicle and / or perceiving the environment of the motor vehicle or for another application for evaluating captured sensor data of at least one sensor of the motor vehicle and for generating control signals, comprising a data processing device (2), wherein the data processing device (2) is configured to receive or generate at least one multidimensional data structure (20) belonging to the Al-based information processing system (10), wherein, in the at least one multidimensional data structure (20), at least by means of at least one difference measure (13a), determined difference values (22) between source data of the Al-based information processing system (10) obtained for non-augmented data (11a) and output data of the Al-based information processing system (10) obtained for augmented data are stored on the basis of at least the dimensions data set (11), data augmentation definition(s) (12) and difference measure definition(s) (13), to determine at least the robustness (25) of the Al-based information processing system (10) from at least a selection of the difference values (22) and to compare it with at least one robustness requirement (26), and, on the basis of a comparison result (29), either to discard the Al-based information processing system (10), to reassess it with a modified multidimensional data structure (20) or to certify it as robust, wherein the Al-based information processing system (10) is only activated and / or executed in a control unit (30) of the motor vehicle if certification has been demonstrated, wherein the data processing device (2) is further configured to modify or store, for this purpose, information in the memory of the control unit (30) which includes permission to activate or use the Al-based information processing system (10), and to effect the activation by means of a result signal (27) which contains the certification, which is supplied to the control unit (30) and which causes the activation in the control unit (30).

11. Computer program comprising commands which, when the computer program is executed by a computer, cause said computer to execute the method steps of the method according to any of claims 1 to 9.

12. Data carrier signal which transmits a computer program according to claim 11.