Inspection system, method and apparatus for selecting authentic tests and testing systems

Abstract

The invention relates to a method and a device for selecting a real test and a test system. The method (10) for a test system (20) is characterized by the following features: - input parameters (Z) of the system (20) are divided (11) into a first group (X) and a second group (Y), - a first selection (21) is made (12) in the case of the input parameters of the first group (X) being occupied, according to a first method, - a second selection (22) is made (13) in the case of the input parameters of the second group (Y) being occupied, according to a second method, - a characteristic variable (23) is calculated (14) from the second selection (22), and - the first selection (21) is adjusted (15) in dependence on the characteristic variable (23).

Description

TECHNICAL FIELD

[0001] The present application relates to a method for testing a system. In addition thereto, the present application relates to a corresponding device, a corresponding computer program and a corresponding storage medium. BACKGROUND

[0002] In software technology, the use of models for automating test activities and for generating test artifacts in the course of testing is summarized under the umbrella concept "model-based testing (MBT)". It is well known, for example, to generate test cases from models that describe the target behavior (Sollverhalten) of the system to be tested.

[0003] In particular, embedded systems rely on sensoric input signals that are decisive for the system and, in turn, simulate their environment by output signals to very different actuators. In the course of the validation and pre-development phase of such systems, therefore in a regulation loop (Regelschleife), models of the system (model in the loop, MiL), software (software in the loop, SiL), processors (processor in the loop, PiL) or entire hardware (hardware in the loop, HiL) are simulated together with models of the environment. In vehicle technology, simulators for testing electronic control devices corresponding to this principle are sometimes called component test benches, module test benches or integration test benches, depending on the test phase and the test object.

[0004] DE 103 03 489 Al discloses a method for testing software of a control unit of a vehicle, a power tool or a robot system, wherein at least a part of a regulation section (Regelstrecke) controllable by the control unit is simulated by a test system in such a way that output signals are generated by the control unit and these output signals of the control unit are transmitted via a first connection to a first hardware component and signals of a second hardware component are transmitted as input signals to the control unit via a second connection, wherein the output signals are provided in the software as first control values and additionally transmitted in real time with respect to the regulation section to the test system via a communication interface.

[0005] Such simulations are popular in different technical fields and are, for example, employed in order to verify the suitability of embedded systems in electric power tools, in engine control devices of drive systems, steering systems and brake systems, in camera systems, in systems with components of artificial intelligence and machine learning, in robotic systems or in autonomous vehicles at an early stage of their development. Nonetheless, the results of simulation models according to the prior art are only limitedly included in approval decisions due to a lack of trust in their reliability. SUMMARY

[0006] The present application provides a method for verifying a system, a corresponding device, a corresponding computer program and a corresponding storage medium according to the independent claims.

[0007] The solution is advantageous in that the combination according to the present application of a classical test, which strives for properties in the worst case, and a statistical or probabilistic method, which provides a more comprehensive benchmark for the system, on the one hand, and on the other hand. With this method, it is possible to select tests which are performed in a physical (real) environment or only virtually (in a simulation). Furthermore, the method can be used to search for critical test scenarios (or further environmental and input conditions) and to estimate the global performance of autonomous vehicles, to test functions learned by machine learning and image processing algorithms and to generate training data for machine learning and vision (computer vision).

[0008] In the following, the term "verification" is used as a synonym for testing and describes the terms "testing", "search-based testing" and "uncertainty quantification".

[0009] The approach according to the present application is based on the insight that rigorous testing is required in order to guarantee the reliability and safety of complex systems, such as, for example, autonomous vehicles. The system to be tested (system under test, SUT) is run under determined environmental conditions and with different inputs. In the following, the term "input" is used not only for the direct inputs of the SUT, but also for variables which describe the environmental conditions under which the SUT is run. The SUT can be run either in a physical structure (real environment) or in a model of the physical structure, i.e. in the framework of a simulation.

[0010] The purpose of such a test is to search for inputs or environmental conditions (subsequently summarized as "inputs") of the SUT, in which the SUT does not meet its requirements regarding the desired properties, or in which the performance of the SUT is poor or as low as possible. If the test does not indicate such critical inputs or environmental conditions, it is the starting point that the SUT meets its requirements regarding the desired properties, or that the performance of the SUT in the worst case is known. The possible (in the sense of valid or allowed) input range and environmental conditions can be limited before or after the test, and the final result is adapted to all inputs.

[0011] Furthermore, in the context of search-based testing (SBT), as an automatic test generation method, a proposed method has been developed, in which optimization techniques are used in order to select the closest test input each time. In this case, existing optimization algorithms (for example, a Bayesian optimizer) generate inputs for the SUT, whose performance is evaluated by a performance monitor, with the goal of minimizing the performance of the SUT.

[0012] In contrast to classical testing, statistical probability methods, such as uncertainty quantification (UQ), do not focus on the performance of the SUT in the worst case alone, but rather, more precisely, attempt to evaluate the overall performance of the SUT, taking into account the uncertainty and randomness of the inputs, including possible environmental conditions. The test inputs for the SUT are derived based on a determined probability distribution, which can be given explicitly (for example, in terms of the mean and standard deviation of a Gaussian process) or implicitly by a determined environmental structure and its parameterization. The output is a probability distribution in the form of a histogram, which summarizes the performance of the SUT. In this case, the probability is only valid if the explicit or implicit input sampling distribution has been correctly selected. By specifying a threshold value for the performance (and thus a definition of the requirements), UQ can state the probability that the SUT meets its requirements.

[0013] The first challenge is that testing of a system in the physical (real) environment is laborious. Rigorous testing in the physical environment can even be impossible for time reasons or safety reasons. Thus, a method for testing a system in a simulated (virtual) environment is considered.

[0014] The approach according to the invention recognizes in this context the impossibility to dispense with all physical tests. At a given time, the simulation environment itself has to be validated, calibrated and has to be measured and the differences and divergences between the physical and the virtual environment have to be considered in the overall approach. Taking into account the effects of the uncertainties about the model parameters, the approach makes it easy to select or prioritize tests of the kind that have to be performed in the real environment. According to the invention, the selection of tests to be repeated in the real environment is made by simulation only.

[0015] Known techniques for selecting real test cases either make use of predefined sampling strategies or calculate measurement uncertainties. The described approach selects test cases based on the properties of the simulation model given the uncertainties about the model parameters.

[0016] The approach also solves other problems that are not directly related to the differences between real and virtual tests described below: In machine learning, the existence of so-called adversarial examples is a second challenge. Adversarial examples are small changes in the input that lead to an undesired output. In the case of two images of a car that only slightly differ in a few pixel values and look identical to a human being, a neural network for example classifies one of the images as a car and the other one as a different object.

[0017] Current machine learning algorithms are not resistant to adversarial examples and effective methods for generating adversarial examples are known. A related generator (adversarial example generator, AEG) generates an input A' from an input A for which a given neural network produces a correct output, in the case of which input A' the same network produces an incorrect output. In the classical test in which it is the aim to find errors, the AEG thus solves the test problem to find a "successful test" in the sense of an equally effective input A' for the test input A, in the case of which the SUT fails. Traditionally, it can thus be concluded that the SUT does not meet its requirements and has to be corrected or even if the error is unacceptable, the machine learning does not work at all.

[0018] The approach according to the invention recognizes that the view of classical testing for machine learning based applications is too strict because even if such, by chance, an error is discovered, the probability can be very low or insignificant to construct it according to the AEG method. Probabilistic statistical methods calculate the "average case behavior" which is not sufficient for applications that are critical to safety.

[0019] The described approach combines analysis of worst case and average case in order to achieve a suitable trade-off in this context and to find most critical test scenarios or test cases. A test scenario in this sense is a (sometimes very broad) test space.

[0020] This test space grows exponentially with the number of input parameters of the SUT and its environment. A third challenge is the testing or analysis of a system with so many inputs.

[0021] The basic idea explained in the independent claim can advantageously be improved and refined by the measures set out in the dependent claims. BRIEF DESCRIPTION OF DRAWINGS

[0022] Embodiments of the invention are illustrated in the attached drawings, and will be described in more detail in the following description. In the drawings:

[0023] Figure 1 A flow chart of a method according to a first implementation form is shown.

[0024] Figure 2 The approach according to the invention is schematically shown.

[0025] Figure 3 A workstation according to a second implementation form is shown. DETAILED DESCRIPTION

[0026] Figure 1 The method according to the invention (10) is now to be explained in terms of a block diagram according to Figure 2 The method provides for splitting (process 11— Figure 2 ) a set of input parameters Z (reference numeral 20— Figure 2 ) of the SUT and an environment (reference numeral 27— Figure 1 ) of the SUT into two parameter groups X and Y and then checking the two parameter groups according to two methods A and B. Method A is a worst case focused test method which forms (process 12— Figure 1 ) a sample (reference numeral 21— Figure 2 ) of values for X, while method B is a probabilistic method which forms (process 13— Figure 1 ) a sample (reference numeral 22— Figure 2 ) of values for Y. From the selection (22), a statistical characteristic variable (reference numeral 23— Figure 1 ) is calculated (process 14— Figure 2 ), which is in turn used to influence (process 15— Figure 1The choice of X. As a result, probabilistic method B therefore calculates the mathematical projection of Z onto X, which is used by method A.

[0027] Experts therefore divided the parameter Z into two parameter groups, X and Y, as described above. Typically, but not always, the number of parameters X is less than that of Y, i.e. Parameter X represents the boundary conditions (see attached figure 24—). Figure 2 The parameter Y is determined by the following constraints (see attached figure 25—). Figure 2 The constraints, in this regard, may include hard boundary conditions or distributions, which may be explicitly given in advance as a probability distribution function (PDF), or implicitly given in advance via sampling methods (e.g., environmental conditions).

[0028] This method can be summarized by the following algorithm:

[0029] r1 = []

[0030] Solange nicht A_TestEndeX(r1):

[0031] x = A_GenTestX(r1, XRandbedingungen)

[0032] r2 = [] / / Empty list

[0033] Solange nicht B_TestEndeY(r2):

[0034] y = B_GenStichprobeY(r2, YBeschränkungen)

[0035] r2 = r2. anhängen( VollständigesSUT(x,y)) ) )

[0036] r1 = Statistik(r2, x)

[0037] endergebnis=sortieren(r1).

[0038] The candidates for method A (A_TestEndeX, A_GenTestX) are the search-based tests mentioned above. The candidates for B (B_TestEndeY, B_GenStichprobeY) are the uncertainty quantifications described above.

[0039] The function “VollständigesSUT” (see attached figure 26—) Figure 2 The evaluation function (28) of the SUT (20) together with its virtual environment (27), possible interference models, and characteristics or outputs of the SUT can be, for example, in the form of a performance monitor, test criterion, or simply an output signal selector. However, the sub-components (27, 28) of the simulation (26) are optional, except for the SUT (20) itself.

[0040] The function “Statistik” (see attached figure 23—) Figure 2 The result r2 is a combination of the results for a fixed x and a variable y; this is to be understood as the projection of y onto the current x. Examples of suitable feature variables (23) are the minimum, mean, expected value, standard deviation, difference between the maximum and minimum values, or error probability. The variable r1 is a list of tuples or other data structures that associate each value x with the corresponding statistical result.

[0041] The functions “A_TestEndeX” and “B_TestEndeY” can be defined, for example, according to the following pseudocode: “|r1|< MaxSamplesA” and “|r2| < MaxSamplesB”. More complex methods (such as coverage-based methods) are also possible.

[0042] The statistical evaluation (23), along with the associated parameter occupancy (Parameterbelegung) X, is summarized in a function (Figure 29) and presented to the user as a result. The function's performance may be, for example, the screening, selection, or visualization of test cases based on the calculated statistics.

[0043] The final result is a sieved list of statistical results, which defines the priority order of the test scenarios for X.

[0044] The algorithm effectively searches for the assignment of X in which the variation of Y yields the most unfavorable statistical value, or in which the statistical sensitivity of the model is maximized. Since x is contained within the complete test space Z, it can be understood as a test scenario with variable parameters Y.

[0045] In view of the first of the challenges outlined above, the parameter X is typically an input which can be controlled without problems in real tests, that is to say is to some extent a "free" parameter, such as the steering deflection (Lenkeinschlag) or acceleration of a car. The parameter Y is typically difficult to control in the meantime (taking into account wheel friction, engine temperature or wind conditions), but the starting point is that the parameter is also taken into account in the simulation model (26). The output of the algorithm is a prioritization of test scenarios for the real environment, which are to be regarded as the most critical test scenarios as a result of the statistics used.

[0046] In view of the second challenge, the application of machine vision is observed, for example in the case of autonomous driving. The input of the relevant algorithm is typically an image, and the output of the algorithm corresponds to a classification of the objects visible in the image. Here too, the following is observed: the input to the algorithm originates from the environment (27), which is either simulated by means of three-dimensional computer graphics or can be recorded with a camera in reality.

[0047] In this case, the user selects a parameter X which describes the scenario, for example in terms of road constellation, objects in the image or time of day. In addition, the user selects a parameter Y which can vary in any scenario, for example camera position and camera orientation, intrinsic camera parameters and the position and orientation of the objects in the scenario. The variation of the parameter Y can in this case be used as a calculation of the probability of the occurrence of adversarial samples in the scenario.

[0048] The algorithm according to the application provides the scenarios which are most critical with regard to variations in Y. In this way, the safety of different operating ranges of autonomous vehicles can be determined or evaluated.

[0049] In view of the third challenge, test problems with many (for example 50) parameters are difficult due to the so-called state space explosion. The described scheme helps to solve this problem in that Z is split into such a way that |X| « |Y|, for example |X| = 5 and |Y| = 45. The user selects the most important parameters as X and the less important parameters as Y. The scheme enables the parameters X and Y to be treated according to two different sampling methods, and the results of the Y variations to be projected onto the X space. In this way, a coarse analysis of the Y space is performed and a detailed analysis of the X space is performed.

[0050] The method (10) can be implemented, for example in a workstation (30), for example in software or hardware, or in a hybrid of software and hardware, as explained in the schematic diagram of Figure 3 .

Claims

1. A method (10) for verifying a system (20) embedded in an at least partially autonomous robot or vehicle, characterized in that the following features: - input parameters (Z) of the system (20) are partitioned (11) into a first group (X) and a second group (Y), - a first selection (21) is made (12) in the case of input parameters of the first group (X) in accordance with a first method, wherein the first method comprises a search-based test, wherein the first selection (21) is made using an optimization technique that minimizes the performance of the system (20), - a second selection (22) is made (13) in the case of input parameters of the second group (Y) in accordance with a second method based on a determined probability distribution, - from the second selection (22), a characteristic variable (23) is calculated (14), and - the first selection (21) is adjusted (15) in accordance with the characteristic variable (23).

2. The method (10) according to claim 1, characterized in that at least one of the following features: - the input parameters of the first group (X) are determined by boundary conditions (24), and - the input parameters of the second group (Y) are determined by constraints (25).

3. The method (10) according to claim 1, characterized in that the following features: - a simulation (26) of the system (20) is performed in accordance with the first selection (21) and the second selection (22).

4. The method (10) according to claim 3, characterized in that the following features: - the simulation (26) includes an environment (27) of the system (20).

5. The method (10) according to claim 3 or 4, characterized in that the following features: - the simulation (26) provides a performance evaluation (28) of the system (20).

6. The method (10) according to any one of claims 1 to 4, characterized in that the following features: - the partitioning (11) of the input parameters is made manually.

7. The method (10) according to any one of claims 1 to 4, characterized in that the following features: - the first group (X) is smaller than the second group (Y).

8. The method (10) according to any one of claims 1 to 4, characterized in that the following features: - the method contains a verification method, such as a test or a worst-case oriented method, along with a filtering of statistically irrelevant errors, such as adversarial samples, which occur in the context of machine learning and computer vision.

9. The method (10) according to any one of claims 1 to 4, characterized in that, According to the verification, system errors identified in the verification are automatically improved.

10. A computer program product having a computer program which is set up so as to carry out the method (10) according to any one of claims 1 to 9 when run on a processor.

11. A machine-readable storage medium on which a computer program is stored which is set up so as to carry out the method (10) according to any one of claims 1 to 9 when run on a processor.

12. An apparatus comprising a memory and a processor, wherein a computer program is stored in the memory, the computer program being arranged to implement the method (10) according to any one of claims 1 to 9 when run on the processor.

Images