Method and apparatus for inspecting a technical system

Abstract

Method (10) and device for checking a technical system, characterized by the following features: performing a test (12) by means of a simulation (11) of the system, evaluating the test (12) in terms of a satisfaction measure (13) of quantitative requirements of the system and an error measure (14) of the simulation (11), classifying (15) the test (12) as reliable (16) or unreliable (17) depending on the satisfaction measure (13) and the error measure (14), and selecting (33) or improving (34) a simulation model for the simulation (11) based on the classification (15).

Description

Technical Field

[0001] This invention relates to a method for inspecting a technical system. Furthermore, the invention also relates to a corresponding device, a corresponding computer program, and a corresponding storage medium. Background Technology

[0002] In software technology, the overarching concept of "model-based testing" (MBT) summarizes the use of models to automate testing activities and generate test artifacts during the testing process. For example, it is well known that test cases are generated from models that describe the target behavior of the system under test.

[0003] In particular, embedded systems rely on deterministic input signals from sensors and stimulate their environment by outputting signals to vastly different actuators. Therefore, during the verification and early development phases of such systems, the system's model in the loop (MiL), software in the loop (SiL), processor in the loop (PiL), or hardware in the loop (HiL) is simulated along with the environment model within a controlled loop. In automotive technology, simulators used to inspect electronic control equipment according to testing phases and test objects, corresponding to this principle, are sometimes called component test benches, module test benches, or integration test benches.

[0004] DE10303489A1 discloses a method for testing software of a control unit in a vehicle, power tool, or robot system, wherein an adjustment segment controllable by the control unit is at least partially simulated by a test system in such a way that an output signal is generated from the control unit and the output signal of the control unit is transmitted to a first hardware module via a first connection, and a signal from a second hardware module is transmitted to the control unit as an input signal via a second connection, wherein the output signal is provided as a first control value in the software and is additionally transmitted to the test system in real time relative to the adjustment segment via a communication interface.

[0005] Such simulations are widely used in various technical fields, and are used, for example, to examine the suitability of embedded systems in power tools, motor control devices for drive, steering, and braking systems, camera systems, systems with artificial intelligence and machine learning components, robotic systems, or autonomous vehicles in their early development stages. However, due to a lack of confidence in the reliability of simulation model results based on existing technologies, these results are only incorporated to a limited extent into release decisions. Summary of the Invention

[0006] This invention provides a method, corresponding equipment, corresponding computer program, and corresponding storage medium for inspecting a technical system. According to one method of inspecting a technical system, tests are performed by means of a simulation of the technical system. The tests are evaluated in terms of a metric for meeting quantitative requirements of the technical system and a metric for error in the simulation. The result of evaluating the quality of the simulation model for a specific input using an uncertainty quantification method is called the error metric. The tests are classified as reliable or unreliable based on the metric and the error metric. A simulation model for the simulation is selected or improved based on the classification. For specific parameters of the test, the metric and the error metric are respectively represented in a feature space expanded through the specific parameters. After the evaluation, the classification is visualized in the feature space. Preferably, the technical system is at least partially autonomous robots or vehicles. The classification is performed by a classifier based on feature vectors, and the metric and the error metric form components of the feature vectors. The classifier maps the feature vectors to one of a plurality of categories, and the classification is performed within pre-given decision boundaries between the categories. In the preparation phase, the simulation is validated through experimental measurements on the technical system. The decision boundary is plotted such that the satisfaction metric obtained, on the one hand, in the simulation and on the other hand, deviates as little as possible. Other tests to be performed in the preparation phase can be automatically selected. The evaluation is performed such that the satisfaction metric is positive if the technical system meets the quantitative requirements, or negative if the technical system does not meet the quantitative requirements. Errors in the technical system identified through the checks are automatically improved. The invention also provides a computer program product having a computer program configured to perform various methods of the invention. Accordingly, the invention also relates to a machine-readable storage medium having a computer program stored thereon configured to perform various methods of the invention. The apparatus of the invention is configured to perform various methods of the invention.

[0007] The solution according to the invention is based on the understanding that the quality of the simulation model plays a decisive role in the correctness and predictability of test results that can be achieved using the simulation model. In the field of MBT (Multi-Level Testing), the sub-discipline of validation has the task of comparing real measurements with simulation results. For this purpose, various metrics, numerical values, or other comparators are used, which should logically correlate signals with each other and are collectively referred to below as signal metrics (SMs). Examples of such signal metrics are metrics that compare magnitude, phase shift, and correlation. Some signal metrics are defined by relevant standards, such as according to ISO 18571.

[0008] Generally, uncertainty quantization techniques support the estimation of simulation and model quality. The result of evaluating model quality using signal metrics, or more generally using uncertainty quantization methods, for a specific input X (which can be parameters or a scenario), is referred to below as the simulation model error metric (SMerrorX). To generalize (interpolate and extrapolate) SMerrorX for previously unexamined inputs, parameters, or scenarios X, machine learning models can be used, such as machine learning models based on so-called Gaussian processes.

[0009] During verification, the system under test (SUT) is typically inspected based on requirements, specifications, or performance metrics. It should be noted that Boolean requirements or specifications can often be converted into quantitative measurements using formalism such as signal temporal logic (STL). This formalism serves as the basis for quantitative semantics, where positive values ​​indicate satisfaction of the requirement and negative values ​​indicate violation; this quantitative semantics represents the generalization of verification. In the following text, such requirements, specifications, or performance metrics are collectively referred to as “quantitative requirements” (QSpec).

[0010] This quantitative requirement can be checked based on a real System Under Test (SUT) or a model of a real SUT (i.e., a "virtual SUT"). To perform this verification, a catalog is compiled with the test conditions that the SUT must meet to determine whether the SUT possesses the expected performance and security characteristics. These test conditions can be parameterized to cover any number of individual tests.

[0011] In this context, the proposed solution considers the need for reliable test results to guarantee the performance and safety characteristics of the SUT. It is precisely by performing tests based on simulations of the system or sub-components (rather than the real system) that the reliability of the simulation results is ensured.

[0012] Typically, to achieve this goal, simulation engineers strive to improve their simulation models based on validation metrics that correlate simulation with reference metrics from practice. However, it is often unclear which validation metrics to use and what validation results to achieve. Therefore, in addition to these results, the method according to the invention also uses the QSpec requirements for the SUT in test conditions. This provides a standard for the quality of the simulation model in its intended use. Such a standard can be used in two ways: on the one hand, to improve the simulation model, and on the other hand, to select different simulation models or test platforms, if alternatives exist.

[0013] The advantage of the solution to this task according to the present invention lies in its ingenious combination of two approaches, contrary to concepts based solely on validation or verification. For this purpose, a "virtual test classifier" is introduced, which combines the requirements of model validation and product testing. This is achieved by logically associating information from simulation and model quality (SMerrorX) validation on the one hand with information from test requirements (QSpec) on the other.

[0014] The corresponding tests can be applied in very different fields. For example, the functional safety of automated systems, such as those used for automated driving, should be considered.

[0015] The basic ideas of this invention can be advantageously extended and improved through the measures listed in this disclosure. This allows for the establishment of automated, computer-implemented testing environments to significantly and automatically improve the quality of the hardware or software products under test. Attached Figure Description

[0016] Embodiments of the invention are illustrated in the accompanying drawings, and the embodiments are explained in more detail in the following description.

[0017] Figure 1 A virtual test classifier is shown.

[0018] Figure 2 A first scheme for generating decision boundaries for classifiers based on data is shown.

[0019] Figure 3 A second scheme for generating the decision boundary of a classifier based on the formula solution is shown.

[0020] Figure 4 The use of a classifier to improve the simulation model is shown.

[0021] Figure 5 This demonstrates the use of a classifier to select different simulation models or test platforms.

[0022] Figure 6 The classification results are visualized in the feature space expanded by the test parameters.

[0023] Figure 7 The workstation is shown schematically. Detailed Implementation

[0024] According to the present invention, the simulation model error SMerrorX is evaluated within the range of test X, and the quantitative specification QSpec is estimated based on the simulation of the SUT, wherein test X can be obtained from a test catalog as a test case or as an instance of a parametric test. A virtual test classifier uses SMerrorX and QSpec as inputs and makes a binary determination of whether the test results based on the simulation are reliable.

[0025] Based on the language commonly used in informatics, and especially in pattern recognition, a classifier here should be understood as any algorithm or mathematical function that maps a feature space to a set of categories that are formed and bounded together during the classification process. In order to determine which category an object should be classified into (often also referred to as "classification"), a classifier uses so-called class boundaries or decision boundaries. The term "classifier" is used in technical language if the distinction between method and instance is not important, and is also partially synonymous with "classification" or "categorization" below.

[0026] Figure 1 This classification in the current application example is illustrated. Here, each point corresponds to a test performed through simulation and for which the required QSpec satisfaction metric 13 and error metric 14 SMerrorX are calculated. In this case, QSpec is defined such that it takes a positive value (reference numeral 24) when the test indicates that the system meets the corresponding requirement, and a negative value (reference numeral 25) when the system does not meet the requirement.

[0027] As shown in the figure, the decision boundary 19 of classifier 18 subdivides the space into four categories A, B, C, and D. The system will pass the test for category A with high reliability. For the tests of categories B and C, the simulation only provides unreliable results; therefore, such tests must be performed on a real system. The test for category D will fail on the system with high reliability.

[0028] The virtual test classifier 18 is based on the following consideration: the closer the test results are to the boundary between passing and failing tests, the more confidence can be placed in the simulation model. In other words, only when the model error 14 is assumed to be at most a marginal error can the requirements barely met in the simulation be used to replace the testing of the real system. On the other hand, when the absolute value of the quantitative requirement QSpec's satisfaction metric 13 is high (i.e., it has far exceeded the predetermined value or clearly missed the predetermined value), a certain deviation between the simulation results and the corresponding experimental measurements is acceptable.

[0029] Since this evaluation method presupposes an understanding of the model error SMerrorX of the simulation model, it is assumed that the simulation model has been validated and verified before using the virtual test classifier 18. Within the scope of verification, for example, a generalized model should be formed through machine learning based on Gaussian processes or other methods, which provides SMerrorX for a given X. It should be noted that the reliability of the simulation largely depends on the correctness of this generalized model.

[0030] Figure 2 The decision boundary 19 for generating classifier 18 based on data is shown. Figure 1 Possible solutions are proposed. Simulation 11 is validated in preparation phase 20 through experimental measurements 21 on the technical system. A decision boundary 19 is drawn such that the satisfaction metric 13 obtained, on the one hand, in simulation 11 and on the other hand, in experimental measurements 21, deviates as little as possible. As indicated by reference numeral 22, other tests 12 to be performed in preparation phase 20 are automatically selected. In the simplest case, boundary 19 is distributed along a straight line passing through the origin. The slope of this line is preferably chosen such that all the following points lie in regions C and B, and these regions are also as small as possible, at which points the quantitative requirement for the satisfaction metric 13 of QSpec to differ in sign between simulation 11 and actual measurements 21, i.e., all tests 12 that the simulation model fails.

[0031] Furthermore, considering a more general decision boundary 19, such as a polynomial, the function curve of this decision boundary is adapted using linear programming so that it satisfies the criteria of classifier 18 VTC. In this case, all the following points also lie in regions C and B, where the quantitative requirement for the satisfaction metric 13 of "QSpec" is different in sign between simulation 11 and actual measurement 21, i.e., equivalent to all tests 12 where the simulation model fails.

[0032] Figure 3An alternative scheme is shown for defining classifier 18 by solving a system of equations in the form of formula 23, satisfying the defining equations for metric 13 and error metric 14 based on this system of equations. The resulting function can be described deterministically or stochastically, assigning truth values ​​to the feature vectors 13, 14 formed by these two metrics.

[0033] For the following execution, assume I is the input set, O is the output set (which may also include input), and It is a system model and a real system as a function, and the function can only be observed through simulation 11 or experimental measurement 21 for a limited number of inputs. Furthermore, it is assumed that... This is the simulation model error SMerrorX, which is the distance or error metric between two corresponding outputs. Finally, it is assumed... It is the set of all the following inputs, for which the error metric 14 takes the value ϵ.

[0034] Starting from these definitions, for each input i∈I ϵ The deviation from the requirement to meet metric 13 can be upwardly constrained by the following terms, which depend neither on m1 nor on m2: Formula 1 .

[0035] Therefore, classifier 18 yields Formula 2 .

[0036] Here, in In the case where m1 and m2 are consistent with p, the simulation model is classified as reliable. It should be noted that classifier 18 requires the reciprocal of q.

[0037] The main advantage of this representation is that the virtual test classifier 18 can be formulated independently of m1 and m2, because the virtual test classifier depends only on the satisfaction metric 13 and error metric 14 of the quantitative requirements. Therefore, starting from a single error metric 14 and a complex number of n quantitative requirements, n virtual test classifiers 18 can be computed, one for each requirement. Thus, the model only needs to be validated once for the error metric 14, instead of, for example, for each individual requirement.

[0038] For a complex number of m error measures and a complex number of n quantitative requirements, this examination can be generalized in a simple way, where m is typically very small and n is large. In this case, n·m virtual test classifiers 18 can be computed. If one of these classifiers 18 provides a value W, the simulation result can be considered reliable. This allows for more accurate classification because some error measures 14 may be better suited to a particular requirement than others.

[0039] Alternatively, a virtual test classifier 18 can be defined within a random range, in which the input is assumed to be randomly distributed according to an arbitrary probability density function. This indicates the value taken in error metric 14. Under the assumption that the conditional cumulative distribution function of the bias of metric 13 is satisfied, and with the threshold of the probability of classifier 18 making a correct judgment being τ∈(0.1)—the value τ is therefore typically close to 1—the virtual test classifier 18 can be defined as follows: Formula 3 .

[0040] Figure 4 The method 10 for improving simulation models according to the present invention is explained from an application perspective under the following assumptions: • A model suitable for simulation 11 and a set of tests 12 are given, along with defined input parameters. • It can perform real tests for experimental measurements.21 • The requirement QSpec is quantifiable and pre-given, and implemented within the scope of a monitoring system that evaluates test 12 against a satisfaction metric 13 for these requirements. In this figure, both satisfaction metrics 13 relate to the same requirement QSpec, but one is estimated based on simulation 11 and the other is estimated through experimental measurement 21 on the system. • SMerrorX is a predefined error metric 14. Therefore, simulation 11 and measurement 21 have been performed for some test inputs, and error metric 14 generalizes the corresponding test 12 into a new, previously unperformed experiment with a certain reliability, determined for example by an upper and lower bound of error metric 14. For classifier 18 ( Figures 1 to 3 Only the most unfavorable, i.e., the highest, error metric 14 is used. It should be noted that classifier 18 can be used to further improve error metric 14.

[0041] Under these assumptions, method 10 can be designed as follows: 1. Perform test 12 using simulation 11, where an output signal is generated. 2. The output signal is evaluated in terms of the satisfaction metric 13 of the required QSpec and the error metric 14 of the simulation 11, based on the SMerrorX error model. 3. The satisfaction metric 13 and error metric 14 are fed to the classifier 18. 4. For each test 12, classifier 18 is assigned to the following categories A, B, C, D ( Figure 1 One of the following categories 15: Test 12 succeeded in simulation 11 and its result is reliable 16; the test failed in simulation 11 and its result is reliable 16; or the result of simulation 11 is unreliable 17. 5. Selectively add the test results of reliable 16 (for which simulation 11 is now considered reliable) to the corresponding database 31. 6. The unreliable test 12 can be used as an opportunity to suggest to the user 32 to perform the corresponding measurement 21 on the system or to automatically initiate such measurement 21. 7. Optionally, additional tests can be created in the context of the current test input X, such as through search-based testing (SBT), the goal of which is to maximize the error metric SMerrorX. 8. Each of these tests was performed, and new measurements were provided through the real system under study.21 9. As indicated by reference numeral 34 in the attached figure, improve the simulation model based on the measurement 21. This can be achieved, for example, by using parameter identification techniques or machine learning techniques, or through manual intervention by a simulation model engineer, such as improvements to the physical modeling or the implementation of the simulation model.

[0042] A variation of this method for selecting different simulation models or test platforms is based on the following assumptions: • Several models or test platforms suitable for simulation 11 and a set of tests 12 are given, along with defined input parameters. • The requirement QSpec is quantifiable and pre-given, and implemented within the scope of a monitoring system that evaluates test 12 against a satisfaction metric 13 for these requirements. In this figure, both satisfaction metrics 13 relate to the same requirement QSpec, but one is estimated based on simulation 11 and the other is estimated through experimental measurement 21 on the system. • SMerrorX is a predefined error metric 14. Therefore, simulation 11 and measurement 21 have been performed for some test inputs, and error metric 14 generalizes the corresponding test 12 into a new, previously unperformed experiment with a certain reliability, determined for example by an upper and lower bound of error metric 14. For classifier 18 ( Figures 1 to 3 Only the most unfavorable, i.e., the highest, error metric 14 is used. It should be noted that classifier 18 can be used to further improve error metric 14.

[0043] Under these assumptions, the method can be designed as follows: 1. For each simulation model or each test platform, define classifier 18 according to the above explanation. 2. Perform test 12 for each test platform 35, generating output signals. 3. The output signal is evaluated based on the SMerrorX error model in terms of the requirement of QSpec satisfaction metric 13 and the error metric 14 of simulation 11. 4. The satisfaction metric 13 and error metric 14 are fed to the classifier 18. 5. For each test 12, classifier 18 is assigned to the following categories A, B, C, D ( Figure 1 One of the following categories 15: Test 12 succeeded in simulation 11 and its result is reliable 16; the test failed in simulation 11 and its result is reliable 16; or the result of simulation 11 is unreliable 17. 6. Select 33 test platforms or simulation models that best meet the expected criteria, such as the maximum number of (positive or negative) test results classified as reliability 16, the maximum number of positive test results classified as reliability 16, or the maximum number of negative test results classified as reliability 16.

[0044] Figure 6 A possible visualization of the classification results in the test parameter feature space (hereinafter referred to as the "parameter space") expanded by the test parameters is drawn. For specific parameters 26 and 27 of test 12—exemplarily, in this diagram, the distance 26 and mass 27 of vehicles cutting into their own lane—the satisfaction metric 13 and error metric 14 are represented as points in the parameter space, respectively. Then, in the virtual test environment 29, the classification 15 of test 12 is visualized in the parameter space by classifier 18.

[0045] like Figure 7 As shown in the schematic diagram, method 10 can be implemented, for example, in software or hardware or a combination of software and hardware, for example in workstation 30.

Claims

1. A method for inspecting a technical system (10). It is characterized by the following features: The test (12) is performed by means of the simulation (11) of the technical system. The test (12) is evaluated in terms of a metric (13) for the quantitative requirements of the technical system and an error metric (14) for the simulation (11), wherein the result of evaluating the quality of the simulation model for a specific input using an uncertainty quantification method is referred to as the error metric (14). The test (12) is classified (15) as reliable (16) or unreliable (17) based on the satisfaction metric (13) and the error metric (14). Based on the classification (15), a simulation model is selected or improved (34) for the simulation (11). For the specific parameters of the test (12), the satisfaction metric (13) and the error metric (14) are respectively represented in the feature space expanded through the specific parameters, and Following the evaluation, the classification (15) is visualized (28) in the feature space.

2. The method (10) according to claim 1. It is characterized by the following features: The technological system is at least partially autonomous robots or vehicles.

3. The method (10) according to claim 1. Its features The following characteristics: The classification (15) is performed based on the feature vector by the classifier (18), and The metric (13) and the error metric (14) form the components of the feature vector.

4. The method (10) according to claim 3. It is characterized by the following features: The classifier (18) maps the feature vector to one of multiple categories, and The classification (15) is performed within a pre-given decision boundary (19) between the categories.

5. The method (10) according to claim 4. Its features The following characteristics: The simulation (11) is confirmed by experimental measurements (21) on the technical system during the preparation phase (20). The decision boundary (19) is drawn such that the satisfaction metric (13) obtained in the simulation (11) and the experimental measurement (21) deviates as little as possible.

6. The method (10) according to claim 5. It is characterized by the following features: Automatically select (22) other tests (12) to be performed in the preparation phase (20).

7. The method (10) according to any one of claims 1 to 6. It is characterized by the following features: The evaluation is conducted in such a manner that the satisfaction measure (13) is positive if the technical system meets the quantitative requirements of (24), or negative if the technical system does not meet the quantitative requirements of (25).

8. The method (10) according to any one of claims 1 to 6, characterized in that, The errors of the technical system identified through the inspection are automatically improved.

9. A computer program product having a computer program configured to perform the method (10) according to any one of claims 1 to 8.

10. A machine-readable storage medium having a computer program stored thereon, the computer program being configured to perform the method (10) according to any one of claims 1 to 8.

11. An apparatus configured to perform the method (10) according to any one of claims 1 to 8.

Images