Method and apparatus for inspecting a technical system

Abstract

Method (10) and device for checking a technical system, characterized by the following features: a test (12) is performed by means of a simulation (11) of the system, the test (12) is evaluated in terms of a measure of satisfaction (13) of quantitative requirements of the system and a measure of error (14) of the simulation (11), the test (12) is classified (15) as reliable (16) or unreliable (17) depending on the measure of satisfaction (13) and the measure of error (14).

Description

Technical Field

[0001] This invention relates to a method for inspecting a technical system. Furthermore, the invention also relates to a corresponding device, a corresponding computer program, and a corresponding storage medium. Background Technology

[0002] In software technology, the overarching concept of "model-based testing" (MBT) summarizes the use of models to automate testing activities and generate test artifacts during the testing process. For example, it is well known that test cases are generated from models that describe the target behavior of the system under test.

[0003] In particular, embedded systems rely on deterministic input signals from sensors and stimulate their environment by outputting signals to vastly different actuators. Therefore, during the verification and early development phases of such systems, the system's model in the loop (MiL), software in the loop (SiL), processor in the loop (PiL), or hardware in the loop (HiL) is simulated along with the environment model within a control loop. In automotive technology, simulators used to inspect electronic control devices according to testing phases and test objects, corresponding to this principle, are sometimes called component check benches, module check benches, or integration check benches.

[0004] DE10303489A1 discloses a method for testing software of a control unit in a vehicle, power tool, or robotic system, wherein a controlled segment controllable by the control unit is at least partially simulated by a test system in such a way that an output signal is generated from the control unit and the output signal of the control unit is transmitted to a first hardware module via a first connection, and a signal from a second hardware module is transmitted to the control unit as an input signal via a second connection, wherein the output signal is provided as a first control value in the software and is additionally transmitted to the test system in real time relative to the controlled segment via a communication interface.

[0005] Such simulations are widely used in various technical fields, and are used, for example, to examine the suitability of embedded systems in power tools, motor control devices for drive, steering, and braking systems, camera systems, systems with artificial intelligence and machine learning components, robotic systems, or autonomous vehicles in their early development stages. However, due to a lack of confidence in the reliability of simulation model results based on existing technologies, these results are only included to a limited extent in release decisions. Summary of the Invention

[0006] This invention provides a method, corresponding apparatus, corresponding computer program, and corresponding storage medium for examining a technical system according to the independent claims.

[0007] The method according to the invention is based on the understanding that the quality of the simulation model plays a decisive role in the correctness and predictability of test results that can be achieved using the simulation model. In the field of MBT (Multi-Level Testing), the sub-discipline of validation has the task of comparing real measurements with simulation results. For this purpose, various metrics, numerical values, or other comparators are used, which should logically correlate signals with each other and are collectively referred to below as signal metrics (SMs). Examples of such signal metrics are metrics that compare magnitude, phase shift, and correlation. Some signal metrics are defined by relevant standards, such as according to ISO 18571.

[0008] Generally, uncertainty quantization techniques support the estimation of simulation and model quality. The result of evaluating model quality using signal metrics, or more generally using uncertainty quantization methods, for a specific input X (which can be parameters or a scenario), is referred to below as the simulation model error metric (SMerrorX). To generalize (interpolate and extrapolate) SMerrorX for previously unexamined inputs, parameters, or scenarios X, machine learning models can be used, such as machine learning models based on so-called Gaussian processes.

[0009] During verification, the system under test (SUT) is typically inspected based on requirements, specifications, or performance metrics. It should be noted that Boolean requirements or specifications can often be converted into quantitative measurements using formalism such as signal temporal logic (STL). This formalism serves as the basis for quantitative semantics, where positive values ​​indicate satisfaction of the requirement and negative values ​​indicate violation; this quantitative semantics represents the generalization of verification. In the following text, such requirements, specifications, or performance metrics are collectively referred to as “quantitative requirements” (QSpec).

[0010] This quantitative requirement can be checked based on a real SUT or a model of a real SUT (i.e., a "virtual SUT"). To perform this verification, a catalog is compiled with the test conditions that the SUT must meet to determine whether the SUT possesses the desired performance and safety characteristics. These test conditions can be parameterized to cover any number of individual tests.

[0011] In this context, the proposed solution considers the need for reliable test results to guarantee the performance and safety characteristics of the SUT. It is precisely by performing tests based on simulations of the system or sub-components (rather than the real system) that the reliability of the simulation results is ensured.

[0012] Therefore, the goal of this approach is to obtain reliable test results based on simulation, so that these results can be used to replace real-world test scenarios. Consequently, testing costs should be reduced by minimizing the number of actual trials.

[0013] This document presents a series of tests, such as a test catalog or parameterized tests that the System Under Test (SUT) should meet. The solution subdivides the test set into two groups: tests that must be executed on a real system, and tests that can also be performed based on simulation.

[0014] The proposed scheme enables the system to advise end users on whether virtual tests are trustworthy; to advise users when tests should be performed on a real system; and to automatically initiate the execution of real tests if virtual tests should not be classified as reliable.

[0015] The advantage of the solution to this task according to the present invention lies in its ingenious combination of two approaches, contrary to concepts based solely on validation or verification. For this purpose, a "virtual test classifier" is introduced, which combines the requirements of model validation and product testing. This is achieved by logically associating information from simulation and model quality (SMerrorX) validation on the one hand with information from test requirements (QSpec) on the other.

[0016] The application of corresponding tests can be considered in very different fields. For example, the functional safety of automated systems, such as those used for automated driving, should be considered.

[0017] The measures listed in the dependent claims can be used to advantageously extend and improve the basic ideas described in the independent claims. This allows for the establishment of automated, computer-implemented testing environments to significantly automate the improvement of the quality of the hardware or software products under test. Attached Figure Description

[0018] Embodiments of the invention are illustrated in the accompanying drawings, and these embodiments are explained in detail in the following description.

[0019] Figure 1 A virtual test classifier is shown.

[0020] Figure 2 A first scheme for generating the decision boundary of a classifier based on data is shown.

[0021] Figure 3 A second scheme for generating the decision boundary of the classifier based on the formula solution is shown.

[0022] Figure 4 A description of the method according to the invention from an application perspective is shown.

[0023] Figure 5 The visualization of the classification results in the feature space expanded by the test parameters is shown.

[0024] Figure 6 The control device according to the second embodiment is shown schematically. Detailed Implementation

[0025] According to the present invention, the simulation model error SMerrorX is evaluated within the range of test X, and the quantitative specification QSpec is estimated based on the simulation of the SUT, wherein test X can be obtained from a test catalog as a test case or as an instance of a parametric test. A virtual test classifier uses SMerrorX and QSpec as inputs and makes a binary determination of whether the test results based on the simulation are reliable.

[0026] Based on the language commonly used in informatics, and especially in pattern recognition, a classifier here should be understood as any algorithm or mathematical function that maps a feature space to a set of categories that are formed and bounded together during the classification process. In order to determine which category an object should be classified into (often also referred to as "classification"), a classifier uses so-called class boundaries or decision boundaries. The term "classifier" is used in technical language if the distinction between method and instance is not important, and is also partially synonymous with "classification" or "categorization" below.

[0027] Figure 1This classification in the application example is illustrated. Here, each point corresponds to a test performed through simulation and for which the required QSpec satisfaction metric 13 and error metric 14 SMerrorX are calculated. In this case, QSpec is defined such that it takes a positive value (reference numeral 24) when the test indicates that the system meets the corresponding requirement, and a negative value (reference numeral 25) when the system does not meet the requirement.

[0028] As shown in the figure, the decision boundary 19 of classifier 18 subdivides the space into four categories A, B, C, and D. The system will pass the test for category A with high reliability. For the tests of categories B and C, the simulation only provides unreliable results; therefore, such tests must be performed on a real system. The test for category D will fail on the system with high reliability.

[0029] The virtual test classifier 18 is based on the following considerations: only when it should be assumed that the model error 14 is at most marginal, the requirement that is barely met in the simulation can replace the testing of the real system. On the other hand, if the absolute value of the quantitative requirement QSpec's satisfaction metric 13 is high (i.e., it has far exceeded the predetermined value or clearly missed the predetermined value), a certain deviation between the simulation results and the corresponding experimental measurements is acceptable.

[0030] Since this evaluation method presupposes an understanding of the model error SMerrorX of the simulation model, it is assumed that the simulation model has been validated and verified before using the virtual test classifier 18. Within the scope of verification, for example, a generalized model should be formed through machine learning based on Gaussian processes or other methods, which provides SMerrorX for a given X. It should be noted that the reliability of the simulation largely depends on the correctness of this generalized model.

[0031] Figure 2 The decision boundary 19 for plotting classifier 18 based on data is shown. Figure 1 Possible solutions are proposed. Simulation 11 is validated in preparation phase 20 through actual measurements 21 on the system. Decision boundaries 19 are drawn such that the satisfaction metric 13 obtained in simulation 11 and measurement 21 deviates as little as possible. Preferably, additional tests 12 to be performed in preparation phase 20 are automatically selected. In the simplest case, boundary 19 is distributed along a straight line passing through the origin. The slope of this line is preferably chosen such that all the following points lie in regions C and B and these regions are also as small as possible, at which points the quantitative requirement for the satisfaction metric 13 of QSpec differs in sign between simulation 11 and actual measurement 21, i.e., all tests 12 that the simulation model fails.

[0032] Furthermore, considering a more general decision boundary 19, such as a polynomial, the function curve of this decision boundary is adapted using linear programming so that it satisfies the criteria of classifier 18 VTC. In this case, all the following points also lie in regions C and B, where the quantitative requirement for the satisfaction metric 13 of QSpec is that the sign differs between simulation 11 and real measurement 21, i.e., all tests 12 that the simulation model fails.

[0033] Figure 3 An alternative is shown to define the classifier (18) by solving a system of equations in the form of formula 23, satisfying the defining equations of metric 13 and error metric 14 based on this system of equations. The resulting function can be described deterministically or stochastically, which assigns the truth value to the feature vectors 13, 14 formed by these two metrics.

[0034] For the following execution, assume I is the input set, O is the output set (which may also include input), and It is a system model and a real system as a function, and the function can only be observed through simulation 11 or experimental measurement 21 for a limited number of inputs. Furthermore, it is assumed that... This is the simulation model error SMerrorX, which is the distance or error metric between two corresponding outputs. Finally, it is assumed... It is the set of all the following inputs, for which the error metric 14 takes a value. .

[0035] Starting from these definitions, for each input The deviation from the requirement to meet metric 13 can be upwardly constrained by the following terms, which depend neither on m1 nor on m2: Formula 1 .

[0036] Therefore, classifier 18 yields Formula 2 .

[0037] Here, in In the case where m1 and m2 are consistent with p, the simulation model is classified as reliable. It should be noted that classifier 18 requires the reciprocal of q.

[0038] The main advantage of this representation is that the virtual test classifier 18 can be formulated independently of m1 and m2, because the virtual test classifier depends only on the satisfaction metric 13 and error metric 14 of the quantitative requirements. Therefore, starting from a single error metric 14 and a complex number of n quantitative requirements, n virtual test classifiers 18 can be computed, one for each requirement. Thus, the model only needs to be validated once for the error metric 14, instead of, for example, for each individual requirement.

[0039] For a complex number of m error measures and a complex number of n quantitative requirements, this examination can be generalized in a simple way, where m is typically very small and n is large. In this case, n·m virtual test classifiers 18 can be computed. If one of these classifiers 18 provides a value W, the simulation results can be considered reliable. This allows for more accurate classification, as some error measures 14 may be better suited to a particular requirement than others.

[0040] Alternatively, a virtual test classifier 18 can be defined within a random range, in which the input is assumed to be randomly distributed according to an arbitrary probability density function. This indicates the value taken in error metric 14. Under the assumption that the conditional cumulative distribution function of the bias of metric 13 is satisfied, and with the threshold of the probability of classifier 18 making a correct judgment being τ∈(0.1)—the value τ is therefore typically close to 1—the virtual test classifier 18 can be defined as follows: Formula 3 .

[0041] Figure 4 The method 10 according to the present invention is illustrated from an application perspective under the following assumptions: The set of simulation model 11 and test 12, as well as the defined inputs, are given in advance.

[0042] The requirement QSpec is quantifiable and pre-given, and implemented within the scope of a monitoring system that evaluates test 12 against a satisfaction metric 13 for these requirements. In this figure, both satisfaction metrics 13 relate to the same requirement QSpec, but one is estimated based on simulation 11 and the other is estimated through experimental measurement 21 on the system.

[0043] SMerrorX is a predefined error metric 14. Therefore, simulation 11 and measurement 21 have been performed for some test inputs, and error metric 14 generalizes the corresponding test 12 to a new, previously unperformed experiment with a certain reliability, determined for example by an upper and lower bound of error metric 14. For classifier 18 ( Figures 1 to 3 Only the most unfavorable, i.e., the highest, error metric 14 is used. It should be noted that classifier 18 can be used to further improve error metric 14.

[0044] Under these assumptions, method 10 can be designed as follows: 1. Define classifier 18 based on the above explanation.

[0045] 2. Test 12 is performed with the aid of simulation 11 and experimental measurement 21, in which an output signal is generated.

[0046] 3. The output signal is evaluated based on the SMerrorX error model in terms of both the requirement QSpec satisfaction metric 13 and the error metric 14 of simulation 11.

[0047] 4. The satisfaction metric 13 and error metric 14, taken in simulation 11 and measurement 21 respectively, are fed to classifier 18.

[0048] 5. For each test 12, classifier 18 is assigned to the following categories A, B, C, D ( Figure 1 One of the following categories 15: Test 12 succeeded in simulation 11 and its result is reliable 16; the test failed in simulation 11 and its result is reliable 16; or the result of simulation 11 is unreliable 17.

[0049] 6. Add the test results of reliability 16 (for which simulation 11 is now considered reliable) to the corresponding database 31.

[0050] 7. The unreliable test 12 can be used as an opportunity to suggest to the user 32 to perform the corresponding measurement 21 on the system.

[0051] 8. Optionally, this measurement 21 can be started manually or automatically.

[0052] 9. If as per the above Figure 2 The decision boundary 19 of classifier 18, as explained by the data, is plotted. Figure 1 Alternatively, classifier 18 can be updated based on the results of measurement 21 and improved using these data points.

[0053] Figure 5A possible visualization of the classification results in the test parameter feature space (hereinafter referred to as the "parameter space") expanded by the test parameters is drawn. For specific parameters 26 and 27 of test 12—exemplarily, in this diagram, the distance 26 and mass 27 of vehicles cutting into their own lane—the satisfaction metric 13 and error metric 14 are represented as points in the parameter space, respectively. Then, in the virtual test environment 29, the classification 15 of test 12 is visualized in the parameter space by classifier 18.

[0054] like Figure 6 As shown in the schematic diagram, method 10 can be implemented, for example, in software or hardware or a combination of software and hardware, for example in workstation 30.

Claims

1. A method for inspecting a technical system, It is characterized by the following features: The tests are performed using a simulation model of the technical system, wherein the simulation model has been verified and validated. The test is evaluated in terms of a metric for meeting the requirements of the technical system and a metric for error in the simulation model, wherein the requirements are quantifiable, the metric for meeting the requirements measures whether the technical system meets or violates the requirements, and the metric for error in the simulation model measures the difference between the results of the simulation model and the results of experimental measurements. The tests are categorized as reliable or unreliable based on both the satisfaction metric and the error metric. If the test is classified as unreliable, a real test will be automatically initiated on the technical system. The classification is performed by a classifier based on feature vectors, and the components of the feature vectors are formed by satisfying a metric and an error metric. The classifier maps the feature vector to one of a plurality of categories, and the classification is performed within a pre-given decision boundary between the categories.

2. The method according to claim 1, It is characterized by the following features: The simulation was validated through experimental measurements on the system during the preparation phase. The decision boundary is drawn such that the satisfaction measure obtained, on the one hand, in the simulation and on the other hand, in the experimental measurement, deviates as little as possible.

3. The method according to claim 1, Its features The following characteristics: The classifier is defined by solving a system of equations, and The set of equations includes the defining equations that satisfy both the metric and the error metric.

4. The method according to any one of claims 1 to 3, It is characterized by the following features: The evaluation is conducted as follows: if the technical system meets the requirements, the satisfaction metric is positive; if the technical system does not meet the requirements, the satisfaction metric is negative.

5. The method according to any one of claims 1 to 3, Its features The following characteristics: For specific parameters of the test, the satisfaction metric and error metric are respectively represented in the feature space expanded through the specific parameters, and Following the evaluation, the classification is visualized in the feature space.

6. The method according to claim 1, characterized by the following features: The technological system is at least partially autonomous robots or vehicles.

7. A computer program product configured to perform the method according to any one of claims 1 to 6.

8. A machine-readable storage medium having a computer program product according to claim 7 stored thereon.

9. An apparatus configured to perform the method according to any one of claims 1 to 6.

Images