Method and apparatus for inspecting a technical system

Abstract

Method (10) for checking a technical system, characterized by the following features: - performing tests (12) by means of simulations (11) of the system; - analyzing the tests (12) in terms of a degree of satisfaction (13) of quantitative requirements on the system and of a degree of error (14) of the simulations (11); - classifying (15) the tests (12) as either reliable (16) or not reliable, depending on the case, in relation to each of the degree of satisfaction (13) and the degree of error (14); and - selecting (32) at the degree of error (14) in relation to the tests (12) classified as reliable (16).

Description

Technical Field

[0001] This invention relates to a method for inspecting a technical system. Furthermore, this invention relates to a corresponding apparatus, a corresponding computer program, and a corresponding storage medium. Background Technology

[0002] In software technology, the overarching concept of "model-based testing" (MBT) encompasses the use of models for automating testing activities and for generating test artifacts during the testing process. For example, generating test cases from models that describe the target behavior of the system to be tested is well-known.

[0003] Embedded systems, in particular, rely on the decisive input signals from sensors and, in turn, stimulate their environment through output signals to various actuators. Therefore, in the verification and upstream development phases of such systems, the system's model (Model-in-the-Loop, MiL), software (Software-in-the-Loop, SiL), processor (Processor-in-the-Loop, PiL), or all of its hardware (Hardware-in-the-Loop, HiL) are simulated along with a model of the surrounding environment within the control loop. In vehicle technology, simulators used to inspect electronic control equipment based on this principle are sometimes called component check benches, module check benches, or integration check benches, depending on the testing phase and the object being tested.

[0004] DE10303489A1 discloses a method for testing software of a control unit of a vehicle, power tool, or robotic system, wherein an adjustment path controllable by the control unit is at least partially simulated by a test system in such a way that the control unit generates an output signal and transmits the output signal of the control unit to a first hardware building block via a first connection and transmits a signal of a second hardware building block as an input signal to the control unit via a second connection, wherein the output signal is provided as a first control value in the software and is additionally transmitted to the test system in real time with respect to the adjustment path via a communication interface.

[0005] Such simulations have been extended to various technical fields and are used, for example, to conduct suitability checks on embedded systems in power tools, motor control devices for drive, steering, and braking systems, camera systems, systems with artificial intelligence and machine learning components, robotic systems, or autonomous vehicles in the early stages of their development. Nevertheless, the results of simulation models based on existing technologies are only included in publication decisions to a limited extent due to their lack of reliability. Summary of the Invention

[0006] The present invention provides a method for inspecting a technical system, a corresponding apparatus, a corresponding computer program, and a corresponding storage medium according to the independent claims.

[0007] The solution according to the invention is based on the understanding that the quality of the simulation model is crucial to the accurate predictability of the test results obtained therefrom. In the field of MBT, the classification method for verification involves the task of comparing actual measurements with simulation results. For this purpose, different metrics (Metriken), dimensional numbers (Maßzahlen), or other comparators are used, which correlate signals with each other and are hereinafter referred to inductively as signal metrics (Signalmetriken, SM). Examples of such signal metrics include metrics that compare magnitude, phase shift, and correlation. Some signal metrics are defined by relevant standards, such as according to ISO 18571.

[0008] Generally, uncertainty quantification techniques support the estimation of simulation quality and model quality. When signal metrics are introduced, or when uncertainty quantification techniques are used in general, the evaluation result for a model quality that may involve parameters or scenarios for a specific input X is referred to as the simulation model error metric – or simply: error metric – SMerrorX. To generalize (interpolate and extrapolate) SMerrorX for previously unconsidered inputs, parameters, or scenarios X, machine learning models can be used, for example, based on so-called Gaussian processes.

[0009] During verification, the test object (system under test, SUT) is typically investigated based on requirements, specifications, or capability characteristics. It's important to note that Boolean requirements or specifications can often be converted into quantitative measurements using forms such as signal temporal logic (STL). These forms serve as the basis for quantitative semantics, which, in this respect, is represented as a generalization of the verification, with positive values ​​indicating satisfaction of the requirement and negative values ​​indicating non-satisfaction. In the following text, this requirement, specification, or capability level is inductively referred to as the "quantitative requirement" (QSpec).

[0010] Such quantitative requirements can be verified either against a real SUT or against its model—like a "virtual SUT." For verification purposes, the catalog is combined with the test conditions that the SUT must meet in order to determine whether it possesses the desired capability and deterministic properties. These test conditions can be parameterized and thus cover any number of individual tests.

[0011] In this context, the proposed scheme considers the need for acceptable test results to guarantee the capability and deterministic characteristics of the SUT. When performing tests based on simulations of the system or subcomponents—rather than simulations of the real system—it is crucial to ensure the reliability of the simulation results.

[0012] For this purpose, validation techniques are used to evaluate the degree of fit between the simulation model and real measurements. The comparison between simulation results and real measurements is performed using the validation error metric SMerrorX, which generalizes through interpolation and extrapolation, allowing the error metric to be predicted for new inputs X without requiring corresponding measurements. However, the predictions of SMerrorX are related to uncertainty, which can be modeled as intervals or probability distributions.

[0013] The proposed method facilitates the selection of an appropriate simulation and model quality verification metric, SMerrorX. While different verification metrics are known in principle, such as mean squared error or the already mentioned standard ISO 18571, selecting among these metrics in practice remains challenging because the applicability of a particular metric largely depends on the requirements QSpec imposed on the SUT. To address this problem, the solution according to the invention therefore traces back to the analysis of tests already performed with respect to these requirements QSpec.

[0014] One advantage of this solution is that it cleverly combines the two approaches, unlike approaches based solely on verification or validation. To this end, a "virtual test classifier" is introduced, which combines the needs of model validation and product testing. This is achieved by correlating information from the verification of simulation quality and model quality (SMerrorX) on one hand, and information from the test requirements (QSpec) on the other.

[0015] The application of such tests can be considered in a variety of different fields. For example, the functional determinism of automated systems, such as those used for automated driving functions, can be considered.

[0016] The measures listed in the dependent claims enable advantageous improvements and enhancements to the basic concept described in the independent claims. Therefore, an automated, computer-implemented testing environment can be set up to autonomously improve the quality of the hardware or software product under test to a large extent. Attached Figure Description

[0017] Embodiments of the invention are shown in the accompanying drawings and explained in detail in the following description. Wherein: Figure 1 A virtual test classifier is shown. Figure 2 A first scheme for determining the decision boundary (Entscheidungsgrenze) for generating a classifier based on data is shown. Figure 3 A second scheme for generating the decision boundary of a classifier based on a formal solution is shown. Figure 4 The use of a classifier is shown to select the appropriate validation metric SMerrorX. Figure 5 The workstation is shown schematically. Detailed Implementation

[0018] According to the present invention, test X can be extracted from the test catalog as a test case or obtained as an instance (Instanz) of a parametric test. Within the scope of test X, the simulation model error SMerrorX is analyzed and the quantitative specification QSpec is evaluated based on the simulation of the SUT. The virtual test classifier uses SMerrorX and QSpec as input values ​​and makes a binary decision on whether the simulation-based test results are reliable.

[0019] Based on the linguistic usage commonly used in informatics, and especially in pattern recognition, a classifier in this context can be understood as any algorithm or mathematical function that maps a feature space onto a certain number of ranks formed and separated from each other during the classification process. In order to determine which rank or category an object should be classified into (colloquially, "classification"), a classifier uses so-called rank boundaries or decision boundaries. The term "classifier" is used in technical language, and hereinafter, partly synonymous with "ranking" or "classification," unless the distinction between method and instance is significant.

[0020] Figure 1 This gradation is illustrated in the current application example. In this case, each point corresponds to a test, which is performed by simulation and for which the degree of satisfaction of requirement QSpec (Erfüllungsmaß) (13) and the error metric (Fehlermaß) (14) SMerrorX are calculated. In this case, QSpec is defined such that a positive value (Figure 24) is used when the test can infer that the system meets the corresponding requirement, and a negative value (Figure 25) is used when the system does not meet the requirement.

[0021] As can be seen from the diagram, the decision boundary (19) of the classifier (18) divides the space into four levels: A, B, C, and D. A system with high reliability will pass the test at level A. For tests at levels B and C, the simulation only provides unreliable results; therefore, such tests should be performed on a real system. The test at level D will fail on a system with high reliability.

[0022] The virtual test classifier (18) is based on the consideration that only when the model error (14) can be assumed to be at most marginal can the requirements barely met in the simulation be substituted for the experiment on the real system. On the other hand, when the quantitative requirement "QSpec" is met to a high degree of numerical satisfaction (13), that is, when the preset is far exceeded or obviously wrong, a certain deviation between the simulation results and the corresponding experimental measurements can be tolerated.

[0023] Since this approach presupposes knowledge of the model error SMerrorX of the simulation model, it can be assumed that the simulation model has been verified and validated before using the virtual test classifier (18). Within the scope of this validation, a generalized model should be formed—for example, based on Gaussian processes or otherwise through machine learning—that provides SMerrorX for a given X. It is important to note that the reliability of the simulation depends decisively on the accuracy of this generalized model.

[0024] Figure 2 The decision boundary (19-) for generating a classifier (18) based on data is shown. Figure 1 One possible approach is as follows: In the preparation phase (20), the simulation (11) is validated by experimental measurements (21) on the system; the decision boundary (19) is set such that the satisfaction (13) adopted in the simulation (11) and the measurement (21) deviates as little as possible; and preferably, other tests (12) to be performed in the preparation phase (20) are autonomously selected (22). In the simplest case, the boundary (19) extends along a straight line from the origin. Preferably, the slope of the line is chosen such that all points where the satisfaction (13) of the quantitative requirement QSpec between the simulation (11) and the actual measurement (21) differs in sign—i.e., all tests (12) where the simulation model fails—are located in regions C and B and these regions are also as small as possible.

[0025] Another consideration could be a general, such as a polynomial, decision boundary (19), whose function curve is adapted by means of linear programming to satisfy the classifier (18) VTC criteria. In this case, all points where the satisfaction (13) of the quantitative requirement QSpec between simulation (11) and real measurement (21) differs in sign—that is, all tests (12) where the simulation model fails—are also located in regions C and B.

[0026] Figure 3 An alternative to defining the classifier (18) by solving the system of equations (23) is shown, based on which the equations defining the satisfaction (13) and error metric (14) are derived. The synthesis function, which assigns the true values ​​(Wahrheitswert) to the feature vector (13, 14) formed by these two metrics, can be specified deterministically or randomly in an optional manner.

[0027] For the purposes of the following implementation plan, For the input group (Eingabemenge). For - possibly including input - output groups, and The system model and the real system are treated as functions that can only be observed through simulation (11) or experimental measurement (21) for a limited number of inputs. Furthermore, let... To simulate the model error SMerrorX, which is the distance metric or error metric between two corresponding outputs (14). Finally, let It is a group of all inputs, and for this group, the error metric (14) takes the value ϵ.

[0028] Starting from these definitions, for each input The deviation of the satisfaction of the requirement (13) can be restricted upward by a term as follows, which is related to... and None of them are relevant: Formula 1 This yields the classifier (18) from Formula 2. Formula 2 .

[0029] The simulation model is here In the following sense, it is classified as reliable, namely, and It is consistent with p. Note that the classifier (18) requires the reciprocal of q (Umkehrung).

[0030] The main advantage of this formulation is that the virtual test classifier (18) can be used with... and It is formulated irrelevantly because it is only related to the degree of satisfaction of quantitative requirements (13) and error measure (14). From the single error measure (14) and complex number n Starting from a quantitative requirement, it is possible to calculate n One virtual test classifier (18) is computed for each quantitative requirement. Therefore, the model only needs to be validated once with respect to the error metric (14), without needing to be validated for each individual requirement, for example.

[0031] For complex numbers m An error measure and a complex number n A quantitative requirement allows this consideration to be generalized in a simple way, where... m Usually very small, and n Usually large. In this case, it can be calculated. A virtual test classifier (18). If one of these classifiers (18) provides a value W, the simulation result can be considered reliable. This achieves a more accurate classification because some error measures (14) may be better suited to specific requirements than others.

[0032] Alternatively, a virtual test classifier (18) can be defined within a random domain, where the input is assumed to be randomly distributed based on an arbitrary probability density function. For this purpose, let... The cumulative distribution function representing the deviation of satisfaction (13) under the assumption that the error metric (14) takes the value ϵ. The threshold value is the probability of making the correct decision using the classifier (18). -The value τ is thus usually close to 1-, so the virtual test classifier (18) can be defined as follows: Formula 3 Figure 4 The method (10) according to the present invention is explained from an application perspective under the following assumptions: • A model for simulation (11) and a set of tests (12) are given along with defined input parameters. • The requirement QSpec is quantifiable and pre-defined, and is implemented within the scope of a monitoring system that analyzes the tests (12) in terms of the degree of satisfaction (13) of these requirements. In this figure, both degrees of satisfaction (13) involve the same requirement QSpec, but the requirement is evaluated once by simulation (11) and once by experimental measurement (21) on the system. •SMerrorX is a predefined error metric (14). For some test inputs, simulations (11) and measurements (21) have already been performed, and the error metric (14) generalizes the corresponding test (12) to a new, previously unperformed experiment with a certain reliability, for example determined by upper and lower bounds for the error metric (14). For classifiers (18- Figures 1 to 3 Only the most unfavorable, i.e., the highest, error metric is considered (14). It should be noted that the classifier can be used to further refine the error metric (14). Within the scope of the following implementation, for the sake of simplicity, the classifier is equated with the grading performed by the classifier (15).

[0033] Under these assumptions, method (10) can be designed as follows: 1. Based on the explanation given above, a classifier is defined for each error model (31). It should be noted that this method (10) can be generally interpreted as a simulation and model quality method. 2. Perform test (12) by means of simulation (11), wherein an output signal is generated. 3. Analyze these output signals in terms of the degree of satisfaction of the QSpec requirement (13) and the error metric (14) for each error model (31). 4. For each error model (31), the classifier based on it is ranked in the following levels (A, B, C, D-). Figure 1The test (12) is classified in one of the following ways: (15) the test (12) is successfully completed in the simulation (11) and its result is reliable (16); the test fails in the simulation (11) and its result is reliable (16); or the result of the simulation (11) is unreliable. 5. In relation to the multiple tests (12) that are classified as reliable (16), select (32) from multiple error metrics (14), wherein the classifier is selected according to one of the following criteria: a) The results are classified as the maximum possible number of all tests (12) for reliability (16). b) The maximum possible number of tests (12) that failed and whose results were classified as reliable (16), - therefore, focus on the key tests - or c) The maximum possible number of successful tests (12) whose results are graded as reliable (16), - therefore, attention is focused on the tests that have passed. 6. Return the error model of the selected classifier.

[0034] like Figure 5 As shown in the diagram, the method (10) can be implemented, for example, in software or hardware or in a hybrid form of software and hardware, for example in a workstation (30).

Claims

1. A method (10) for inspecting a technical system, characterized in that... The following characteristics: - In the preparation phase (20), the simulation model is validated by experimental measurements on the technical system; - To perform multiple tests using the simulation model of the technical system; - The test is analyzed one by one for the degree of satisfaction of the quantitative requirements of the technical system (13) and the multiple different error measures of the simulation model; - Based on the satisfaction level (13) and each error metric, classify the plurality of tests as reliable or unreliable; and - Based on the number of all tests that are classified as reliable, select from the multiple different error metrics.

2. The method (10) according to claim 1. Its features The following characteristics: - For each error metric, the classification is performed based on the feature vector by a classifier (18); and - The satisfaction degree (13) and the corresponding error metric form the components of the feature vector.

3. The method (10) according to claim 2. Its features The following characteristics: - The classifier (18) maps the feature vector to one of multiple levels A, B, C, D; and - The grading is carried out within a pre-given decision boundary (19) between the grades A, B, C, and D.

4. The method (10) according to claim 3. Its features The following characteristics: - The decision boundary (19) is set in such a way that the satisfaction level in the simulation model and the satisfaction level (13) used in the experimental measurement deviate as little as possible; and - Independently select other tests to be performed in the preparation phase (20).

5. The method (10) according to claim 3. Its features The following characteristics: - Define a classifier for the corresponding error metric by solving the system of equations (18); and - The set of equations includes the definition equations of the satisfaction degree (13) and the corresponding error metric.

6. The method (10) according to any one of claims 1 to 5. Its features The following characteristics: - The analysis is performed as follows: when the technical system meets the quantitative requirements, the satisfaction degree (13) is positive, and when the technical system does not meet the quantitative requirements, the satisfaction degree is negative.

7. The method (10) according to any one of claims 1 to 5. Its features The following characteristics: - The selection falls on the error metric where the total number of tests classified as reliable is the largest for that error metric; or - The selection falls on the following error metric: for this error metric, the number of tests classified as reliable is the largest, and according to these tests, the technical system meets the quantitative requirements; or - The selection falls on the following error metric: for this error metric, the total number of tests classified as reliable is the largest, and according to these tests, the technical system does not meet the quantitative requirements.

8. The method (10) according to any one of claims 1 to 5, characterized in that, The defects in the technical system identified through the inspection are used to automatically improve the technical system.

9. The method (10) according to any one of claims 1 to 5, characterized in that, The technological system is at least a partially autonomous robot or at least a partially autonomous vehicle.

10. A computer program product configured to implement the method (10) according to any one of claims 1 to 9.

11. A machine-readable storage medium on which a computer program is stored, the computer program being used to implement the method (10) according to any one of claims 1 to 9.

12. Provision of means for implementing the method (10) according to any one of claims 1 to 9.

Images