Method and apparatus for inspecting a technical system

Abstract

Method (10) and device for checking a technical system, characterized by the following features: a test (12) is performed by means of a simulation (11) of the system, the test (12) is evaluated in terms of a measure of fulfillment (13) of quantitative requirements of the system and a measure of error (14) of the simulation (11), the test (12) is classified as reliable or unreliable depending on the measure of fulfillment (13) and the measure of error (14), and a threshold value (34) applicable to the measure of error (14) is derived (31) from the classification by a user (33) and a predefinition (32) associated therewith.

Description

Technical Field

[0001] This invention relates to a method for inspecting a technical system. Furthermore, the invention also relates to a corresponding device, a corresponding computer program, and a corresponding storage medium. Background Technology

[0002] In software technology, the overarching concept of "model-based testing" (MBT) summarizes the use of models to automate testing activities and generate test artifacts during the testing process. For example, it is well known that test cases are generated from models that describe the target behavior of the system under test.

[0003] In particular, embedded systems rely on deterministic input signals from sensors and stimulate their environment by outputting signals to vastly different actuators. Therefore, during the verification and early development phases of such systems, the system's model in the loop (MiL), software in the loop (SiL), processor in the loop (PiL), or hardware in the loop (HiL) is simulated along with the environment model within a controlled loop. In automotive technology, simulators used to inspect electronic control equipment according to testing phases and test objects, corresponding to this principle, are sometimes called component test benches, module test benches, or integration test benches.

[0004] DE10303489A1 discloses a method for testing software of a control unit in a vehicle, power tool, or robot system, wherein an adjustment segment controllable by the control unit is at least partially simulated by a test system in such a way that an output signal is generated from the control unit and the output signal of the control unit is transmitted to a first hardware module via a first connection, and a signal from a second hardware module is transmitted to the control unit as an input signal via a second connection, wherein the output signal is provided as a first control value in the software and is additionally transmitted to the test system in real time relative to the adjustment segment via a communication interface.

[0005] Such simulations are widely used in various technical fields, and are used, for example, to examine the suitability of embedded systems in power tools, motor control devices for drive, steering, and braking systems, camera systems, systems with artificial intelligence and machine learning components, robotic systems, or autonomous vehicles in their early development stages. However, due to a lack of confidence in the reliability of simulation model results based on existing technologies, these results are only incorporated to a limited extent into release decisions. Summary of the Invention

[0006] This invention provides a method, corresponding apparatus, corresponding computer program, and corresponding storage medium for examining a technical system according to the independent claims.

[0007] The solution according to the invention is based on the understanding that the quality of the simulation model plays a decisive role in the correctness and predictability of test results that can be achieved using the simulation model. In the field of MBT (Multi-Level Testing), the sub-discipline of validation has the task of comparing real measurements with simulation results. For this purpose, various metrics, numerical values, or other comparators are used, which should logically correlate signals with each other and are collectively referred to below as signal metrics (SMs). Examples of such signal metrics are metrics that compare magnitude, phase shift, and correlation. Some signal metrics are defined by relevant standards, such as according to ISO 18571.

[0008] Generally, uncertainty quantization techniques support the estimation of simulation and model quality. The result of evaluating model quality using signal metrics, or more generally using uncertainty quantization methods, for a specific input X (which can be parameters or a scenario), is referred to below as the simulation model error metric (SMerrorX). To generalize (interpolate and extrapolate) SMerrorX for previously unexamined inputs, parameters, or scenarios X, machine learning models can be used, such as machine learning models based on so-called Gaussian processes.

[0009] During verification, the system under test (SUT) is typically inspected based on requirements, specifications, or performance metrics. It should be noted that Boolean requirements or specifications can often be converted into quantitative measurements using formalism such as signal temporal logic (STL). This formalism serves as the basis for quantitative semantics, where positive values ​​indicate satisfaction of the requirement and negative values ​​indicate violation; this quantitative semantics represents the generalization of verification. In the following text, such requirements, specifications, or performance metrics are collectively referred to as “quantitative requirements” (QSpec).

[0010] This quantitative requirement can be checked based on a real System Under Test (SUT) or a model of a real SUT (i.e., a "virtual SUT"). To perform this verification, a catalog is compiled with the test conditions that the SUT must meet to determine whether the SUT possesses the expected performance and security characteristics. These test conditions can be parameterized to cover any number of individual tests.

[0011] In this context, the proposed solution considers the need for reliable test results to guarantee the performance and safety characteristics of the SUT. It is precisely by performing tests based on simulations of the system or sub-components (rather than the real system) that the reliability of the simulation results is ensured.

[0012] Validation techniques are used to estimate the degree of agreement between a simulation model and actual measurements. The comparison between simulation results and actual measurements is performed using the validation error metric SMerrorX, which is generalized through interpolation and extrapolation, allowing the prediction of the error metric for a new input X without requiring a corresponding measurement. However, the prediction of SMerrorX is related to uncertainty, which can be modeled as an interval or probability distribution.

[0013] One problem faced by validation engineers is setting appropriate thresholds for the error metric SmerrorX. This method allows for the derivation of corresponding requirements for the validation model. A common difficulty arises because it is often unknown which values ​​of the validation metric are considered sufficient, leading to subjective interpretation of validation results. Therefore, the method according to the invention allows for the creation of guidelines for validation engineers that incorporate knowledge gained within the scope of earlier product releases and systematically specify the requirements for the error metric.

[0014] The advantage of this solution lies in its ingenious combination of two approaches, contrary to concepts based solely on validation or verification. To this end, a "virtual test classifier" is introduced, which combines the requirements of model validation and product testing. This is achieved by logically associating information from simulation and model quality (SMerrorX) validation on the one hand with information from test requirements (QSpec) on the other.

[0015] The corresponding tests can be applied in very different fields. For example, the functional safety of automated systems, such as those used for automated driving, should be considered.

[0016] The measures listed in the dependent claims can be used to advantageously extend and improve the basic ideas described in the independent claims. This allows for the establishment of automated, computer-implemented testing environments to significantly automate the improvement of the quality of the hardware or software products under test. Attached Figure Description

[0017] Embodiments of the invention are illustrated in the accompanying drawings, and these embodiments are explained in more detail in the following description.

[0018] Figure 1 A virtual test classifier is shown.

[0019] Figure 2 A first scheme for generating decision boundaries for classifiers based on data is shown.

[0020] Figure 3 A second scheme for generating the decision boundary of a classifier based on the formula solution is shown.

[0021] Figure 4 A description of the method according to the invention from an application perspective is shown.

[0022] Figure 5 The workstation is shown schematically. Detailed Implementation

[0023] According to the present invention, the simulation model error SMerrorX is evaluated within the range of test X, and the quantitative specification QSpec is estimated based on the simulation of the SUT, wherein test X can be obtained from a test catalog as a test case or as an instance of a parametric test. A virtual test classifier uses SMerrorX and QSpec as inputs and makes a binary determination of whether the test results based on the simulation are reliable.

[0024] Based on the language commonly used in informatics, and especially in pattern recognition, a classifier here should be understood as any algorithm or mathematical function that maps a feature space to a set of categories that are formed and bounded together during the classification process. In order to determine which category an object should be classified into (often also referred to as "classification"), a classifier uses so-called category boundaries or decision boundaries. The term "classifier" is used in technical language if the distinction between method and instance is not important, and is also partially synonymous with "classification" or "categorization" below.

[0025] Figure 1This classification in the current application example is illustrated. Here, each point corresponds to a test performed through simulation and for which the required QSpec satisfaction metric 13 and error metric 14 SMerrorX are calculated. In this case, QSpec is defined such that it takes a positive value (reference numeral 24) when the test indicates that the system meets the corresponding requirement, and a negative value (reference numeral 25) when the system does not meet the requirement.

[0026] As shown in the figure, the decision boundary 19 of classifier 18 subdivides the space into four categories A, B, C, and D. The system will pass the test for category A with high reliability. For the tests of categories B and C, the simulation only provides unreliable results; therefore, such tests must be performed on a real system. The test for category D will fail on the system with high reliability.

[0027] The virtual test classifier 18 is based on the following considerations: only when the assumed model error 14 is at most a marginal error can the requirements that are barely met in the simulation be used to replace the testing of the real system. On the other hand, when the absolute value of the satisfaction metric 13 of the quantitative requirement "QSpec" is high (i.e., it has far exceeded the predetermined value or clearly missed the predetermined value), a certain deviation between the simulation results and the corresponding experimental measurements is acceptable.

[0028] Since this evaluation method presupposes an understanding of the model error SMerrorX of the simulation model, it is assumed that the simulation model has been validated and verified before using the virtual test classifier 18. Within the scope of verification, for example, a generalized model should be formed through machine learning based on Gaussian processes or other methods, which provides SMerrorX for a given X. It should be noted that the reliability of the simulation largely depends on the correctness of this generalized model.

[0029] Figure 2 The decision boundary 19 for generating classifier 18 based on data is shown. Figure 1 Possible solutions. In preparation phase 20, simulation 11 is validated by actual measurements 21 on the system. A decision boundary 19 is drawn such that the satisfaction metric 13 obtained, on the one hand in simulation 11 and on the other hand in measurement 21, deviates as little as possible. Preferably, 22 is automatically selected to perform other tests in preparation phase 20. In the simplest case, boundary 19 is distributed along a straight line passing through the origin. The slope of this line is preferably chosen such that all the following points lie in regions C and B, and these regions are also as small as possible, at which points the quantitative requirement for the satisfaction metric 13 of QSpec to differ in sign between simulation 11 and actual measurements 21, i.e., equivalent to all tests 12 where the simulation model fails.

[0030] Furthermore, considering a more general decision boundary 19, such as a polynomial, the function curve of this decision boundary is adapted using linear programming so that it satisfies the criteria of classifier 18 VTC. In this case, all the following points also lie in regions C and B, where the quantitative requirement for the satisfaction metric 13 of QSpec is to differ in sign between simulation 11 and actual measurement 21, i.e., equivalent to all tests 12 where the simulation model fails.

[0031] Figure 3 An alternative scheme is shown for defining classifier 18 by solving a system of equations in the form of formula 23, satisfying the defining equations for metric 13 and error metric 14 based on this system of equations. The resulting function can be described deterministically or stochastically, assigning truth values ​​to the feature vectors 13, 14 formed by these two metrics.

[0032] For the following execution, assume I is the input set, O is the output set (which may also include input), and It is a system model and a real system as a function, and the function can only be observed through simulation 11 or experimental measurement 21 for a limited number of inputs. Furthermore, it is assumed that... This is the simulation model error SMerrorX, which is the distance or error metric between two corresponding outputs. Finally, it is assumed... It is the set of all the following inputs, for which the error metric 14 takes a value. .

[0033] Starting from these definitions, for each input The deviation from the requirement to meet metric 13 can be upwardly constrained by the following terms, which depend neither on m1 nor on m2: Formula 1 .

[0034] Therefore, classifier 18 yields Formula 2 .

[0035] Here, in In the case where m1 and m2 are consistent with p, the simulation model is classified as reliable. It should be noted that classifier 18 requires the reciprocal of q.

[0036] The main advantage of this representation is that the virtual test classifier 18 can be formulated independently of m1 and m2, because the virtual test classifier depends only on the satisfaction metric 13 and error metric 14 of the quantitative requirements. Therefore, starting from a single error metric 14 and a complex number of n quantitative requirements, n virtual test classifiers 18 can be computed, one for each requirement. Thus, the model only needs to be validated once for the error metric 14, instead of, for example, for each individual requirement.

[0037] For a complex number of m error measures and a complex number of n quantitative requirements, this examination can be generalized in a simple way, where m is typically very small and n is large. In this case, n·m virtual test classifiers 18 can be computed. If one of these classifiers 18 provides a value W, the simulation result can be considered reliable. This allows for more accurate classification because some error measures 14 may be better suited to a particular requirement than others.

[0038] Alternatively, a virtual test classifier 18 can be defined within a random range, in which the input is assumed to be randomly distributed according to an arbitrary probability density function. This indicates the value taken in error metric 14. Under the assumption that the conditional cumulative distribution function of the bias of metric 13 is satisfied, and with the threshold of the probability of classifier 18 making a correct judgment being τ∈(0.1)—the value τ is therefore typically close to 1—the virtual test classifier 18 can be defined as follows: Formula 3 .

[0039] Figure 4 The method 10 according to the present invention is illustrated from an application perspective under the following assumptions: • The model for simulation 11 and the set of tests 12 are given, along with the defined input parameters. • The requirement QSpec is quantifiable and pre-given, and implemented within the scope of a monitoring system that evaluates test 12 against a satisfaction metric 13 for these requirements. In this figure, both satisfaction metrics 13 relate to the same requirement QSpec, but one is estimated based on simulation 11 and the other is estimated through experimental measurement 21 on the system. • SMerrorX is a predefined error metric 14. Thus, simulation 11 and measurement 21 have been performed for some test inputs, and error metric 14 generalizes the corresponding test 12 to a new, previously unperformed experiment with a certain reliability, determined for example by an upper and lower bound of error metric 14. For classifier 18 (see below), only the most unfavorable, i.e., the highest, error metric 14 is used. It should be noted that classifier 18 can be used to further improve error metric 14.

[0040] Under these assumptions, method 10 can be designed as follows: 1. Perform test 12 using simulation 11, where an output signal is generated. 2. Define the initial classifier 18 based on the above explanation. 3. User 33 pre-defines a desired classifier for 32 based on the same data points, which allows for the violation of the correctness conditions usually imposed on classifiers. 4. Based on the SMerrorX error model, the output signal is evaluated in terms of both the requirement QSpec satisfaction metric 13 and the error metric 14 of simulation 11. For example, if the mean square error is used as the error metric 14 and the result is 1.5, then method 10 issues a suggestion to user 33 to achieve the target value corresponding to the pre-given value 32. 5. Typically, according to pre-given 32, the decision boundary 19 extends along a steeper straight line, thus more tests 12 are classified as reliable. If, according to pre-given 32, tests 12 that have been classified as unreliable by the currently used classifier 18 should be classified as reliable, then a threshold 34 of error metric 14 is determined, which repositions the decision boundary 19 of classifier 18 such that the considered test 12 instead falls into the threshold. Figure 1 In category B or C shown above the middle boundary 19. Confirm that the engineer uses this value as a criterion or standard to determine whether the determined error metric should be considered "good" or "bad" 36.

[0041] like Figure 5 As shown in the schematic diagram, method 10 can be implemented, for example, in software or hardware or a combination of software and hardware, for example in workstation 30.

Claims

1. A method for inspecting a technical system, Its features The following characteristics: The test is performed by simulating the system using a simulation model. The test is evaluated in terms of a measure of the degree to which the quantitative requirements of the system are met and a measure of the error in the simulation, wherein the measure of meeting the quantitative requirements of the system is a measure of the degree to which the simulation model is consistent with actual measurements. The test is classified as reliable or unreliable based on a satisfaction metric and an error metric, wherein the classification is performed by a classifier based on a feature vector, and the satisfaction metric and error metric are components of the feature vector. A threshold suitable for the error metric is derived from the user-defined classifications and related pre-defined parameters. The classifier maps the feature vector to one of multiple categories, and the classification is performed within a pre-given decision boundary between the categories. The plurality of categories includes at least three categories: a first category indicating that the test has passed, a second category indicating that the test has failed, and a third category indicating that the results provided by the simulation are unreliable and must be tested on a real system, wherein the pre-given decision boundary is selected such that the region of the third category is as small as possible.

2. The method according to claim 1, It is characterized by the following features: The simulation was validated through experimental measurements on the system during the preparation phase. The decision boundary is drawn such that the satisfaction metric obtained, on the one hand, in the simulation and on the other hand, in the measurement, deviates as little as possible.

3. The method according to claim 1, Its features The following characteristics: The classifier is defined by solving a system of equations, and The set of equations includes the defining equations that satisfy both the metric and the error metric.

4. The method according to any one of claims 1 to 3, It is characterized by the following features: The evaluation is performed as follows: if the system meets the requirements, the satisfaction metric is positive; if the system does not meet the requirements, the satisfaction metric is negative.

5. The method according to any one of claims 1 to 3, Its features The following characteristics: Add the threshold to the criteria used to validate the simulation.

6. The method according to claim 1, It is characterized by the following features: The technological system is at least partially autonomous robots or vehicles.

7. A computer program product configured to perform the method according to any one of claims 1 to 6.

8. A machine-readable storage medium having a computer program product according to claim 7 stored thereon.

9. An apparatus configured to perform the method according to any one of claims 1 to 6.

Images