Method and apparatus for evaluating and selecting signal comparison metrics

Abstract

The present invention relates to a method and apparatus for evaluating and selecting signal comparison metrics. The method (20) for evaluating a simulation model (22) is characterized by the following features: - calculating a first performance index (24) in the simulation model (22) for a selected test case (21); - obtaining a second performance index (24) in a real test environment (23) for the same test case (21); - forming a difference (25) between the first performance index (24) and the second performance index (24) and determining a signal metric (26) for each test case in the test case (21); - examining the relationship (27) between the difference (25) and the corresponding signal metric (26) for each of the signal metrics (26); and - selecting (28) the signal metric (26) that has the closest relationship (27) with the difference (25).

Description

Technical Field

[0001] This invention relates to a method for evaluating (Bewerten) simulation models. In addition, this invention relates to a corresponding device, a corresponding computer program, and a corresponding storage medium. Background Technology

[0002] In software technology, using models to automate testing activities and generate test artifacts during the testing process is summarized by the overarching concept of "model-based testing (MBT)." A familiar example is generating test cases from a model that describes the target characteristics (Sollverhalten) of the system to be tested.

[0003] In particular, embedded systems rely on the decisive input signals from sensors and, in turn, simulate their environment by providing output signals to vastly different actuators. During the verification and pre-development phases of such systems, the system's model (model-in-the-loop, MiL), software (software-in-the-loop, SiL), processor (processor-in-the-loop, PiL), or the entire hardware (hardware-in-the-loop, HiL) is simulated along with a model of the environment, thus within the regulation loop. In automotive technology, depending on the testing phase and the test object, simulators used to verify electronic control devices, corresponding to this principle, are sometimes called component benches, module benches, or integration benches.

[0004] DE10303489A1 discloses a method for testing software of a control unit of a vehicle, wherein a regulation segment controllable by the control unit is at least partially simulated by a test system in such a way that the control unit generates output signals, and these output signals of the control unit are transmitted to a first hardware component via a first connection, and signals from a second hardware component are transmitted to the control unit as input signals via a second connection, wherein the output signals are provided in the software as first control values, and are additionally transmitted to the test system in real time relative to the regulation segment via a communication interface.

[0005] Such simulations are popular in various technical fields and are used, for example, to test the applicability of embedded systems in power tools, engine control devices for drive, steering, and braking systems, or in fully autonomous vehicles during their early stages of development. Nevertheless, results from simulation models based on existing technologies are only included in approval decisions to a limited extent due to a lack of confidence in their reliability. Summary of the Invention

[0006] This invention provides a method for evaluating a simulation model, a corresponding device, a corresponding computer program, and a corresponding storage medium, as described in the independent claims.

[0007] The solution according to the invention is based on the understanding that the quality of the simulation model is crucial for the correct predictability of the test results achievable thereby. In the field of MBT, a key sub-topic focuses on the task of comparing real measurements with simulation results. For this purpose, various metrics, metric values, or other comparators are used to connect signals together, and these are generally referred to below as signal metrics (SMs). Examples of such signal metrics are those comparing magnitude, phase shift, and correlation. Several signal metrics are defined by standards (e.g., ISO 18571).

[0008] During verification, the system under test (SUT) is typically examined based on requirements, specifications, or performance characteristics. These requirements, specifications, or performance characteristics are then summarized as key performance indicators (KPIs). It's important to note that Boolean requirements or specifications can often be converted into quantitative measurements using forms such as signal temporal logic (STL). KPIs can be evaluated either based on a real physical implementation or a simulation.

[0009] The difference between KPI and signal metrics Figure 1The signal (S1) measured in the real test environment and the signal (S2) obtained in the simulation are similar here. Therefore, the relevant signal metric is small, but the performance index (KPI) of the two signals is above a pre-given threshold (10), which separates the favorable index value (11) from the unfavorable index value (12). The performance index (KPI) is therefore to be evaluated as unfavorable (12) in this invention. The absolute offset indicated by reference numeral 13 should not be attributed in this case to a misrepresentation of the signal metric, but rather to a misrepresentation of the performance index (KPI).

[0010] Further elaboration is based on the concepts explained below.

[0011] Signal metrics are measures of the similarity between two signals, typically comparing signals from real experiments with signals from simulations. A signatur is... , where S indicates the basic quantity of the possible signal.

[0012] KPIs are metrics that, in a way that is understandable to humans and measurable by computers, define how well a system performs (indicated by signals): .

[0013] The requirement for signal s is based on the threshold t for the KPI. It defines whether the system characteristics represented by the signal are acceptable, and thus enables binary decision-making: .

[0014] Signal metrics and KPIs therefore have different labels. Correspondingly, signal metrics and KPIs deal with different content. For example, in... Figure 1 As shown, the signal metric between the real output signal (S1) and the simulated output signal (S2) can be small, but the two signals (S1, S2) may not meet the system requirements and thus have small or negative KPIs.

[0015] The proposed method also considers the situation where it is sometimes unclear which of the many signal metrics should be used when validating a simulation model based on measurements. This is especially true if the requirements or performance indicators for the entire target SUT have not yet been determined during validation. The described method addresses this problem and helps select the most appropriate signal metrics (based on defined KPIs).

[0016] The distinction between KPIs and requirements addresses the following issue: Clearly defined thresholds are often not specified. That is, defining thresholds may require gathering experience from trials and finding suitable trade-offs. Separating KPIs from requirements allows for postponing the determination of acceptable thresholds.

[0017] There are also cases where there is no meaningful relationship between the KPI and the signal metric. This is true if the KPI includes a reference signal and is defined based on the signal metric. In this case, the proposed method is only meaningful to a limited extent, as the result is irrelevant.

[0018] In summary, the advantage of the solution according to the invention lies in providing a mathematically positive standard for selecting signal metrics.

[0019] The measures described in the dependent claims may advantageously improve and enhance the basic concept described in the independent claims. Attached Figure Description

[0020] Embodiments of the present invention are shown in the accompanying drawings and will be described in more detail in the following description. Wherein:

[0021] Figure 1 A visualization of the difference between signal metrics and KPIs is shown.

[0022] Figure 2 The data flow diagram is shown according to the implementation forms representing different algorithm implementations.

[0023] Figures 3 to 7 The diagram shows cases with different ratios between ΔKPI and SM, and the resulting interrelationships.

[0024] Figure 8 and Figure 9 The method is shown according to possible implementation forms ( Figure 2 An exemplary diagram of the calculation of ), wherein the simulated signal S is held and only the measured signal mi changes.

[0025] Figure 10 A workstation according to a second embodiment of the invention is illustrated schematically. Detailed Implementation

[0026] The calculation according to the present invention is passed through Figure 2 To clarify, and following the basic idea: for the selected test case (21, 29), the change in value is calculated by the change in the output signal from simulation (22) and observation (23) of different real measurements. On the other hand, the changed signal metric (26) is calculated. According to the scheme of the invention, the calculation is further configured to... The relationship between the calculated values ​​and the signal metric (27). The signal metric with the closest correlation was selected (28). Value In this case, the difference (25) is indicated by the performance index (24) calculated in the simulation model (22) and the performance index (24) obtained in the real measurement environment (23).

[0027] Changes in simulation output are achieved by varying several simulation parameters (e.g., input variables). Changes in measurements can be achieved through repeated trials or by trying multiple times under different conditions (for example, with different parameters).

[0028] As already mentioned, signal measurement Map the two signals to real values. In contrast, KPI maps the signal (and optionally the initial SUT input X) to real values. Therefore, functions SM and KPI have different flags, and thus the calculation is performed differently. and The interrelationship between them.

[0029] However, the commonly used definition of correlation is unsuitable because (unlike in...) Figure 3 (As shown in the ideal case) it is possible that, despite the different simulated signals and measurements, Or SM is constant. In this case, the variance... or A value equal to zero makes the corresponding correlation coefficient undefined. Considering, for example, in... Figures 4 to 7 The situation and numerical instability shown in the figure demonstrate that the modified correlation is merely expedient:

[0030] Equation 1

[0031] in It is the XOR operator.

[0032] It should be noted that Equation 1 can also use other functions, such as covariance with the described modifications.

[0033] For example, in software or hardware, or in a hybrid form consisting of software and hardware, the method (20) can be implemented, for example, in a control device, such as... Figure 10 As clearly explained in the diagram.

Claims

1. A method (20) for evaluating a simulation model (22) of a robot or vehicle that is at least partially autonomous. Its features The following characteristics: - For the selected test scenario (21), calculate the first key performance indicator (KPI) (24) in the simulation model (22), where the first key performance indicator (KPI) is a measure that defines how well the system performs. - For the same test situation (21), the second key performance indicator (KPI) (24) is obtained in the real test environment (23). - For each test case in the test case (21), form a KPI difference (25) between the first key performance indicator (KPI) (24) and the second key performance indicator (KPI) (24), and determine at least one signal metric among a variety of different types of signal metrics (26), wherein each signal metric is a measure of the similarity between two signals. - For each of the signal metrics (26), a corresponding correlation coefficient is determined, which defines the relationship (27) between the KPI difference (25) and the corresponding signal metric (26), and - Select (28) the following signal metric (26): The correlation coefficient of the signal metric (26) indicates the closest correlation (27) with the KPI difference (25).

2. The method (20) according to claim 1. Its features are, The signal metric (26) involves at least one of the following: - Signal strength, - Phase shift, or - Correlation.

3. The method (20) according to claim 1. Its features are, Select the test case (21) according to one of the following methods (20): - Random methods, - Quantification of uncertainty, or - Search-based testing methods.

4. The method (20) according to any one of claims 1 to 3. Its features The following characteristics: - By changing the parameters of the simulation model (22), the first key performance indicator (KPI) (24) is changed.

5. The method (20) according to any one of claims 1 to 3. Its features The following characteristics: - By repeating the test case (21), the second key performance indicator (KPI) (24) is changed.

6. The method (20) according to any one of claims 1 to 3. Its features The following characteristics: - If not only the difference (25) but also the signal metric (26) is constant, then the interrelationship (27) is defined as one. - Otherwise, if the difference (25) or the signal metric (26) is constant, then the interrelationship (27) is defined as zero, and - Otherwise, the interrelationship (27) is defined by the degree of correlation between the difference (25) and the signal metric (26).

7. The method (20) according to claim 6. Its features The following characteristics: - The degree of correlation is the correlation coefficient.

8. The method (20) according to any one of claims 1 to 3, characterized in that, Based on the selected signal metric (26), the errors of the system modeled by the simulation model (22) identified by the signal metric (26) are automatically improved.

9. A computer program product having a computer program configured to perform the method (20) according to any one of claims 1 to 8 when it is run on a processor.

10. A machine-readable storage medium on which a computer program is stored, the computer program being configured to perform the method (20) according to any one of claims 1 to 8 when it is run on a processor.

11. An apparatus configured to perform the method (20) according to any one of claims 1 to 8.

Images