Computer-implemented method for determining validation metric

Abstract

To provide a computer-implemented method for determining a validation metric.SOLUTION: A computer-implemented method for determining a validation metric for determining model quality includes the steps of: receiving a first data series comprising results of simulation; receiving a second data series comprising results of measurement; receiving visual validation of the first data series based on a comparison to the second data series to obtain an assessment of the first data series; applying a plurality of predetermined metrics to the first data series and the second data series; selecting one or more metrics from the plurality of metrics based on a predetermined condition; and determining a validation metric based on optimization of parameters using the selected one or more metrics and the assessment of the first data series.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a method for determining validation metrics for determining model quality. Furthermore, the invention relates to a corresponding computer system, a corresponding computer program, and a corresponding computer-readable medium or signal. [Background technology]

[0002] Highly automated or autonomous systems are receiving increasing attention, for example in the fields of robotics and automobiles. Simulations and models play a key role in the development of technologies such as autonomous driving. They allow data to be collected without necessarily relying on physical test setups. Advanced simulation software allows complex scenarios to be mimicked in a virtual environment, covering a large number of test cases that may be difficult to replicate in the real world. Virtual testing allows engineers to efficiently evaluate and optimize various algorithms, sensor configurations, and driving scenarios without having to perform costly and time-consuming physical tests. In this way, simulations and models accelerate the development process and contribute to the safety and reliability of autonomous vehicles.

[0003] In addition to reliability and objectivity, validity is especially used as a quality criterion for models, measurement methods, and test methods. In the context of models for simulating technological processes, validation is a subprocess during model construction. This validation aims to answer a central question within the framework of quality assurance: Is the simulation suitable for its intended use? Only through the validation process can the necessary proof of quality be provided that the simulation results reflect reality or are suitable for the intended use and can be used in further product development stages.

[0004] In this context, so-called operational validation is used to assess the quality of executable simulation models, which is of great practical importance, for example, in vehicle technology, since it directly compares the behavior of a virtual vehicle with the behavior of a real vehicle. To perform it, it is not necessary to know the conceptual model on which the simulation is based, which can be very complex. Since operational validation is based on an experimental comparison of simulation data with measurement data, it is applicable to a wide variety of simulation models and tools. Summary of the Invention

[0005] A first general aspect of the present disclosure relates to a computer-implemented method that includes receiving a first data series including results of a simulation, receiving a second data series including results of a measurement, receiving a visual validation of the first data series based on a comparison with the second data series to obtain a rating of the first data series, applying a plurality of predetermined metrics to the first data series and the second data series, and selecting one or more metrics from the plurality of metrics based on preconditions, and optimizing parameters using the selected one or more metrics and determining validation metrics based on the rating of the first data series.

[0006] A second general aspect of the present disclosure relates to a computer system configured to perform a computer-implemented method for determining validation metrics for determining model quality according to the first general aspect (or an embodiment thereof).

[0007] A third general aspect of the present disclosure relates to a computer program configured to perform a computer-implemented method for determining validation metrics for determining model quality according to the first general aspect (or an embodiment thereof).

[0008] A fourth general aspect of the present disclosure relates to a computer-readable medium or signal storing and / or including a computer program according to the third general aspect (or an embodiment thereof).

[0009] The method according to the first general aspect (or an embodiment thereof) proposed in the present disclosure can be used to provide a computer-implemented method for determining validation metrics for determining model quality. This method can be used to provide objective validation metrics for determining the (comparable) quality of (simulation) models. In an example, the validation metrics can be stored in a database and used to validate future simulations. In an example, the validation metrics can be provided via a web app and can be scaled accordingly. The disclosed method can reduce the need for physical test benches and / or test setups in development, facilitating the use of simulations and models in product development. This can be advantageous in reducing development time. In a further example, the method can be used to incorporate related metrics into the validation metrics, e.g., to remove correlated metrics. A further advantage can be seen in the ability to determine confidence intervals, which can be used to check the validation metrics and, for example, provide information regarding possible updates to the validation metrics.

[0010] A further advantage is that the models validated by the validation metrics can be used to develop, test, validate, and / or verify vehicle functions, robot functions, building automation functions, power tool automation functions, and / or home appliance automation functions. In an example, the data can be used to train, test, and / or validate machine learning models. A further advantage can be that the machine learning models can be used to control and / or regulate vehicle functions, robot functions, building automation functions, power tool automation functions, and / or home appliance automation functions.

[0011] In this disclosure, several terms are used as follows: A "vehicle" may be any device that transports passengers and / or cargo. A vehicle may be an automobile (e.g., a car or truck) or a rail car. A vehicle may be a motorized two-wheeled or three-wheeled vehicle. However, floating and flying devices may also be vehicles. A vehicle may operate at least in part autonomously or may be assisted. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an exemplary method for determining validation metrics for determining model quality. [Figure 2] FIG. 2 is a schematic diagram of a first data series and a second data series. [Figure 3] FIG. 1 is a schematic representation of the ratings of the first data series of several simulations 1-4 in the form of a box plot. [Figure 4] FIG. 1 is a schematic diagram of an exemplary architecture for performing a method for determining validation metrics for determining model quality. DETAILED DESCRIPTION OF THE INVENTION

[0013] FIG. 1 is a flow diagram illustrating possible steps of a computer-implemented method 100 for determining validation metrics for determining model quality. A computer-implemented method 100 for determining validation metrics for determining model quality includes receiving 110 a first data series 11a including results of a simulation, receiving 120 a second data series 11b including results of a measurement, and receiving 130 a visual validation of the first data series 11a based on a comparison with the second data series 11b to obtain an assessment of the first data series 11a. The method includes applying 140 a plurality of predetermined metrics to the first data series 11a and the second data series 11b and selecting 150 one or more metrics from the plurality of metrics based on preconditions. The method 100 includes determining 160 validation metrics based on parameter optimization using the selected one or more metrics and the assessment of the first data series. In an example, applying 140 a plurality of predetermined metrics to the first data series 11a and the second data series 11b may include applying the plurality of predetermined metrics to a combination of the first data series 11a and the second data series 11b.

[0014] FIG. 2 shows, by way of example, a first data series 11a of a simulation in the form of a time series and a second data series 11b of measurements in the form of a time series. For example, the measurements may include rack and pinion steering position values. For example, a prototype of the rack and pinion steering may be mounted on a test bench. In an example, an electric cylinder may apply a sinusoidal (or sudden) force to the rack end. On the test bench, the position of the rack to be adjusted may be measured. In an example, the position of the rack to be adjusted may correspond to the second data series 11b. In an example, the same load case may be applied to the simulation. The simulation model may include a physical steering model and its associated position adjustment function. The output of the simulation may include the rack position according to the simulated load case. The output of the simulation may correspond to the first data series 11a.

[0015] Metrics can include mathematical operators that map two data series to a scalar, which is also sometimes called a metric. For example, some predetermined metrics can include the mean squared error, which can be expressed mathematically by the following equation:

[0016]

number

[0017] In an example, the first data series 11a and / or the second data series 11b may include a sampled time series, where the first data series 11a is Y i and the second data series 11b corresponds to Y iThe step 140 of applying the plurality of predetermined metrics may, in an example, include determining a reference quantity between the first data series 11 a and the second data series 11 b. In an example, the plurality of predetermined metrics may include metrics that distinguish between errors in phase, magnitude, and slope. These metrics from individual error components, such as the difference between two time points in the first data series 11 a and the second data series 11 b, may be used to determine a validation metric by weighted sum. In an example, the plurality of predetermined metrics may be used to provide a quantification of how well the simulation mimics the measurement.

[0018] In examples, the validation metric may include the sum of one or more addends. In examples, one or more addends may be weighted. For example, the optimized parameters may be weighted by addends c0, c1, c2, ..., c i The addends may contain one or more metrics M1, M2, ..., M i For example, validation metrics can be expressed mathematically as follows:

[0019] R=c0+c1M1+c2M2+c3M3+…+c i M i In the example, a validation metric may include a boundary condition that the parameters sum to the value 1: c0 + c1 + c2 + … + c i =1.

[0020] In an example, determining 160 validation metrics may include calculating a regression curve. In an example, the ratings of the first data series 11a may include the dependent variable of the regression curve, and the independent variables of the regression curve may include the parameters. For example, the calculated regression curve may include the validation metrics to be determined. For example, the dependent variable R may include the ratings of the first data series 11a. For example, the independent variables may include the parameters c0, c1, c2, ..., c iIt may include:

[0021] FIG. 4 illustrates in schematic form an exemplary architecture 10 for implementing the method for generating validation metrics for determining model quality. In an example, the visual validation may be based on an evaluation interval. The rating of the first data series 11a may include a vector having multiple entries. In an example, each entry may include an evaluation by an expert 12. For example, each entry of the vector may include a value from the evaluation interval. In an example, as shown in FIG. 4, multiple experts 12 may view the first data series 11a of simulations and the second data series 11b of measurements. The results may be one vector per simulation, and the vector may include multiple entries. The results of the visual validation provide a rating of the corresponding simulation result. In an example, the first data series 11a may include results of multiple simulations. In an example, the second data series 11b may include results of multiple measurements. In an example, the rating of the first data series 11a may include a vector for each simulation. The visual validation may be performed for multiple simulations and multiple measurements, whereby the rating of the first data series 11a includes a matrix. In an example, each column of the matrix may include a vector with multiple expert rating entries for each simulation. In an example, multiple simulations and associated measurements may be divided into a training data set and a test data set. In an example, the training data set may include a first data series 11a of a first number of simulations and a second data series 11b of a first number of measurements, which are used to determine validation metrics. In an example, the test data set may include a first data series 11a of a second number of simulations and a second data series 11b of a second number of measurements, which are used to test validation metrics. In an example, the ratio of the size of the training data set to the size of the test data set may be 80:20. In an example, the allocation of the training data set and the test data set may be (pseudo)random or based on a mixture of the respective first data series 11a and second data series 11b.For example, this can reduce the risk of masking slow changes over time in successively stored data series. For example, in a set of 10 measurements, the friction parameter can be adjusted over the first four measurements and left constant for the remaining six measurements.

[0022] FIG. 3 shows a schematic representation in the form of a box plot of the ratings of a first data series 11a of a number of simulations S1-S4. In an example, the evaluation interval may include values ​​between 0 and 10. In an example, the evaluation interval may include values ​​between 0 and 1 or 0 and 100, where 0 may represent the worst model quality. As shown in FIG. 3, for example, in the fourth simulation S4, it can be seen that multiple experts rated the first data series 11a, which includes the results of this third simulation, as a better simulation (closer to 1.0) and closer to each other in their ratings of that simulation than, for example, the fifth simulation S5. In an example, the ratings of the first data series 11a may be stored in a database 13.

[0023] In an example, the calculation of the regression curve may involve fitting parameters so that the regression curve, i.e. the validation metric to be determined, represents, to a predetermined extent, the assessment of the first data series 11a. To this end, the parameters c0, c1, c2, ..., c i The values ​​for σ are determined such that the regression curve represents the ratings of the first data series 11a to a predetermined extent. In an example, the regression curve may pass through the median values ​​of the respective ratings of the simulation. In an example, the parameter determination may include parameter optimization, where parameter optimization includes optimizing the parameters to a particular extent, but not necessarily to an (absolute) optimum value.

[0024] In an example, the step 150 of selecting one or more metrics from the plurality of metrics and the step of determining the validation metric may be performed by LASSO regression. In an example, individual metrics from the plurality of metrics may be correlated. In an example, the step 150 of selecting one or more metrics may include removing correlated metrics from the plurality of metrics. In an example, a precondition based on which the selection 150 is performed may include a correlation coefficient. This may be advantageous for reducing the number of metrics for the validation metric without reducing the accuracy of the validation metric. In an example, the selection 150 may be performed by LASSO (Least Absolute Shrinkage and Selection Operator) regression. Here, the sum of the absolute values ​​of the independent variables, i.e., the parameters to be determined, may be limited. In an example, this may cause some parameters to become zero, which may be synonymous with the metric weighted by the corresponding parameter being removed from the validation metric. In an example, the step 150 of selecting one or more metrics may be performed based on minimum-redundancy-maximum-relevant (mRMR) selection.

[0025] In an example, the method 100 may include dividing the first data series 11a and / or the second data series 11b into one or more sections. In an example, the plurality of predetermined metrics may include the length of one section of each of the data series 11a, 11b. In an example, the one or more sections may correspond to a temporal section. In an example, the length of a section may be defined by a first time point and a second time point. This can be used to provide validation metrics that are valid within a specific time section. This can be advantageous for providing validation metrics that are valid for time sections with different characteristics.

[0026] In examples, the plurality of predetermined metrics may include at least one of mean absolute error, mean squared deviation, median absolute deviation, cross-correlation, normalized mean squared deviation, and / or Sprague Geers.

[0027] In an example, when calculating the regression curve, a normal distribution of the disturbance factors may be assumed, and confidence intervals for the expectation and variance may be calculated. In an example, the disturbance factors may include external forces applied to rack-and-pinion steering. In an example, a non-parametric method for determining the confidence intervals may be used, such as "bootstrapping."

[0028] In an example, the method 100 may include recalculating the regression curve, for example, when the confidence interval of the validation metric becomes too large. In an example, the method may include validating the new simulation using the validation metrics. In an example, the new simulation may include a simulation of a vehicle function, a robot function, a building automation function, a power tool automation function, and / or a home appliance automation function.

[0029] In examples, the simulation and / or new simulation may include a simulation of a vehicle function (particularly for controlling a driving function). For example, the vehicle function may include a function for autonomous driving and / or assisted driving. In examples, the vehicle function may include adjustment and / or control of rack-and-pinion steering. For example, a simulation may apply specific load cases (external forces) to the rack-and-pinion steering. In examples, the simulation may include a position adjustment function for the rack-and-pinion steering. An output of the simulation may include a rack position depending on the load case. In examples, validation metrics may be used to determine model quality for the rack-and-pinion steering simulation.

[0030] In other examples, the simulation and / or new simulation may include simulation of a robot function (particularly for controlling a motor function of the robot). For example, the robot function may be a lateral guidance and / or longitudinal guidance function of the robot.

[0031] In one example, the simulation and / or new simulation may include a simulation of a building function (particularly for controlling building automation functions). For example, the building function may be a function for adjusting room temperature, lighting, and / or security devices.

[0032] In one embodiment, the computer-implemented method 100 may include applying validation metrics to the machine learning model to determine model quality. In an example, the method may include using the machine learning model to control and / or regulate a vehicle function, a robot function, a building automation function, a power tool automation function, and / or a home appliance automation function. In an example, the method 100 may include deploying the machine learning model to a computer system of the vehicle, robot, building, power tool, home appliance, machine tool, personal assistant, access control system, and / or medical device.

[0033] Further disclosed is a computer system configured to execute a computer-implemented method 100 for determining validation metrics for determining model quality. The computer system may include at least one processor and / or at least one memory. The computer system may further include a (non-volatile) memory. In an example, all steps of the method 100 may be performed by the computer system. In some examples, individual steps of the method 100 may be performed by the computer system. Optionally, results of individual process steps not performed by the computer system may be received by the computer system. In an example, the computer system may include a user interface for receiving visual validation of the first data series 11a based on a comparison with the second data series 11b. In an example, the computer system may include a cloud on which at least a portion of the computer-implemented method 100 is executed. In an example, the validation metrics may be provided to a user via a web app. In an example, the computer system may be configured to receive simulation results from a user via a CI / CD pipeline and display model quality of the simulation model.

[0034] Also disclosed is a computer program configured to perform the computer-implemented method 100 for determining validation metrics for determining model quality. The computer program may be, for example, in an interpretable or compiled format. The computer program may be loaded, for example, as a sequence of bits or bytes, into a computer's RAM for execution (even in part).

[0035] Further disclosed is a computer-readable medium or signal that stores and / or includes the computer program or at least a portion thereof. The medium may include, for example, one of a RAM, a ROM, an EPROM, a HDD, a SDD, etc., on which the signal is stored. [Explanation of symbols]

[0036] 10 Architecture 11a First data series 11b Second Data Series 12. Experts 13 Database 100 ways 110 receiving a first data series (11a) 120 receiving a second data series (11b) 130 receiving visual validation of the first data series 140 Steps for applying multiple predetermined metrics 150 Selecting one or more metrics from multiple metrics 160 Steps to determine validation metrics

Claims

1. 1. A computer-implemented method (100) for determining validation metrics for determining model quality, comprising: receiving (110) a first data series (11a) comprising the results of a simulation; receiving (120) a second data series (11b) comprising the results of the measurement; receiving (130) a visual validation of said first data series (11a) based on a comparison with said second data series (11b) to obtain a rating of said first data series (11a); applying (140) a plurality of predetermined metrics to the first data series (11a) and the second data series (11b), and selecting (150) one or more metrics from the plurality of metrics based on preconditions; determining (160) validation metrics based on the determination of parameters using the selected one or more metrics and the evaluation of the first data series; A computer-implemented method (100) comprising:

2. 2. The computer-implemented method of claim 1, wherein the validation metric comprises a sum of one or more addends, the one or more addends are weighted, and the optimized parameters comprise the weights of the addends, the addends comprising the one or more metrics.

3. 3. The computer-implemented method of claim 1, wherein the step of determining validation metrics comprises calculating a regression curve, the ratings of the first data series comprising a dependent variable of the regression curve, and independent variables of the regression curve comprising the parameters.

4. 4. The computer-implemented method (100) of claim 1, 2, or 3, wherein the visual validation is based on an evaluation interval, and the ratings of the first data series (11a) comprise a vector having multiple entries, each entry comprising an expert's (12) rating, and each entry of the vector comprising a value from the evaluation interval.

5. 5. The computer-implemented method of claim 1, wherein the step of selecting one or more metrics from the plurality of metrics and the step of determining the validation metric are performed by LASSO regression.

6. dividing the first data series (11a) and / or the second data series (11b) into one or more sections, wherein the plurality of predetermined metrics comprises the lengths of the sections of the respective data series (11a, 11b); The computer-implemented method (100) of any one of claims 1 to 5.

7. 7. The computer-implemented method of claim 1, wherein the plurality of predetermined metrics comprises at least one of mean absolute error, mean squared deviation, median absolute deviation, cross-correlation, normalized mean squared deviation, and / or Sprague-Gears.

8. validating a new simulation using the validation metrics, wherein the new simulation comprises a simulation of a vehicle function, a robot function, a building automation function, a power tool automation function, and / or a home appliance automation function; The computer-implemented method (100) of any one of claims 1 to 7.

9. A computer system configured to perform the computer-implemented method (100) for determining validation metrics for determining model quality according to any one of claims 1 to 8.

10. 9. A computer program comprising instructions that, when executed by a computer system, cause the computer system to perform a computer-implemented method (100) for determining validation metrics for determining model quality according to any one of claims 1 to 8.

11. A computer readable medium or signal storing and / or including a computer program according to claim 10.

Images