Method for checking authenticity of synthetic training data of machine learning model

Abstract

The invention relates to a method for checking the trueness of synthetic training data of a machine learning model, comprising the steps of: providing synthetic training data, the synthetic training data being described by statistical variables, the synthetic training data imitating sensor data, and in the range of training the machine learning model, determining the trueness of the synthetic training data; the upper limit of the confidence interval of the statistical variable is determined based on the synthetic training data, real data is provided, the real data is also described by the statistical variable, the real data comprises sensor data, the sensor data is generated by detection of at least one sensor, and the upper limit of the confidence interval of the statistical variable is determined in the range of reasoning of the machine learning model. A lower limit of a confidence interval of the statistical variable is determined on the basis of the real data, the lower limit is continuously determined from the beginning of the reasoning, the authenticity of the composite training data is checked on the basis of a comparison of the continuously determined lower limit with the determined upper limit, and a system deviation of the composite training data relative to the real data is detected. The invention also relates to a computer program, a device and a storage medium.

Description

Field of the Invention

[0001] The present invention relates to a method for checking the authenticity of synthetic training data for a machine learning model. The present invention also relates to a computer program, a device, and a storage medium for this purpose. Background Art

[0002] In the field of machine learning, synthetically generated data is very important. For example, if the scope of a data set is too small to train a classifier, synthetically generated data can be used to augment the data set. In this context, it is very important to evaluate how authentic these synthetic data are compared to the available corresponding real data.

[0003] In the case of image data, a series of metrics or measures from the literature can be used to answer this question, such as the so-called FID score. The FID score is especially related to the subjective human evaluation of authenticity, but has a serious drawback, namely that on the order of 10,000 data points or images are required to calculate the FID score. Summary of the Invention

[0004] The subject matter of the present invention is a method having the features of claim 1, a computer program having the features of claim 8, a device having the features of claim 9, and a computer-readable storage medium having the features of claim 10. Further features and details of the present invention are derived from the corresponding dependent claims, the description, and the drawings. Here, the features and details described in connection with the method according to the present invention naturally also apply to the computer program according to the present invention, the device according to the present invention, and the computer-readable storage medium according to the present invention, and vice versa, so that the disclosure of the various inventive aspects always refers to each other or can refer to each other.

[0005] The subject matter of the present invention particularly relates to a method for checking the authenticity of synthetic training data for a machine learning model, including the following steps, where these steps can be executed repeatedly and / or successively. Authenticity particularly means to what extent the synthetic training data corresponds to real reference images, or simply put, to what extent the synthetic training data appears real to a user or a machine learning model. From this, it can be determined whether the synthetic training data is suitable for training a machine learning model, because untrue training data may cause the trained machine learning model to be distorted.

[0006] In a first step, preferably, synthetic training data is provided, where the synthetic training data is described by statistical variables. The synthetic training data particularly mimics sensor data. The statistical variables can be, for example, the mean or variance. Thus, in a simplified example, it can be thought that the statistical variables reflect the average value of the amplitude of the sensor data. The synthetic training data can be provided, for example, via a simulation or digital storage medium or via a database, and corresponding interfaces can be set up for this purpose.

[0007] In a further step, preferably within the scope of training a machine learning model, an upper limit of a confidence interval of a statistical variable is determined based on synthetic training data. The machine learning model can be a supervised learning model, such as a neural network, a support vector machine or a decision tree, depending on the type of task (classification, regression, etc.). Preferably, the machine learning model is trained with synthetic training data. For example, during training, the machine learning model learns to recognize patterns and relationships in the synthetic training data in order to make predictions or estimates for new unknown data. After training, preferably a statistical variable is calculated. The statistical variable can be, for example, a measure of model performance, such as accuracy, F1 score, mean squared error or outlier score, depending on the type of task. Within the scope of determining the confidence interval, in particular the upper limit of the confidence interval of the previously calculated statistical variable is determined. This requires, for example, statistical methods to quantify the uncertainty of the variable estimate. For this purpose, the Monte Carlo method or the Bootstrap method can be used, in which the machine learning model is repeatedly trained with different samples of the synthetic data to obtain the distribution of the statistical variable. Based on the distribution of the statistical variable from the previous step, the upper limit of the confidence interval can be calculated. This value in particular indicates below which limit the true value of the statistical variable will be with a specific probability (e.g., 95%).

[0008] In a further step, preferably real data is provided, where the real data is also described by a statistical variable. The real data includes or in particular sensor data, where the sensor data is generated by the detection of at least one sensor. The sensor data can be, for example, image data, radar data, LiDAR data or ultrasonic data.

[0009] In a further step, preferably within the scope of the inference of the machine learning model, a lower limit of the confidence interval of the statistical variable is determined based on the real data, where the lower limit is continuously determined from the start of the inference. The inference in particular refers to the process of applying the trained machine learning model to new data (i.e., in particular real data) to provide a prediction. This continuous determination can be carried out cyclically, for example, every second or every 100 milliseconds. Preferably, the lower limit is determined in the same way as the previously determined upper limit, i.e., for example, by the Monte Carlo method or the Bootstrap method.

[0010] In a further step, in particular, the authenticity of the synthetic training data is checked based on the comparison of the continuously determined lower limit with the determined upper limit, where a systematic deviation of the synthetic training data with respect to the real data is detected. This comparison is carried out, for example, by forming the difference between the value of the continuously determined lower limit and the value of the determined upper limit. For example, when the continuously determined lower limit exceeds the determined upper limit, a detected systematic deviation exists. This may indicate a low authenticity of the synthetic training data and thus may lead to a distortion in the training of the machine learning model.

[0011] A confidence interval is a statistical concept in particular, which can be used to describe the uncertainty in estimating a specific parameter from a sample (i.e., within the scope of the present invention, for example, a statistical variable). Within the scope of the present invention, the confidence interval is preferably a range of values that is assumed to include the true value of an unknown population parameter (such as the population mean) with a specific probability (i.e., the confidence level). For example, the confidence level expressed as a percentage (such as 95%) particularly indicates the probability that the confidence interval includes the true parameter value. For example, a 95% confidence interval means that if sampling is repeated and the interval is calculated, the true parameter value will be within the interval in 95% of the cases. The calculation of the confidence interval is particularly based on the sample distribution of the estimated parameter or statistical variable. For example, a normal distribution (for larger samples) or a t-distribution (for smaller samples) can be used for the mean. The upper and lower limits of the confidence interval particularly limit the range of values within which the true value of the parameter under study is assumed to exist with a specific probability. These limits can be calculated based on the sample data and the selected confidence level.

[0012] For a typical confidence interval of the mean, the following limitations can be described, for example. The lower limit is particularly the lowest value of the confidence interval. This lowest value can be calculated by multiplying the critical value (depending on the selected distribution and confidence level) by the sample standard error and subtracting the sample mean. The upper limit is particularly the highest value of the confidence interval. This highest value can be calculated by multiplying the same critical value as the lower limit by the sample standard error and adding it to the sample mean. The specific values of the upper and lower limits particularly depend on the sample data, the selected confidence level, and the applied statistical method.

[0013] Preferably, it can be stipulated that an order is set in the real data so that the lower limit of the confidence interval is determined using the real data with the set order. Thereby, the time dependence in the real data can be advantageously considered, and the time dependence may also be relevant when evaluating the authenticity of the synthetic training data. The method is particularly sensitive to the order of the real data. By optimizing the order of the real data, the sensitivity of the method to the offsets actually present in the distribution can be advantageously increased.

[0014] It can be conceived that the method further includes the following steps:

[0015] - Reasoning based on partial synthetic training data, where the lower limit is continuously determined,

[0016] - Reasoning based on real data, where the lower limit is continuously determined,

[0017] - Checking the synthetic training data based on an analysis of the increase in the continuously determined lower limit when transitioning from the partial synthetic training data to the real data.

[0018] Using this method, it is advantageously possible to additionally infer a systematic deviation and therefrom infer a low truthfulness in the synthetic training data. Thus, a systematic deviation may exist when the continuously determined lower limit increases significantly when transitioning from the partial synthetic training data to the real data.

[0019] It is also possible that, in the range of checking the synthetic training data, if the continuously determined lower limit exceeds the determined upper limit, there is a systematic deviation of the synthetic data with respect to the real data.

[0020] It can be further advantageously provided that the method further comprises the following steps:

[0021] - Perform measures according to the result of checking the synthetic training data, wherein the measures at least include initiating the output of a warning message.

[0022] For example, the result may indicate that the synthetic training data has a systematic deviation and may thus, for example, lead to a distortion in the training of a machine learning model. The warning message may include the existence of this systematic deviation. The output can be carried out via a display screen and / or a speaker, thus graphically and / or aurally.

[0023] Within the scope of the present invention, it can be preferably provided that the method further comprises the following steps:

[0024] - Define a threshold for the false alarm rate.

[0025] In particular, if a systematic deviation of the synthetic data with respect to the real data is detected, but there is actually no such deviation, there is a false alarm. If a universe is defined as the realization of an infinite number of values of a statistical variable, which values exist as a time series without systematic deviation during the inference phase, and an infinite number of such universes are sampled, the false alarm rate in particular indicates the proportion of universes in which a systematic deviation is erroneously detected. Furthermore, the false alarm rate can be set as a hyperparameter: However, the lower the false alarm rate, for example, the lower the true alarm rate (i.e., the ratio of detecting an actual systematic deviation). Optimizing the order of the real data can increase the true alarm rate again.

[0026] Optionally, it can also be envisaged that the sensor data includes measurement data of a production process, in particular image data, and the statistical variable represents a defect of a corresponding component. In principle, there can be many possibilities to define the statistical variable. For example, the output of an outlier detector can be used. This means that the systematic deviation to be detected in the distribution can be the deviation from the average of the outlier scores of the outlier detector. In other words, the distribution or the deviation of the distribution can be characterized according to its ability or frequency to generate outliers. It should also be noted in this case that the method particularly takes into account the time-varying process when evaluating possible deviations. It can also be envisaged to use the method for optical inspection.

[0027] However, this only represents one such possibility, and other definitions are possible in principle. In addition, squares can be calculated for each such statistical variable, such that the method is sensitive not only to deviations from the mean, but also to deviations from the variance.

[0028] The proposed method can be used in many technical applications where the generation of synthetic data is important. Thus, sensor data can be, for example, measurement data collected for each part of production and can be divided into OK (properly functioning parts) and NOK (defective parts) from production. Typically, due to high-quality production, there are too few reference examples, especially in NOK data (e.g., for training an OK / NOK classifier), so synthetically generated training data must be used, for example.

[0029] Another example can be an autonomous vehicle or at least a partially automated vehicle, where images that are partially available for many possible traffic scenarios are also scarce. The vehicle can be configured, for example, as a motor vehicle and / or a passenger vehicle and / or an autonomous vehicle. The vehicle can have vehicle devices (e.g., for providing autonomous driving functions) and / or driver assistance systems. The vehicle devices can be designed to control and / or accelerate and / or brake and / or steer the vehicle at least partially automatically. Synthetically generated images can also be used here for assistance and the method described within the scope of the present invention can be applied.

[0030] The subject matter of the present invention also relates to a computer program, in particular a computer program product, comprising instructions which, when the computer program is executed by a computer, cause the computer to perform the method according to the present invention. Thus, the computer program according to the present invention brings the same advantages as those described in detail with reference to the method according to the present invention.

[0031] The subject matter of the present invention also relates to a device for data processing, which is arranged to perform the method according to the present invention. As such a device, for example, a computer arranged to execute the computer program according to the present invention can be provided. The computer can have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program is stored and can be read by the processor from the non-volatile data memory for execution.

[0032] The subject matter of the present invention can also relate to a computer-readable storage medium, which has the computer program according to the present invention and / or comprises instructions which, when executed by a computer, cause the computer to perform the method according to the present invention. The storage medium is, for example, configured as a data memory, such as a hard disk drive and / or a non-volatile memory and / or a memory card. The storage medium can be integrated into the computer, for example.

[0033] Furthermore, the method according to the present invention can also be designed as a computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Further advantages, features, and details of the present invention result from the following description, in which embodiments of the present invention are described in detail with reference to the accompanying drawings. Here, the features mentioned in the claims and the description can each individually or in any combination be essential parts of the present invention.

[0035] Figure 1 A schematic visualization of a method, a device, a storage medium, and a computer program according to an embodiment of the present invention is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] Figure 1 A method 100, a device 10, a storage medium 15, and a computer program 20 according to an embodiment of the present invention are schematically shown.

[0037] Figure 1 In particular, an embodiment of a method 100 for checking the authenticity of synthetic training data for a machine learning model is shown. In a first step 101, synthetic training data is provided, where the synthetic training data is described by statistical variables, and where the synthetic training data mimics sensor data. In a second step 102, within the scope of training the machine learning model, an upper limit of a confidence interval of the statistical variables is determined based on the synthetic training data. In a third step 103, real data is provided, where the real data is also described by statistical variables, and where the real data includes sensor data, and where the sensor data is generated by the detection of at least one sensor. In a fourth step 104, within the scope of the inference of the machine learning model, a lower limit of the confidence interval of the statistical variables is determined based on the real data, and this lower limit is determined continuously from the start of the inference. In a fifth step 105, the authenticity of the synthetic training data is checked based on a comparison of the continuously determined lower limit with the determined upper limit, where a systematic deviation of the synthetic training data with respect to the real data is detected.

[0038] In the method according to the embodiment, advantageously less real data is required than in the methods of the prior art, and furthermore, it is advantageously possible to control the false alarm rate. In particular, if a systematic deviation of the synthetic data with respect to the real data is detected, but there is actually no such deviation, there is a false alarm. The real alarm rate can be increased again by optimizing the order.

[0039] In the context of the present invention, in particular, a method from the literature (https: / / arxiv.org / abs / 2110.06177, hereinafter referred to as the Podkopaev method) is used, the content of which is hereby incorporated by reference. In the sense of this method, synthetic training data corresponds in particular to "reference data", and real data corresponds to "test data". According to an embodiment, this method is in particular used to detect temporal changes in the distribution compared to the reference data distribution. Furthermore, using this method, an upper limit of the false alarm rate that can be freely set can be established.

[0040] Within the scope of the present invention, the Podkopaev method is in particular used such that synthetic training data serves as the reference distribution, and real data including sensor data serves as the test distribution, where the test distribution preferably connects seamlessly to the reference distribution in time. It is also conceivable to swap the roles of the two data sets and obtain a similar method. In the Podkopaev method, in particular, a lower bound of the so-called "running risk" is continuously calculated based on the synthetic training data during the inference phase, while an upper bound of the "running risk" has been calculated during the training phase. Now if this lower bound exceeds the (constant) upper bound of the training phase during the test phase, an alarm is preferably triggered and the method detects a change in the distribution, i.e., in particular, a systematic deviation is detected.

[0041] The lower bound can be calculated for different hyperparameter settings (e.g., different loss functions). The synthetic training data as well as the real data can represent inspection images from a production line. Here, the training data and the real data can be presented in the order in which they actually occur or may occur.

[0042] According to an alternative embodiment, in addition to using the Podkopaev method, classical hypothesis tests such as the t-test or the Wilcoxon signed-rank test can also be used to perform the above method. However, the above method based on the Podkopaev method in particular provides different limitations, allows any loss function to be used as the basis for calculating the running risk, and can also take into account the order in the real data, and the real data is introduced into the method in this order. This method can open up additional degrees of freedom because now the order can be optimized to obtain as strict a test as possible, i.e., the test is based on a stricter criterion of equality of the distributions of the two data sets (= null hypothesis) and can therefore exhibit inequality, i.e., a systematic deviation, even when the deviation is small. Another advantage of the Podkopaev method is in particular that it makes fewer strong assumptions (e.g., about normal distribution) compared to the classical case where such strong assumptions exist.

[0043] The above description of the embodiments only describes the present invention within the scope of examples. Of course, if technically reasonable, the various features of the embodiments can be freely combined with each other without departing from the scope of the present invention.

Claims

1. A method (100) for checking the authenticity of synthetic training data for a machine learning model, comprising the following steps: - Providing (101) the synthetic training data, wherein the synthetic training data is described by statistical variables, and wherein the synthetic training data mimics sensor data, - Determining (102) an upper limit of a confidence interval of the statistical variables based on the synthetic training data within the scope of training the machine learning model, - Providing (103) real data, wherein the real data is also described by the statistical variables, and wherein the real data includes sensor data, and wherein the sensor data is generated by detection of at least one sensor, - Determining (104) a lower limit of the confidence interval of the statistical variables based on the real data within the scope of inference of the machine learning model, wherein the lower limit is continuously determined starting from the beginning of the inference, - Checking (105) the authenticity of the synthetic training data based on a comparison of the continuously determined lower limit with the determined upper limit, wherein a systematic deviation of the synthetic training data relative to the real data is detected.

2. The method (100) according to claim 1, characterized in that, An order is set in the real data so that the lower limit of the confidence interval is determined using the real data with the set order.

3. The method (100) according to any one of the preceding claims, characterized in that, The method further comprises the following steps: - Performing the inference based on partial synthetic training data, wherein the lower limit is continuously determined, - Performing the inference based on the real data, wherein the continuous determination of the lower limit continues, - Checking the synthetic training data based on an analysis of an increase in the continuously determined lower limit when transitioning from the partial synthetic training data to the real data.

4. The method (100) according to any one of the preceding claims, characterized in that, In the scope of checking the synthetic training data, if the continuously determined lower limit exceeds the determined upper limit, there is a systematic deviation of the synthetic data relative to the real data.

5. The method (100) according to any one of the preceding claims, characterized in that, The method further comprises the following steps: - Executing measures according to the result of checking the synthetic training data, wherein the measures at least include initiating the output of a warning message.

6. The method (100) according to any one of the preceding claims, characterized in that, The method further comprises the following steps: - Defining a threshold for the false alarm rate.

7. The method (100) according to any one of the preceding claims, characterized in that, The sensor data includes measurement data of a production process, in particular image data, and the statistical variables represent defects of corresponding components.

8. A computer program (20), comprising instructions that, when the computer program (20) is executed by a computer (10), cause the computer (10) to execute the method (100) according to any one of the preceding claims.

9. An apparatus (10) for data processing, the apparatus being configured to perform the method (100) according to any one of claims 1 to 7.

10. A computer-readable storage medium (15) comprising instructions that, when executed by a computer (10), cause the computer to perform the steps of the method (100) according to any one of claims 1 to 7.

Images