Method for evaluating training data set for machine learning model

Abstract

The invention relates to a method (100) for evaluating a training data set for a machine learning model, comprising the following steps:-providing (101) sensor data, a portion of which has detection features,-generating (102), by means of a further machine learning model, synthetic data on the basis of the portion of the sensor data having the detection features, -determining (103) a ratio between the share of the synthetic data of the training data set and the share of the sensor data with the detection feature,-evaluating (104) the training data set with the determined ratio based on the at least one metric. The invention further relates to a computer program, a device and a storage medium for this purpose.

Description

Technical Field

[0001] The invention relates to a method for evaluating a training data set for a machine learning model. The invention further relates to a computer program, a device and a storage medium for this purpose. Background Art

[0002] When performing automated optical inspection of electronic components, such as SMD components, it can be a challenge to detect images with production errors (NOK images) due to the high production quality. This can make reliable, robust and qualitatively high-quality training of machine learning models difficult. The unavailability of NOK images of appropriate quantity and variation may also be a challenge. Therefore, obtaining a representative data set with a sufficient number and variation of NOK images may require a huge time consumption.

[0003] With data augmentation (English: Data Augmentation), it is possible in particular to generate synthetic images, thereby reducing time consumption and creating more diverse data sets. The data set in particular enables the training of more robust or qualitatively higher quality machine learning models. Here, the data set for machine learning or KI models can be artificially expanded in a way that new data points are generated from existing data. In addition, new changes can be created based on existing data. This can be achieved by techniques such as adding noise, rotating images, or changing text in order to increase the amount and diversity of available training data. There are various examples of quantitative metrics for evaluating the generated synthetic images with respect to the ability of the generated model to remain faithful to the domain for which the model was trained. For example, in the automotive field, a generative adversarial network (GAN) preferably generates images of cars, which can be tested accordingly. In addition, an evaluation can be made with respect to the scale for which the domain space is covered. Other examples of metrics are the initial score (Inception Score), the Frechet initial distance (Frechet Inception Distance), the likelihood score, and the maximum average difference.

[0004] Furthermore, there are various quantitative metrics for evaluating the generated synthetic images, which measure the quality of the images. Most of these metrics have been developed for specific datasets, such as MNIST, Celeb, and CIFAR datasets, among others. Summary of the invention

[0005] The subject matter of the invention is a method having the features of claim 1, a computer program having the features of claim 8, a device having the features of claim 9 and a computer-readable storage medium having the features of claim 10. Further features and details of the invention are derived from the respective dependent claims, the description and the drawings. Features and details described in conjunction with the method according to the invention naturally also apply in conjunction with the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, respectively, so that the disclosure of the individual inventive aspects is always referred to or can be referred to one another.

[0006] The subject of the invention is in particular a method for evaluating a training data set for a machine learning model, comprising the following steps:

[0007] - providing sensor data, wherein a portion of the sensor data has a detection characteristic,

[0008] - Generate synthetic data based on the parts of the sensor data with detected features through other machine learning models,

[0009] - determining the ratio between the share of synthetic data of the training dataset and the share of sensor data with detected features,

[0010] - evaluating the training data set using the determined ratio based on at least one metric.

[0011] Furthermore, in another step, it may be provided that a training data set is provided based on the evaluation result. Preferably, a machine learning model is trained based on the training data set for detecting and / or classifying and / or segmenting images. The sensor data may be obtained by detection of at least one sensor, wherein the at least one sensor may be, for example, a camera, a radar sensor, an ultrasonic sensor, a lidar or a thermal imaging camera. The sensor data may include at least one single image. The portion of the sensor data having a detection feature may be at least one image having a detection feature. In the case of at least two images in the portion of the sensor data having a detection feature, preferably each image has a detection feature. It is conceivable that the detection feature is, for example, a defect such as a production error, which should be detected by the machine learning model to be trained using the training data set. The sensor data may also have sensor data without a detection feature. In order to generate synthetic data, other machine learning models may first be trained based on the portion of the sensor data having the detection feature. The ratio may, for example, be the amount of individual images of the sensor data having the detection feature relative to the amount of individual images of the synthetic data. The training data set also preferably includes sensor data without a detection feature. In simple terms, the utilization evaluation determines in particular whether a sufficiently good training data set is provided for training a machine learning model using the training data set in a certain ratio, i.e. also using a proportion of synthetic data. The evaluation result of the training data set can indicate, for example, whether at least one metric exceeds a defined limit value. The evaluation result of the training data set can indicate, for example, based on at least one metric, whether the synthetic data are sufficiently similar to the sensor data having the detection feature or are sufficiently suitable for training a machine learning model.

[0012] A further advantage can be achieved within the scope of the invention if the method further comprises the following steps:

[0013] - defining at least two data set groups, wherein the at least two data set groups are defined based on ratios between synthetic data and sensor data having the detection characteristic that are respectively different from one another,

[0014] - Evaluating at least two sets of data sets based on at least one metric.

[0015] The data set group further preferably has a share of sensor data without detection features. The ratio between the share of synthetic data of the training data set and the share of sensor data with detection features can be determined based on the evaluation results of the at least two data set groups. Different ratios between synthetic data and sensor data with detection features can thus be advantageously tested in order to determine the best ratio for the training data set. For example, it is conceivable that different ratios are optimal for different use cases. Determining the ratio based on the evaluation results of the at least two data set groups is in particular expressed as: comparing the respectively determined metrics of the at least two data set groups in order to determine the qualitatively highest quality data set group based on the comparison of the respectively determined metrics in view of at least one metric.

[0016] It is also conceivable to determine at least one metric based on a comparison between synthetic data and sensor data with the detection feature, wherein the comparison is performed with respect to pixel values ​​and / or features of the synthetic data and sensor data with the detection feature. This advantageously allows similarities between synthetic data and sensor data with the detection feature to be determined, which are reflected in the at least one metric. For this purpose, for example, a convolutional neural network can be used and / or a histogram comparison and / or feature extraction can be performed.

[0017] Furthermore, it is conceivable within the scope of the present invention that the method further comprises the following steps:

[0018] - providing a reference dataset set, wherein the reference dataset set comprises sensor data and does not comprise synthetic data,

[0019] - start training of a reference machine learning model based on a reference dataset set,

[0020] -Starting training of a corresponding group of machine learning models based on a corresponding one of the at least two dataset groups.

[0021] At least one metric or another of at least one metric can be determined based on a comparison of the predictive performance of the corresponding trained group machine learning model with the predictive performance of the reference machine learning model. The reference data set group can, for example, only include sensor data, which can simply correspond to real sensor data measured from at least one sensor. Advantageously, different ratios between synthetic data and sensor data with detection features can be evaluated with respect to practical applications by machine learning models that have been trained with corresponding ratios. The predictive performance can here be, for example, the number of false positive and / or false negative detections of the detection feature by the corresponding group machine learning model and the reference machine learning model.

[0022] Furthermore, it is conceivable that a number of n data set groups is specified, wherein the corresponding data set group d iThe (i-1)*x% share of sensor data with the detection characteristic is included, where {i|i∈N, 1≤i≤n}. This makes it possible to advantageously specify individual quantities of the data set group and the share of sensor data with the detection characteristic, which quantities can be specific to a particular use case, for example. For example, the variable x can correspond to a value of 5 or 10.

[0023] It can be advantageous if the other machine learning model within the scope of the present invention is a generative machine learning model, such as a generative adversarial network. In particular, a generative machine learning model is a type of machine learning model that aims to generate new data points, i.e., in particular synthetic data, which are similar to the same statistical properties of the training data of the generative machine learning model. Instead of only predicting or classifying the data, the generative machine learning model can not only detect the underlying probability distribution of the data, but also generate new, previously unseen data points, i.e., in particular synthetic data, therefrom. A generative adversarial network (GAN) comprises in particular two networks: a generator and a discriminator. The generator preferably attempts to imitate real data by generating synthetic data, while the discriminator distinguishes in particular between real data and synthetic data generated by the generator. Through this adversarial process, both networks can be advantageously continuously improved. In addition to generative adversarial networks, for example, variational autoencoders, diffusion models (English: diffusion model) or flow-based approaches (English: flow basedapproaches) are conceivable.

[0024] Furthermore, it is conceivable that the detection feature is a production error of a surface mount device and / or a circuit board for a surface mount device, and that the machine learning model is trained based on the provided training data set for detecting production errors in manufacturing, in particular surface mount technology manufacturing. Thus, the detection feature can be, for example, a defective or non-existent solder joint or an erroneous surface mount device. The detection feature can also be, for example, a damaged or incorrectly extended track on a circuit board or a scratch on a circuit board. The method can be advantageous in particular in the field of manufacturing, in particular surface mount technology manufacturing, because in this case, for example, production errors rarely occur. When using synthetic data, a ratio can be determined, with which the detection feature, i.e., in particular production errors, can be detected equally well or even better using a machine learning model.

[0025] The sensor data may, for example, include image data obtained from detection using a camera sensor. Here, the environment can be represented by the values ​​of image points, preferably pixels, of the image data. By performing classification based on these values ​​by a machine learning model to be trained and preferably performing image classification, objects of the environment or detection features can be detected if necessary. Classification and image classification can also be provided in the form of semantic segmentation (i.e., pixel-by-pixel or region-by-region classification) and / or object detection. The image data may, for example, be images of a radar sensor and / or an ultrasonic sensor and / or a lidar sensor and / or a thermal image camera. Accordingly, the image can also be implemented as a radar image and / or an ultrasonic image and / or a thermal image and / or a lidar image.

[0026] The subject of the present invention is also a computer program, in particular a computer program product, which comprises instructions which, when executed by a computer, cause the computer to perform the method according to the present invention. Thus, the computer program according to the present invention can bring the same advantages as have been described in detail with respect to the method according to the present invention.

[0027] The subject matter of the present invention is also a device for data processing, which is set up to carry out the method according to the invention. For example, a computer can be provided as the device, which executes the computer program according to the invention. The computer can include at least one processor, which is used to execute the computer program. A non-volatile data memory can also be provided, in which the computer program can be stored and from which the computer program can be read out by the processor for execution.

[0028] The subject of the present invention may also be a computer-readable storage medium having a computer program according to the present invention and / or comprising instructions which, when executed by a computer, cause the computer to perform the method according to the present invention. The storage medium is, for example, configured as a data storage device, such as a hard disk and / or a non-volatile memory and / or a memory card. For example, the storage medium may be integrated into a computer.

[0029] Furthermore, the method according to the invention can also be implemented as a computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Further advantages, features and details of the invention are apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the accompanying drawings. The features mentioned in the claims and in the description may be inventive in their own right or in any combination. Among them:

[0031] Figure 1 Schematic visualization showing a method, a device, a storage medium and a computer program according to embodiments of the present invention,

[0032] Figure 2 A schematic diagram showing a surface mount device and a circuit board for the surface mount device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] exist Figure 1 Schematically shown in FIG. 1 are a method 100 , an apparatus 10 , a storage medium 15 and a computer program 20 according to an embodiment of the present invention.

[0034] Figure 1 In particular, a method 100 for evaluating a training data set for a machine learning model according to an embodiment of the present invention is shown. In a first step 101, sensor data is provided, wherein a portion of the sensor data has a detection feature. In a second step 102, synthetic data is generated based on the portion of the sensor data having the detection feature by other machine learning models. In a third step 103, a ratio between the share of synthetic data and the share of sensor data having the detection feature of the training data set is determined. In a fourth step 104, the training data set is evaluated based on at least one metric using the determined ratio. In a fifth step, a training data set (not shown) can be provided based on the evaluation result.

[0035] Figure 2 A schematic diagram showing a surface mount device 1 and a circuit board 2 used for the surface mount device 1 according to an embodiment of the present invention is shown.

[0036] According to embodiments of the present invention, in particular in automated optical inspection (AOI), the quality of more than just the synthetic image can be assessed. The synthetic image can be validated in conjunction with a use case, ie, in particular within the scope of the application of a machine learning model.

[0037] The following describes an evaluation scheme for the usability of synthetic images according to an embodiment of the present invention, especially in conjunction with AOI applications. Here, the detection or prediction performance with respect to the use case can be applied to the synthetic images and given more weight than the pure image quality of the synthetic images.

[0038] Machine learning models, and in particular neural networks, according to embodiments of the present invention are preferably able to meet the required performance characteristics for very different use cases of optical inspection, such as applications such as optical surface mount technology (SMT) inspection. The machine learning model may still be general enough to maintain this performance even in manufacturing, where a variety of electrical components and error possibilities may be observed.

[0039] The evaluation scheme according to an embodiment can be divided into two steps, wherein in the first step synthetic data is prepared and in the second step the actual evaluation of the synthetic data is performed. In order to prepare the synthetic data for evaluation, the original data set can be divided into two parts: the first part preferably includes some randomly selected sensor data with detection features (in the subsequent description of the figures: NOK images), and the second part includes the rest (in the following description of the figures: OK images). The randomly selected NOK images can be used as input for training machine learning models, especially generative models, such as GAN (Generative Adversarial Network). Once the machine learning model is trained, the machine learning model can be used to generate a large amount of synthetic data (in the following description of the figures: synthetic NOK images), for example more than 100 times the original NOK images. The generated synthetic NOK images are preferably used in the next evaluation step together with the rest of the original data set.

[0040] In the evaluation step, a fixed number of original OK images and original NOK images can be maintained as a reference dataset group (holdout test set T). The remaining part can be used to create different dataset groups {d1,...,d{11}}, so that each dataset group d i The invention relates to a method for preparing a plurality of original OK images, a plurality of synthetic NOK images and a plurality of (i-1)*x percentages, especially (i-1)*10 percentages of the original NOK images. These data sets can be used to train {M1, ..., M{11}} groups of machine learning models. A reserved test set T can be used to evaluate prediction performance based on at least one metric, such as the area under the ROC curve of these models, false alarm rate and / or escape rate. The escape rate is measured in the following manner: the number of false negative results is divided by the total number of predictions. The ratio between the synthetic NOK images and the original NOK images of the training data set can now be advantageously determined, and the prediction performance of the basic model using only the original NOK images for training can also be realized or surpassed by utilizing a relatively small amount of the original NOK images and alternatively utilizing the synthetic NOK images.

[0041] The above explanations of the embodiments describe the invention only within the scope of examples. Of course, the individual features of the embodiments can be freely combined with one another as far as technically reasonable, without departing from the scope of the invention.

Claims

1. A method (100) for evaluating a training data set for a machine learning model, comprising the following steps: - providing (101) sensor data, wherein a portion of the sensor data has a detection characteristic, - generating (102) synthetic data based on the portion of the sensor data having the detected features by other machine learning models, - determining (103) a ratio between the share of synthetic data of the training data set and the share of sensor data having detection features, - evaluating (104) the training data set using the determined ratio based on at least one metric.

2. The method (100) according to claim 1, characterized in that: The method further comprises the following steps: - defining at least two data set groups, wherein the at least two data set groups are defined based on ratios between synthetic data and sensor data having the detection characteristic that are respectively different from one another, - evaluating at least two sets of data sets based on at least one metric, Therein, a ratio between a proportion of synthetic data of the training data set and a proportion of sensor data having the detection feature is determined based on the evaluation results of the at least two data set groups.

3. The method (100) according to any one of the preceding claims, characterized in that At least one metric is determined based on a comparison between the synthetic data and the sensor data having the detection feature, wherein the comparison is performed with respect to pixel values ​​and / or features of the synthetic data and the sensor data having the detection feature.

4. The method (100) according to any one of the preceding claims, characterized in that The method further comprises the following steps: - providing a reference dataset set, wherein the reference dataset set comprises sensor data and does not comprise synthetic data, - start training of a reference machine learning model based on a reference dataset set, - starting training of a corresponding set of machine learning models based on a corresponding one of the at least two sets of datasets, Wherein at least one of the metrics is determined based on a comparison of the predictive performance of the corresponding trained group machine learning model with the predictive performance of a reference machine learning model.

5. The method (100) according to any one of claims 2 to 4, characterized in that The specified number of dataset groups is n, where the corresponding dataset group d i Includes the (i-1)*x% share of sensor data having the detection feature, where {i|i∈N, 1≤i≤n}.

6. The method (100) according to any one of the preceding claims, characterized in that Other machine learning models are generative machine learning models.

7. The method (100) according to any one of the preceding claims, characterized in that The detection feature is a production error of a surface mount device (1) and / or a circuit board (2) for the surface mount device (1), and a machine learning model is trained based on a provided training data set to detect production errors in manufacturing.

8. A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to perform the method (100) according to any one of the preceding claims.

9. A device (10) for data processing, which is configured to carry out the method (100) according to any one of claims 1 to 7.

10. A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer to perform the steps of the method (100) according to any one of claims 1 to 7.

Images