Method for evaluating a training dataset for a machine learning model

Abstract

The invention relates to a method (100) for evaluating a training data set for a machine learning model, comprising the following steps: - providing (101) sensor data, wherein a portion of the sensor data includes a detection feature, - generating (102) synthetic data based on the portion of the sensor data with the detection feature by a further machine learning model, - determining (103) a ratio between a portion of synthetic data and a portion of sensor data with the detection feature for the training data set, - evaluating (104) the training data set with the determined ratio based on at least one metric. The invention further relates to a computer program, a device, and a storage medium for this purpose.

Description

The invention relates to a method for evaluating a training dataset for a machine learning model. The invention further relates to a computer program, an apparatus and a storage medium for this purpose.Prior ArtIn automated optical inspection of electronic components, such as SMD components, it may be a challenge to capture images with production errors (NOK images) due to a high production quality. This may make reliable, robust, and high-quality training of machine learning models difficult. Also, unavailability of NOK images in an appropriate amount and variation may already present a challenge. Obtaining a representative data set with a sufficient number and variation of the NOK images may thus require a considerable amount of time.With an enrichment of the data (Data Augmentation), it is possible in particular to generate synthetic images and thereby to reduce the time expenditure and to create a more diverse data set. This makes it possible, in particular, to train a more robust or higher-quality machine learning model. In this case, data sets for machine learning or AI models can be artificially expanded by generating new data points from the existing data. Further, new variations can be created based on the existing data. This can be accomplished by techniques such as adding noise, turning images, or changing texts to increase the amount and variety of training data available. There are several examples of quantitative metrics for evaluating the generated synthetic images in terms of an ability of the generating model to measure the domain for which it was trained to remain treu. For example, a generative adversarial network (GAN) in the automotive area preferably generates images of cars, which can be checked accordingly. Further, an evaluation may be made as to an extent to which a domain space is covered. Further examples of metrics are the Acceptance Score, the Frechet Acceptance Distance, the Likelihood Score and the Maximum Mean Discrepance.Further, there are various quantitative metrics for evaluating the generated synthetic images that measure the quality of the images. Most of these metrics have been developed especially for specific data sets such as MNIST, Celeb, and CIFAR data sets.Disclosure of the InventionThe subject matter of the invention is a method with the features of claim 1, a computer program with the features of claim 8, an apparatus with the features of claim 9 and a computer-readable storage medium with the features of claim 10. Features and details which are described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, so that with regard to the disclosure reference is or can always be made to the individual aspects of the invention in a mutually alternating manner.The invention relates in particular to a method for evaluating a training dataset for a machine learning model, comprising the following steps:providing sensor data, wherein a part of the sensor data has a detection feature,generating synthetic data based on the portion of the sensor data having the detection feature by a further machine learning model,determining a ratio between a portion of synthetic data and a portion of sensor data with the detection feature for the training data set,evaluating the training data record with the determined ratio on the basis of at least one metric.Furthermore, in a further step, it can be provided that the training data set is provided on the basis of a result of the assessment. The machine learning model is preferably trained on the basis of the training dataset for a detection and / or classification and / or segmentation of images. The sensor data can result from a detection of at least one sensor, wherein the at least one sensor can be, for example, a camera, a radar sensor, an ultrasonic sensor, a LiDAR or a thermal imaging camera. The sensor data may include at least a single image. The part of the sensor data with the detection feature can be at least one image which has the detection feature. In the case of at least two images in the part of the sensor data with the detection feature, each image preferably has the detection feature. It is conceivable that the detection feature is, for example, a defect such as a production defect which is to be detected by the machine learning model to be trained with the training dataset. The sensor data may further include sensor data without the detection feature. The further machine learning model can be trained for generating the synthetic data firstly on the basis of the part of the sensor data with the detection feature. The ratio may be, for example, a set of individual images of the sensor data having the detection feature with respect to a set of individual images of the synthetic data. The training data set further preferably comprises sensor data which do not have a detection feature. Expressed in simplified terms, the assessment determines in particular whether a sufficiently good training data set is provided for training the machine learning model with the training data set using the determined ratio, i.e. also using a proportion of synthetic data. The result of the evaluation of the training dataset can indicate, for example, whether the at least one metric exceeds a defined limit value. The result of evaluating the training dataset may indicate, for example, based on the at least one metric, that the synthetic data is sufficiently similar or sufficiently suitable to the sensor data having the detection feature for training the machine learning model.A further advantage can be achieved within the scope of the invention if the method further comprises the following steps:defining at least two data record groups, wherein the defining of the at least two data record groups is carried out on the basis of a respectively mutually different ratio between the synthetic data and the sensor data with the detection feature,evaluating the at least two data record groups on the basis of the at least one metric.The data set groups further preferably have a portion of sensor data without the detection feature. The determination of the ratio between the portion of synthetic data and the portion of sensor data with the detection feature for the training data set may be performed on the basis of a result of the assessment of the at least two dataset groups. Thus, different ratios between the synthetic data and the sensor data can be tested with the detection feature in order to determine an optimum ratio for the training data set. For example, it is conceivable that different conditions are optimal for different application cases. The determination of the ratio on the basis of the result of the evaluation of the at least two dataset groups expresses in particular that the respectively determined metrics of the at least two dataset groups are compared in order to determine a high-quality dataset group with respect to the at least one metric on the basis of the comparison of the respectively determined metrics.It is furthermore conceivable that the at least one metric is determined on the basis of a comparison between the synthetic data and the sensor data with the detection feature, wherein the comparison is carried out with respect to pixel values and / or features of the synthetic data and the sensor data with the detection feature. As a result, it is advantageously possible to determine a similarity between the synthetic data and the sensor data with the detection feature, which similarity is reflected in the at least one metric. For this purpose, for example, a convolutional neural network can be used and / or a histogram comparison and / or a feature extraction can be carried out.Moreover, it is conceivable within the scope of the invention that the method further comprises the following steps:providing a reference data set group, wherein the reference data set group comprises the sensor data and no synthetic data,initiating training of a reference machine learning model on the basis of the reference dataset group,initiating training of a respective group machine learning model on the basis of a respective one of the at least two dataset groups.The at least one metric or a further metric of the at least one metric can be determined on the basis of a comparison of a prediction performance of the respective trained group machine learning models with a prediction performance of the reference machine learning model. The reference dataset group can comprise, for example, only the sensor data, which in simplified terms can correspond to real sensor data from a measurement of at least one sensor. Advantageously, the different relationships between the synthetic data and the sensor data having the detection feature with respect to an actual application can thus be evaluated by a machine learning model which has been trained with a respective relationship. The prediction performance may be, for example, a number of false positive and / or false negative detections of the detection feature by a respective group machine learning model and the reference machine learning model.Furthermore, it is conceivable that a number of n data set groups is defined, wherein a respective data set group d i comprises a proportion of (i-1)*x% of sensor data with the detection feature, wherein {i | i ∈ N, 1≤i≤n}.As a result, an individual set of data set groups and a portion of sensor data can be advantageously defined with the detection feature, taking into account a computing effort, which can be specific for a specific application, for example. For example, the variable x may correspond to a value of 5 or 10.It can be advantageous if, within the scope of the invention, the further machine learning model is a generative machine learning model, for example a generative adversarial network. A generative machine learning model is, in particular, a type of machine learning model that aims to generate new data points, i.e., in particular the synthetic data, that resemble the same statistical properties of training data of the generative machine learning model. Instead of just predicting or classifying data, a generative machine learning model can acquire an underlying probability distribution of data and generate therefrom new, previously unseen data points, i.e. in particular the synthetic data. Generative Adversarial Networks (GANs) include, in particular, two networks: a generator and a discriminator. The generator preferably attempts to mimic real data by generating synthetic data, while the discriminator distinguishes between real and generator generated synthetic data in particular. This adversary process can advantageously improve both networks continuously. In addition to the generative adventrial network, a variational autoencoder, diffusion models (English) or flow-based approaches (English) are also conceivable, for example.Furthermore, it is conceivable that the detection feature is a production defect of a surface-mounted component and / or a printed circuit board for surface-mounted components and the machine learning model is trained on the basis of the provided training data set for detecting production defects in a production, in particular a surface-mount technology production. The detection feature can therefore be, for example, a defective or non-present solder joint or a false surface-mounted component. The detection feature can also be, for example, a damaged or incorrectly running line of the printed circuit board or a scratch on the printed circuit board. The method can be advantageous particularly in the field of production, in particular surface mount technology production, since production errors rarely occur in this case, for example. Using the synthetic data, a ratio can be determined with which an equally good or even better detection of the detection features, i.e. in particular the production errors, is possible with the machine learning model.The sensor data may include, for example, image data resulting from detection with a camera sensor. In this case, the values of image points, preferably pixels, of the image data can represent an environment. By classifying and preferably classifying the image by the machine learning model to be trained on the basis of these values, objects of the environment, or the detection feature, can be detected, if necessary. The classification and image classification can also be provided in the form of semantic segmentation (i.e. a pixel or region classification) and / or object detection. The image data can be images of a radar sensor and / or an ultrasonic sensor and / or a LiDAR sensor and / or a thermal imaging camera, for example. Accordingly, the images can also be embodied as radar images and / or ultrasonic images and / or thermal images and / or lidar images.The invention likewise relates to a computer program, in particular a computer program product, comprising instructions which, when the computer program is executed by a computer, cause the computer program to execute the method according to the invention. Thus, the computer program according to the invention brings with it the same advantages as have been described in detail with reference to a method according to the invention.The invention likewise relates to a device for data processing which is set up to carry out the method according to the invention. The device can be, for example, a computer which executes the computer program according to the invention. The computer may include at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program can be stored and from which the computer program can be read out by the processor for execution.The invention can likewise be a computer-readable storage medium which has the computer program according to the invention and / or comprises instructions which, when executed by a computer, cause the computer to execute the method according to the invention. The storage medium is embodied, for example, as a data memory such as a hard disk and / or a nonvolatile memory and / or a memory card. The storage medium may be integrated into the computer, for example.In addition, the method according to the invention can also be embodied as a computer-implemented method.Further advantages, features and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can be essential to the invention individually or in any combination. The following are shown: FIG. 1 shows a schematic visualization of a method, a device, a storage medium and a computer program according to exemplary embodiments of the invention, FIG. 2 shows a schematic illustration of a surface-mounted component and a printed circuit board for surface-mounted components according to embodiments of the invention.FIG. 1 schematically illustrates a method 100, a device 10, a storage medium 15 and a computer program 20 according to exemplary embodiments of the invention.FIG. 1 shows in particular a method 100 for evaluating a training dataset for a machine learning model according to embodiments of the invention. In a first step 101, sensor data are provided, wherein a part of the sensor data has a detection feature. In a second step 102, synthetic data is generated by a further machine learning model on the basis of the part of the sensor data with the detection feature. In a third step 103, a ratio between a proportion of synthetic data and a proportion of sensor data with the detection feature for the training data set is determined. In a fourth step 104, the training data set with the determined ratio is evaluated on the basis of at least one metric. In a fifth step, the training data set can be provided on the basis of a result of the assessment (not illustrated).FIG. 2 shows a schematic illustration of a surface-mounted component 1 and a printed circuit board 2 for surface-mounted components 1 according to embodiments of the invention.According to embodiments of the invention, more than only the quality of the synthetic image can be evaluated, in particular in the automated optical inspection (AOI). The synthetic images can be validated in connection with a use case, i.e. in particular in the context of an application of a machine learning model.In the following, an evaluation scheme for a usability of synthetic images, in particular in connection with AOI applications, according to embodiments of the invention, is described. More weight can be given to a detection or prediction performance with respect to a case of application to which the synthetic images are applied than a pure image quality of the synthetic images.A machine learning model, in particular a neural network, according to embodiments of the invention is preferably capable of satisfying required performance characteristics for very different applications of optical inspection, such as an application of an optical surface mount technology (SMT) inspection. The machine learning model may nevertheless be general enough to maintain this performance even in a fabrication where a variety of electrical components and failure possibilities may be observed.The evaluation scheme according to embodiments can be divided into two steps, wherein in a first step the synthetic data are prepared and in a second step the actual evaluation of the synthetic data is carried out. In order to prepare the synthetic data for the evaluation, the original data set can be divided into two parts: the first part preferably comprises some randomly selected sensor data with a detection feature (in the following description of the figures: NOK images) and the second part comprises a remainder (in the following description of the figures: OK images). The randomly selected NOK images may be used as input for training a machine learning model, in particular a generative model, for example a GAN (generative adversarial network). Once trained, the machine learning model may be used to generate a large number, for example more than 100 times the original NOK images, of synthetic data (in the following description of the figures: synthetic NOK images). The generated synthetic NOK images are preferably used together with the rest of the original data set in a next evaluation step.In the evaluation step, a fixed number of original OK images and original NOK images may be held as a reference dataset group (holdout test set T). The remainder can be used to create different dataset groups {d1,..., d{11}}} so that each dataset group d i comprises a fixed number of original OK images, a fixed number of synthetic NOK images and a number of (i-1)*x percent, in particular (i-1)*10 percent, original NOK images. These dataset groups can be used to train {M1,..., M{11}}} group machine learning models. The holdout test set T may be used to evaluate a prediction performance based on at least one metric such as an area under a ROC curve, a false positive rate, and / or escape rates of these models. The escape rate is measured in particular by dividing a number of false negative results by a total number of predictions. Advantageously, a ratio between synthetic NOK images and original NOK images can now be determined for a training dataset, with which a prediction performance of a basic model, which uses exclusively original NOK images for the training, can also be reached or exceeded with a smaller amount of original NOK images and instead with synthetic NOK images.The foregoing explanation of the embodiments describes the present invention solely by way of examples. Of course, individual features of the embodiments can be freely combined with one another, insofar as technically expedient, without departing from the scope of the present invention.

Claims

Method (100) for evaluating a training dataset for a machine learning model, comprising the following steps: - providing (101) sensor data, wherein a part of the sensor data has a detection feature, - generating (102) synthetic data on the basis of the part of the sensor data with the detection feature by a further machine learning model, - determining (103) a ratio between a proportion of synthetic data and a proportion of sensor data with the detection feature for the training dataset, - evaluating (104) the training dataset with the determined ratio on the basis of at least one metric.Method (100) according to Claim 1, characterized in that the method further comprises the following steps: - establishing at least two dataset groups, wherein the establishing of the at least two dataset groups is carried out on the basis of a respectively mutually different ratio between the synthetic data and the sensor data with the detection feature, - evaluating the at least two dataset groups on the basis of the at least one metric, wherein the determining of the ratio between the proportion of synthetic data and the proportion of sensor data with the detection feature is carried out for the training dataset on the basis of a result of the evaluating of the at least two dataset groups.Method (100) according to one of the preceding claims, characterized in that the at least one metric is determined on the basis of a comparison between the synthetic data and the sensor data with the detection feature, wherein the comparison is carried out with respect to pixel values and / or features of the synthetic data and the sensor data with the detection feature.Method (100) according to one of the preceding claims, characterized in that the method further comprises the following steps: - providing a reference dataset group, wherein the reference dataset group comprises the sensor data and no synthetic data, - initiating training of a reference machine learning model on the basis of the reference dataset group, - initiating training of a respective group machine learning model on the basis of a respective one of the at least two dataset groups, wherein the at least one metric is determined on the basis of a comparison of a prediction performance of the respective trained group machine learning models with a prediction performance of the reference machine learning model.Method (100) according to one of Claims 2 to 4, characterized in that a number of n data record groups is specified, wherein a respective data record group d i comprises a proportion of (i - 1) * x % of sensor data with the detection feature, wherein {i | i ∈ N, 1 ≤ i ≤ n}.Method (100) according to any of the preceding claims, characterized in that the further machine learning model is a generative machine learning model.Method (100) according to one of the preceding claims, characterized in that the detection feature is a production defect of a surface-mounted component (1) and / or a printed circuit board (2) for surface-mounted components (1) and the machine learning model is trained on the basis of the provided training dataset for a detection of production defects in a production process.A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer program to execute the method (100) according to any one of the preceding claims.Data processing device (10) configured to carry out the method (100) according to any one of claims 1 to 7.A computer readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer to carry out the steps of the method (100) of any one of claims 1 to 7.

Images