Computer-implemented method and device for generating training data and for training on the basis of the generated training data
The iterative method using two distribution models within the computer-implemented system addresses the challenges of manual data labeling and class imbalance by generating high-quality synthetic training data, improving classification accuracy and reducing costs.
Patent Information
- Application Number
- PCT/EP2024/063604
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-21
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-28
AI Technical Summary
Manual labeling of training data for machine learning algorithms is time-consuming and costly, and existing methods for data augmentation do not fully address the imbalance in training data distribution across classes, particularly in scenarios with few examples of non-OK components.
A computer-implemented method involving iterative training and generation of training data using two distribution models, where one model generates synthetic data based on real data and vice versa, to improve data quality and balance across classes, utilizing machine learning algorithms like generative adversarial networks.
This approach successively improves the quality of training data, balances class distribution, and enhances classification accuracy, reducing the need for extensive manual labeling and costly data collection.
Smart Images

Figure EP2024063604_28112024_PF_FP_ABST
Abstract
Description
[0001] R. 404429IP1 - 1 - Description Title to generate and for training based on the The invention relates to a computer-implemented method and a device for generating training data and for training based on the generated training data. State of the art Image classification is well known in the art and concerns the task of extracting information from an image in order to enable the image to be assigned to a specific image category and / or image class based on this information. The resulting cluster from such an image classification can be used, for example, to create thematic categories. Although the classification of image data can in principle be performed by a human analyst, it is increasingly being automated and carried out using machine learning approaches and / or artificial intelligence methods. A machine learning algorithm used for classification and / or recognition tasks usually requires training data forwhich has already been labeled, often manually, by an expert with domain knowledge. Based on this labeled training data, the machine learning algorithm can be trained to recognize whether an input image belongs to a specific image category and / or image class or not. R. 404429IP1 - 2 - One problem is that the manual labeling of training data is time-consuming and costly. Often, a large dataset of potential training images is available, which have been re-encoded, for example, through endurance runs. However, of these potential training images, often only a small portion (e.g., between 1% and 5%) is labeled, so that the corresponding image category and / or image class is known for training the machine learning algorithm. The unlabeled portion of the dataset (e.g., 95-99%) isHowever, they cannot be directly used for training the machine learning algorithm. Instead, these training images would have to be manually labeled, marked, or labeled. This, in turn, is associated with additional costs and is time-consuming. Furthermore, the machine learning algorithm must usually be exposed to various scenarios so that it can correctly map or assign the relationship between images and classes or categories. Accordingly, factors such as class equality, i.e., an approximately equal number of labeled images from all classes, and / or context diversity, i.e., images from different class scenarios, must be taken into account. The collection of such diverse data is also associated with costs, which are generated both by the creation of continuous runs and by the storage of the resulting data. A typical problem with automatic classification is alsoImbalance in the number of training examples for each class in a (training) data set. For example, in the context of the final optical inspection of components, there is typically very little sample data for defective parts (so-called non-OK or nOK components) due to the generally high-quality manufacturing process. This makes successful training of a classifier more difficult. A common way to at least minimize this problem is to artificially generate and / or augment training data for the class(es) for which only little training data exists. In this case, the data is usually generated in a single pass and then used for classification by a classifier. Although this approach leads to a partial minimization of the imbalance, there is still potential for development in this area, particularly with regard to improving the quality of the augmented and / or synthetically generated data.generated (training) data. The invention is therefore based on the object of specifying an improved computer-implemented method and / or a device for generating training data and for training at least one model based on the generated training data. The object is achieved by a computer-implemented method for generating training data and for training on the basis of the generated training data according to the features of patent claim 1. The object is achieved by a device for generating training data and for training on the basis of the generated training data according to the features of patent claim 10. Disclosure of the Invention According to the invention, a computer-implemented method for generating training data and for training on the basis of the generated training data is claimed. The method has the following steps: Providing a first training data generation model; Providing aClass distribution model; Providing a classification model; Providing a second training data generation model; Training the first training data generation model depending on a class condition, including the class distribution model; Generating first training data depending on the class condition, including real data by the first training data generation model; Training the classification model on the basis of the first training data, such that at least one class R. 404429IP1 - 4 - and / or a class probability is assigned to the respective first training data by the classification model and thus classified second training data is generated; Training the second training data generation model based on the first training data, including real data; Generating third training data by the second training data generation model; Retraining thefirst training data generation model based on the generated second and / or third training data; iteratively performing the steps training, generating, training, training, generating until a predetermined termination criterion is met and / or until a predetermined convergence measure is reached. It should be noted that, as an alternative to the class probabilities, as described above for the feedback to the generator, so-called sampling weights for the training data can also be provided by a separate module. Sampling weights ^ are known, for example, from the publication https: / / arxiv.org / pdf / 1909.11671.pdf. The sampling weights can be determined using a separate module, in particular as described in the aforementioned publication with an additional neural network. This module determines the sampling weights for the given data set, whereby the sampling weights effectively enable or disable theDescribe training data points in order to select only good training data points for training. A first distribution model can preferably be specified by the first training data generation model and the class distribution model. A second distribution model can preferably be specified by the classification model and the second training data generation model. It is understood that the steps according to the invention, as well as further optional steps, do not necessarily have to be carried out in the order shown, but can also be carried out in a different order. Furthermore, further intermediate steps can be provided. The individual steps can also comprise one or more sub-steps without thereby departing from the scope of the method according to the invention. R. 404429IP1 - 5 - The invention further provides a device for generating training data and for training based on the generated training data.The device comprises a provision device configured to provide a first training data generation model; a class distribution model; a classification model; and a second training data generation model. The device further comprises an evaluation and computing device configured to perform the following steps: training the first training data generation model depending on a class condition, preferably including the class distribution model; generating first training data depending on the class condition, including real data, by the first training data generation model; training the classification model based on the first training data, such that at least one class and / or a class probability is assigned to the respective first training data by the classification model, and thus generating classified second training data.are; training the second training data generation model on the basis of the first training data, including real data; generating third training data by the second training data generation model; retraining the first training data generation model on the basis of the generated second and / or third training data; iteratively performing the steps of training, generating, training, training, generating until a predetermined termination criterion is met and / or until a predetermined convergence measure is reached. The first training data generation model and the second training data generation model can be the same training data generation model. In particular, it can be the same training data generation model that is trained and / or generates training data in successive iteration steps. The respective model is preferably an algorithm of themachine learning and / or an artificial intelligence algorithm. The respective model can, for example, comprise a neural network or another artificial intelligence model and / or another artificial intelligence network. The first training data generation model is preferably a first training data generation model, conditioned in particular on the basis of the class condition. The first training data generation model is preferably specified by the class condition, for which data is then to be generated. In this case, the class distribution model is preferably also used as a basis, which preferably specifies a prior probability for a label or an identification for the respective generated training data. The real data is preferably data that is available in a labeled form before training and has a high ground truth value.value). The real data is preferably given a high weighting in order to enable the generation of the first training data, in particular based on the real data. The classification model is preferably trained on the basis of the first training data, wherein further training data can preferably also be used during training. Rather, the training data of the classification model are preferably supplemented by at least the first training data (and in particular also by the real data contained therein) in order to have a larger training data set available. By training the classification model, the first training data are also labeled by assigning them at least one class or class probability. The labeled or identified data thereby generates further training data, which are used according to the invention to facilitate the generation and / or training, in particular in a nextIteration step. The second training data generation model is preferably condition-free and generates training data based on the real and artificial data of the previous generation by the first model, without, for example, specifying which training data, in particular which class-specific training data, should be generated. The generation of the third training data is preferably also based on real data, which is taken into account with a high weighting in the generation of the synthetic and / or augmented third training data. R. 404429IP1 - 7 - By means of the method according to the invention and the device, in particular due to the iteration of the data augmentation, the quality of the training data thus generated can be successively improved (per iteration step). By means of the method according to the invention or by means of the interaction and / or iterative linking of the four models, in particular by means of the transmission of theA meta-algorithm is provided for each generated training data, which can be applied to any type of data generator and / or classifier. The invention solves the problem of unbalanced data distribution or class distribution in a set of training data. In addition, the problem of a, in particular absolute, lack of training data and / or training examples for certain classes, e.g., when only a few non-OK training data are available due to high-quality manufacturing, is solved. The solution involves alternating two distribution models, each based on two sub-models, being trained to generate synthetic training data (the first, second, and third training data) for the respective other model. The method according to the invention or a classification model trained by the method according to the invention can be used, for example, in optical inspection and / or visual testing and / orDefect detection of components. Furthermore, a classification model trained according to the invention can be used in the field of autonomous and / or automated driving and / or robotics and / or medical technology and / or in a smart home environment. The present proposed solution is concerned with representing a common distribution of the (training) data X to be classified and the class labels Y in two mathematically different distribution models, and linking these two distribution models synergistically, in particular through data exchange, in such a way that model convergence can be achieved through iterative implementation of training and data generation steps. According to Bayes' rule, both distribution models preferably theoretically comprise the same (probability) distribution. However, this is not always the case in reality, since the separate representation in two differentDistribution models, both of which are trained separately from one another, a deviation can arise between the respectively achieved R. 404429IP1 - 8 - (probability) distribution per distribution model. According to the invention, this deviation can be reduced. The two (probability) distributions are iteratively adjusted to one another in order to establish consistency with Bayes' rule. According to the invention, the first distribution model or the respective training data generation model generates the augmentation data for the second distribution model. The second distribution model or the submodels included therein can be trained based on this augmentation data or training data. Subsequently, the second distribution model or the submodel included and trained therein can itself be used to generate data for retraining the first distribution model or the submodels included therein. Sufficient qualityOne of the two distribution models leads to an augmentation of training data, which enables an improvement (in classification accuracy and / or generation accuracy) of the other distribution model. In this way, a positive feedback loop is created, and both distribution models successively improve their quality in each iteration. Particularly preferably, a mixing coefficient, i.e. a proportion of real data or genuine data in the respective training data set, can be reduced according to a predetermined procedure and / or schedule at the end of each iteration step. For a better understanding of such a procedure, temperature schedules, for example, should be mentioned, as they are used, for example, in annealing simulations. This is particularly preferred if it is ensured that the above positive feedback loop has been initiated, since in this case the quality of the generated training data can be maintained even withoutInclusion of highly weighted real data is sufficiently good. Monitoring of the performance development of the respective distribution model can be carried out by means of a data validation set, on which the model performance, e.g. an accuracy and / or a precision and / or a recall and / or a quality measure for the generated and / or classified training data is / are particularly re-evaluated in each iteration. R. 404429IP1 - 9 - In a preferred embodiment, the first training data generation model and / or the class distribution model and / or the classification model and / or the second training data generation model each comprise / comprise a machine learning algorithm. The type of machine learning algorithm used here is fundamentally arbitrary. In a preferred embodiment, the first training data generation model and / or the second training data generation model comprise / comprise aGenerative Adversarial Network (GAN) and / or a deep neural network. Other data generation networks may also be preferred, with the choice being particularly related to the type of data to be generated. There are many different data generation models, depending on the type of data one wants to generate. Examples include: Generative Adversarial Networks (GANs): GANs are a type of neural network capable of generating new data by pitting two neural networks against each other. One network generates new data, while the other network attempts to determine whether the generated data is real or fake. These two networks work together to achieve increasingly better results; and / or Variational Autoencoders (VAEs); and / or Markov Modeling: Markov models are statistical models that describe the dependence of data over time.model. They can be used to generate new data by setting the state of the model to a specific point in time and then performing a sequence of state transitions to generate new data; and / or Generative Models with Autoregressive Architectures: In this type of model, the probability of each element of the data is generated sequentially, using the previously generated elements as input. The model can be used to generate new data by selecting a random sequence of initial elements and then generating the elements sequentially; and / or Monte Carlo Simulations: Monte Carlo simulations are a method for generating data by repeatedly drawing random values from specific distributions. Monte Carlo R. 404429IP1 - 10 - Simulations are often used in statistics and physics, for example, to test complex mathematical models or to analyze processes.simulate that are difficult to investigate experimentally. Bayesian networks can also be cited as examples of data generation models, which can also be used for non-time-dependent data. In a preferred embodiment, after the retraining step, improved and / or extended first training data is generated depending on the class condition, including real data, by the first training data generation model. The improved and / or extended first training data generated in this way can then in turn be used for retraining the classification model and / or the second data generation model. In a preferred embodiment, the class distribution model is trained and / or adjusted based on the first training data. Depending on how the first training data was generated in the previous iteration step, the class distribution model can be adjusted and / orbe adjusted. The class distribution model can basically be provided by manually specifying a certain probability value. In other words, the class distribution model in a preferred embodiment comprises a particularly categorical and / or predetermined class distribution. In a preferred embodiment, the predetermined termination criterion comprises a number of iteration steps. Alternatively or additionally, the predetermined convergence measure can be achieved by iterative, in particular model-wise minimization of a divergence function, in particular a Kullback-Leibler divergence. According to the invention, the minimization of the divergence function is carried out by iteratively carrying out the training and data generation steps and by the data exchange between the two distribution models. The respective distribution model can namely be optimized by an optimized training data set, which it at least partially receives from the otherDistribution model obtained, (re-)trained, so that in each further iteration step, a divergence between the two distribution models can be reduced. This process can be carried out iteratively until the two distribution models converge or at least a predetermined convergence criterion is met. The result is preferably an optimally trained classification model. In a preferred embodiment, the first training data and / or the second training data and / or the third training data comprise image data and / or video data and / or audio data and / or text data and / or database data and / or numerical data. In a preferred embodiment, the real data is in particular acquired by a sensor, for example in the form of image data from a camera. The real data can preferably be manually labeled as belonging to a specific class. The real data can alsoresult from a simulation model and / or from a database, wherein the real data is preferably verified with regard to plausibility. This can prevent incorrect training data augmentation or incorrect training data generation. The invention also claims a computing device with at least one processor, on which a classification model trained according to the method according to the invention can be executed, so that input data to be classified can be assigned to at least one class by the classification model. The computing device can be implemented, for example, in a camera control unit. The invention also claims a system for the automatic, in particular optical, inspection of a component. The system comprises at least one computing device. Furthermore, the system can comprise a sensor, in particular a camera, by means of which input data from the at least one component can be captured.are. The acquired input data can be classified by the trained classification model into at least one, preferably two classes. For example, the classification model can classify the input data in such a way that the component included therein is classified into a category of "OK" or "not OK". R. 404429IP1 - 12 - According to the invention, a computer program with program code is also claimed in order to carry out at least parts of the method according to the invention in one of its embodiments when the computer program is executed on a computer. In other words, the invention provides a computer program (product) comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method / steps of the method according to the invention in one of its embodiments. According to the invention, a computer-readable data carrier with program code of a computer program is also proposed in order to carry out at least parts of theinventive method in one of its embodiments when the computer program is executed on a computer. In other words, the invention relates to a computer-readable (storage) medium, comprising instructions which, when executed by a computer, cause the computer to carry out the method / steps of the inventive method in one of its embodiments. The described embodiments and developments can be combined with one another as desired. Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention not explicitly mentioned above or described below with regard to the embodiments. Brief description of the drawings The accompanying drawings are intended to provide a further understanding of the embodiments of the invention. They illustrate embodiments and serve in the context of the description, explanation ofPrinciples and concepts of the invention. R. 404429IP1 - 13 - Other embodiments and many of the aforementioned advantages will become apparent with reference to the drawings. The elements shown in the drawings are not necessarily true to scale. They show: Fig. 1 shows a schematic flow diagram of the method according to the invention; and Fig. 2 shows a schematic flow diagram of an embodiment of the method according to the invention. In the figures of the drawings, the same reference numerals designate the same or functionally equivalent elements, parts, or components, unless otherwise stated. Figure 1 shows a schematic flow diagram of a computer-implemented method for generating training data and for training a classifier based on the generated training data. In any embodiment, the method can be carried out at least partially by a device 1. It is understood that the deviceadditionally comprise a storage device and / or an output device and / or a display device and / or an input device. Furthermore, the device 1 can be part of a system not shown in detail. The device can, for example, be or comprise an evaluation and / or computing device of a sensor. This can, for example, be an environmental sensor from the vehicle sector or an optical sensor from the field of optical inspection of components. According to the invention, the computer-implemented method comprises at least the following steps: R. 404429IP1 - 14 - In a step S1, a first training data generation model P2(x|y) is provided. In a step S2, a class distribution model P2(y) is provided. In a step S3, a classification model P1(y|x) is provided. In a step S4, a secondTraining data generation model P1(x). In a step S5, the first training data generation model P2(x|y) is trained depending on a class condition, optionally including the class distribution model P2(y). In a step S6, first training data is generated depending on the class condition and including real data by the first training data generation model P2(x|y). In a step S7, the classification model P1(y|x) is trained based on the first training data in such a way that at least one class is assigned to the respective first training data by the classification model P1(y|x), and second training data classified in this way is generated. In a step S8, the second training data generation model P1(x) is trained based on the first training data and including real data. In a step S9, third training data is generated by thesecond training data generation model P1(x). In a step S10, the first training data generation model P2(x|y) is retrained (S10) based on the generated second and / or third training data. R. 404429IP1 - 15 - In a step S11, steps S6 to S10 are iteratively performed until a predetermined termination criterion is met and / or until a predetermined convergence measure is reached. Fig. 2 shows a schematic flow diagram of an embodiment of the method according to the invention. The joint distribution P 1 / 2(x,y) of the (training) data, for example image data x, and respective class labels y can preferably be specified according to Bayes' rule in both ways shown, i.e. in two different distribution models 200, 202, each leading to its own model representation. P1(x) describes the second training data generation model, which can be designed, for example, as a GAN for generating image data x. P1(y|x) preferably describes the classification model by which (training) data can be classified as belonging to at least one class. P2(x|y) describes the second training data generation model, which comprises a condition-dependent, in particular class-specific data generator. P2(y) describes the class distribution model, which can, for example, have a simple, categorical class distribution.According to Bayes' rule, P1(x,y) = P2(x,y), which is generally not guaranteed by the separate training of both models 200, 202 and can only be achieved at least approximately through the inventive, iterative adjustment of the two models 200, 202. In one embodiment, the inventive method can also be specified as follows: Definitions: R. 404429IP1 - 16 - ^ x: training data and / or features and / or data type of the training data; ^ y: class labels for the training data; ^ D. R (A): Real data of data type A; ^ D G,i(A): Generated data of data type A according to data generation model i. Initialization: ^ Defining the class distribution model P2(y); ^ Training the first training data generation model P2(x|y); which is condition-dependent, i.e., under the condition y, and in particular class-specific for a certain class y; ^ Obtaining or generating first training data of data type A to form a data set D2(x,y) = D R (x,y) ^ D G,2 (x,y) to obtain; Iterate: ^ Minimize a divergence function D KL (D2(x,y)|P1(x,y)), comprising: o Training the classification model P1(y|x) on the basis of the data set D2(x,y); o Training the second training data generation model P1(x) on the basis of the data set D2(x,y); ^ Generating second and / or third training data based on the trained models P1(y|x) and P1(x) to form a data set D1(x,y) = D R (x,y) ^ D G,1 (x,y); ^ Minimize a divergence function DKL (D1(x,y)|P2(x,y)), comprising: o Training the first training data generation model P2(x|y) based on the data set D1(x,y); o Training the class distribution model P2(y) based on the data set D1(x,y); ^ Generating new, extended, first training data based on the trained models P2(x|y) and P2(x) in order to update the data set D2(x,y). R. 404429IP1 - 17 - The embodiment of the method or algorithm according to the invention is preferably based on the principle of iterative, component-wise minimization of the Kullback-Leibler divergences D KL (P 1 / 2 (x,y)| D 2 / 1 (x,y)) between a distribution model P 1 / 2 (x,y) and the (training) data D 2 / 1 (x,y) of the other distribution model (depending on the context, D 2 / 1 (x,y) either the data itself or their empirical distribution). To avoid a common drift away from the real data D R(x,y), it is preferred if the real data is part of D 1 / 2 (x,y). The method according to the invention is similar to the so-called Helmholtz machine.
Claims
R. 404429IP1 - 18 - Claims 1. Computer-implemented method for generating training data and for training on the basis of the generated training data, the method comprising: - providing (S1) a first training data generation model (P2(x|y)); - providing (S2) a class distribution model (P2(y)); - providing (S3) a classification model (P1(y|x)); - providing (S4) a second training data generation model (P1(x)); - training (S5) the first training data generation model (P2(x|y)) as a function of a class condition, optionally including the class distribution model (P2(y)); - generating (S6) first training data as a function of the class condition, including real data, by the first training data generation model (P2(x|y));- Training (S7) the classification model (P1(y|x)) on the basis of the first training data, such that at least one class and / or a weighting, in particular a class probability or an importance of the first training data (sampling weights), which were determined by means of a further module, is assigned to the respective first training data by the classification model (P1(y|x)), and second training data classified in this way is generated; - Training (S8) the second training data generation model (P1(x)) on the basis of the first training data, including real data; - Generating (S9) third training data by the second training data generation model (P1(x)); R. 404429IP1 - 19 - - retraining (S10) the first training data generation model (P2(x|y)) based on the generated second and / or third training data; - iteratively performing (S11) steps (S6) to (S10) until a predetermined termination criterion is met and / or until a predetermined degree of convergence is reached.
2. The method according to claim 1, wherein the first training data generation model (P2(x|y)) and / or the class distribution model (P2(y)) and / or the classification model (P1(y|x)) and / or the second training data generation model (P1(x)) comprises / comprise a machine learning algorithm.
3. The method according to claim 1 or 2, wherein the first training data generation model and / or the second training data generation model comprise / comprise a generative adversarial network and / or a deep neural network. 4.Method according to one of the preceding claims, wherein after the retraining step (S10), improved and / or extended first training data are generated as a function of the class condition, including real data, by the first training data generation model (P2(x|y)).
5. Method according to one of the preceding claims, wherein training and / or adjusting of the class distribution model (P2(y)) takes place on the basis of the first training data.
6. Method according to one of the preceding claims, wherein the class distribution model (P2(y)) comprises a particularly categorical and / or predetermined class distribution.
7. Method according to one of the preceding claims, wherein the predetermined termination criterion comprises a number of iteration steps, and / or wherein the predetermined convergence measure is determined by a. R. 404429IP1 - 20 - iterative, in particular model-wise, minimization of a divergence function is achievable.
8. The method according to one of the preceding claims, wherein the first training data and / or the second training data and / or the third training data comprise image data and / or video data and / or audio data and / or text data and / or database data and / or numerical data.
9. The method according to one of the preceding claims, wherein the real data is in particular acquired by a sensor, and / or wherein the real data is each labeled as belonging to a specific class and / or a specific class probability.
10. A device (1) configured to carry out the method according to one of the preceding claims.
11. A computer program with program code to carry out at least parts of a method according to one of claims 1 to 9 when the computer program is executed on a computer.Computer-readable data carrier with program code of a computer program for carrying out at least parts of a method according to one of claims 1 to 9 when the computer program is executed on a computer.
