Computer-implemented method for training a machine learning algorithm

Abstract

The invention relates to a computer-implemented method for training a machine learning algorithm, wherein the machine learning algorithm is trained to classify an input image (24, 26), the method comprising the following steps: - Providing at least one image (S10, 24, 26); - providing a segment mask (S12) for each image, wherein the segment mask divides the image (24, 26) into a plurality of segments and wherein each segment is assigned one of the at least two classes; - Selecting a first class (S14); - Perform self-supervised learning (S16) with the segments of the first class; - Selecting a second class (S18); - Performing self-supervised learning (S20) with the second class segments; - performing supervised learning (S22) with all segments of the image; and - Providing the trained machine learning algorithm (S24).

Description

[0001] The invention relates to a computer-implemented method for training a machine learning algorithm, which can be used in particular for automated optical inspection in production lines. State of the art

[0002] Many production lines utilize automated optical inspection (AOI) to detect defects in the products they produce. AOI systems can help detect production defects early and thus improve the quality of manufactured products. Automated inspection can detect even the smallest defects, irregularities, or deviations from quality standards that might be overlooked by human inspectors.

[0003] Furthermore, AOI systems can operate very quickly and are capable of performing inspections in real time. This enables more efficient production, as defective products can be immediately identified and sorted out without stopping the entire production process. Early detection of defects can therefore reduce rework costs. AOI systems thus contribute to lowering overall production costs.

[0004] AOI systems provide a consistent and standardized inspection method. Unlike human inspectors, they are not susceptible to fatigue or fluctuations in attention, resulting in consistently high inspection accuracy.

[0005] Various machine learning algorithms are known from the state of the art that can be used for AOI. Regardless of the model used, these machine learning algorithms must be trained. There are two common approaches for this.

[0006] The first group consists of self-supervised learning algorithms such as SimCLR, DCL, DINO, DINO v2, etc. Self-supervised learning allows models to learn without the need for labeled data. In all cases, a special loss function for self-supervised learning is used. However, the idea is the same in all cases: only unlabeled data is required.

[0007] A disadvantage is that self-supervised algorithms often lack the required precision. In some cases, self-supervised learning can even lead to results that are detrimental to the AOI.

[0008] The second group consists of supervised learning algorithms. Supervised learning algorithms are a category of machine learning methods in which a model is trained using training data that has been labeled with appropriate target variables. This means that the model is optimized during training toward a predefined output. The learning process involves feeding sample data pairs, each comprising input data and its associated annotations, into the algorithm, which then learns the relationship between the input data and the annotations. For classification tasks, for example, ResNet and ViTs are often used. These are typically used with an objective function such as cross-entropy or regression loss, depending on the task.

[0009] Machine learning algorithms trained with supervised learning can achieve high precision. However, the effort required to annotate the input data is significant and therefore expensive. In some cases, there is not enough annotated input data available to achieve this level of precision.

[0010] The invention is therefore based on the object of proposing an improved method with which a machine learning algorithm can be trained that achieves a sufficiently high precision in classification to be used for an AOI.

[0011] The problem is solved by the subject matter of the independent claims. Disclosure of the invention

[0012] According to a first aspect of the invention, this object is achieved by a computer-implemented method for training a machine learning algorithm. The machine learning algorithm is trained to classify an input image, wherein the classification comprises assigning the image to one of at least two classes.

[0013] The procedure includes the following steps: - Providing at least one image; - Providing a segment mask for each image, wherein the segment mask divides the image into a plurality of segments and wherein each segment is assigned one of the at least two classes; - Select a first class and hide all segments that are not assigned to the first class; - Performing self-supervised learning on the first-class segments to train the machine learning algorithm to detect first-class segments; - Select a second class and hide all segments not assigned to the second class; - Performing self-supervised learning on the second-class segments to train the machine learning algorithm to detect second-class segments; - Performing supervised learning on all segments of the image to train the machine learning algorithm to assign the image to one of the classes; and - Deploy the trained machine learning algorithm.

[0014] A machine learning algorithm is an algorithm designed to automatically detect patterns and relationships in data and make predictions or decisions. It is created by training on existing data and can then be applied to new, unknown data to generate predictions or classifications.

[0015] A machine learning algorithm can take various forms, such as linear models, decision trees, support vector machines, neural networks, and many others. It is optimized by learning from the training data, identifying patterns and rules to make the best possible predictions or classifications for new data.

[0016] The effectiveness of a machine learning algorithm depends on several factors, including the quality and quantity of training data, the choice of model, the model configuration, and the evaluation of the model using evaluation metrics. The model can be continuously improved and optimized to maximize accuracy and performance.

[0017] Support vector machines (SVMs) are a model used for classification or regression that detects patterns in data and seeks the optimal separation between different classes.

[0018] Decision trees are a model that creates decision rules in the form of a tree diagram. They divide data based on features, thus enabling classification.

[0019] A probabilistic model is Naive Bayes, based on Bayes' theorem, used for classification. It assumes that features are independent of each other and calculates the probability of a particular class based on the given features.

[0020] Neural networks refer to models that are primarily used for processing a wide variety of data. The architecture of a neural network comprises several nodes, neurons, or nodes, arranged in layers.

[0021] The provided image(s) can be provided in a common format suitable for input to the selected model. For example, an image can be provided via a database or from a camera that captured the image. Furthermore, the image can be loaded from a cache for use in training the machine learning algorithm.

[0022] In principle, multiple images can be used to train the machine learning algorithm. For simplicity, the following describes the use of one or two images. The individual steps can also be performed with multiple images.

[0023] Each image used for training has a segment mask. The segment mask divides the image into several segments. The segment mask also contains information about each segment, indicating which class it belongs to. The number of classes for which the machine learning algorithm is trained is therefore determined by the number of classes specified in the segment masks.

[0024] Preferably, two classes are used. In embodiments, more classes may be used if the actual task of the machine learning algorithm requires it. For example, the classes may indicate different quality levels. One class may designate a quality level that marks the manufactured products as "OK." Another class may mark the products as "not OK" or "NOK."

[0025] In addition, embodiments may provide for intermediate classes that indicate, for example, the need for manual review, require post-processing, and / or represent a degree of uncertainty in the classification.

[0026] In self-supervised learning, the segments of an image that come from the same image are evaluated. Typically, in self-supervised learning, it is not specified which segment belongs to a particular class. The algorithm is supposed to adapt to the fact that all segments of an input image come from the same image and recognize this.

[0027] According to the present invention, only segments of a specific class are used for self-supervised learning. This allows the machine learning algorithm to adjust to detect two segments of the same class. If multiple images are used for training, the segments from the multiple images with the same class are used for self-supervised learning in this step.

[0028] This learning process is performed for each class. If the machine learning algorithm is used for two-class classification, it is trained using segments from these two classes. If the images are to be assigned to multiple classes, self-supervised learning is performed more frequently, namely for each class.

[0029] After self-supervised learning has been performed for each class, the machine learning algorithm must be trained to assign one of the classes to the input images. Ideally, each image shows an object. While the segments of the image can be assigned to different classes, the ultimate goal is to determine the class for the object depicted in the image. This means that the object itself must be assigned to one of the classes.

[0030] To this end, the machine learning algorithm is trained using supervised learning to determine the class for the image as a whole, and thus for the object depicted in it. The goal is to train the model so that it is able to make accurate predictions or classifications for new, unseen data. The model learns the underlying structure and patterns from the training data and then applies them to unknown data. To achieve this, the model is optimized during training to minimize the deviation between its predictions and the actual labels. This is done by adjusting the model parameters.

[0031] Finally, the trained machine learning algorithm is deployed so that it can be used in automated optical inspection. The images captured of the objects to be inspected are input into the machine learning algorithm and classified. The classification can then be applied to the object associated with the respective image.

[0032] The proposed method combines the advantages of supervised learning and self-supervised learning for machine learning algorithms. This largely eliminates the disadvantages inherent in each method. As a result, an improved method for training a machine learning algorithm is presented that achieves the objective of the invention.

[0033] In one embodiment, the machine learning algorithm comprises a convolutional neural network.

[0034] A neural network is a machine learning model inspired by the human brain that can be used to make predictions or perform classifications. The model comprises multiple nodes arranged in layers. Each node performs an operation on the data input to it and passes the result to a node in the next layer. The final layer is an output layer, which provides an output for the neural network.

[0035] A convolutional neural network is a special type of neural network that comprises at least one convolutional layer followed by a pooling layer. The input to a convolutional layer is a 3D matrix or 3D tensor, such as an image. The neurons in the convolutional layer are arranged accordingly.

[0036] The convolutional layer uses so-called filters or kernels, which represent small weight matrices. These filters are applied to the input image, and a convolution (an element-wise multiplication and summation) is performed for each local region. This process generates an activation map that represents the spatial presence of certain learned features in the input image.

[0037] In the pooling layer, the activation map is reduced by applying an aggregation function, such as max pooling or average pooling. Max pooling extracts the maximum value from a range, while average pooling calculates the average. Pooling gives the convolutional neural network a certain invariance to small translations in the input data. This means the network can respond to similar patterns in different positions.

[0038] The combination of convolutional and pooling layers enables convolutional neural networks to learn hierarchical features in input data. This is particularly effective in image processing, as convolutional neural networks are able to capture local patterns at lower levels and combine them into more complex and abstract features at higher levels. As an alternative to a convolutional neural network, a transformer network architecture can be used.

[0039] In one embodiment, the self-supervised learning is contrastive learning and the segments are transformed prior to the steps of performing the self-supervised learning.

[0040] Contrastive learning is a self-supervised learning method in which a machine learning algorithm is trained to detect similarities and differences between different data points. The focus is on optimizing the model to place similar examples close together in a feature space while separating dissimilar examples. This method has proven particularly effective for classifying images and image segments.

[0041] In contrastive learning, positive pairs and negative pairs of data points are created. A positive pair consists of two similar segments, while a negative pair consists of two dissimilar segments.

[0042] The main component of contrastive learning is the contrastive loss function. This function measures how well the machine learning algorithm maximizes the similarity between positive pairs and minimizes the differences between negative pairs. During the training process, the machine learning algorithm is optimized to minimize the distance between positive pairs while maximizing the distance between negative pairs. The goal is to create a feature space in which similar data points are close together, while dissimilar data points are far apart.

[0043] Contrastive learning has been shown to be very effective in generating representative features, which are particularly suitable for tasks such as image classification. This allows for the learning of unknown features, eliminating the need for explicit annotations in the data.

[0044] As an alternative to contrastive learning, other methods of self-supervised learning can be used.

[0045] In one embodiment, the transformation includes rotating, shearing, cutting, color-adjusting, color-matching, and / or otherwise transforming the segments.

[0046] A positive pair can be created by applying transformations such as rotations, scaling, mirroring, or brightness adjustments. For example, an image could be rotated or flipped horizontally, and both versions would be considered a positive pair.

[0047] Transformations are therefore a simple means of creating positive pairs.

[0048] In one embodiment, the image is an image of an object to be inspected, in particular the surface of the object to be inspected, and one of the classes identifies the object as defect-free and another class identifies the object as defective.

[0049] In automated optical inspection, the goal is often to assign the inspected objects to one of several classes. The simplest classification is performed with two classes: "OK" and "NOK."

[0050] What is designated as OK and what is designated as NOK must be specified to the machine learning algorithm and is trained in supervised learning or defined by the initial classification of the segments into classes.

[0051] The two-class implementation is a particularly simple form and therefore requires a low complexity of the machine learning algorithm. More classes, on the other hand, would require greater learning effort and a larger dataset for training.

[0052] In one embodiment, supervised learning includes computing a classification loss function.

[0053] In supervised learning methods, a classification loss function is used to quantify the difference between the predicted classes and the actual classes. The classification loss function helps the model improve its predictions by minimizing the deviation between the predictions and the actual labels during training.

[0054] In one embodiment, the classification loss function is a cross-entropy loss function.

[0055] The cross-entropy loss function is a particularly suitable loss function for classification problems, as it determines probabilities that indicate whether an object should be assigned to one class or another.

[0056] In a further aspect, the invention relates to a computer-implemented method for classifying images, wherein the method is carried out using a machine learning algorithm, wherein the machine learning algorithm is trained with a method as described above

[0057] In a further aspect, the invention relates to a computer program with program code for carrying out a method as described above when the computer program is executed on a computer.

[0058] In a further aspect, the invention relates to a computer-readable data carrier with program code of a computer program for carrying out a method as described above when the computer program is executed on a computer.

[0059] In a further aspect, the invention relates to a system for training a machine learning algorithm, wherein the system is designed to carry out a method as described above.

[0060] In summary, the present invention provides a computer-implemented method for training a machine learning algorithm, a computer-implemented method for classifying images, a computer program with program code, a computer-readable data carrier with program code, and a system for training a machine learning algorithm.

[0061] The described designs and further training courses can be combined as desired.

[0062] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or below with regard to the embodiments that are not explicitly mentioned. Short description of the drawings

[0063] The accompanying drawings are intended to provide a further understanding of embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain principles and concepts of the invention.

[0064] Other embodiments and many of the aforementioned advantages will become apparent upon review of the drawings. The elements illustrated in the drawings are not necessarily drawn to scale.

[0065] They show: Fig. 1a and Fig. 1b schematically shows the principle and application of supervised learning; Fig. 2 schematically shows the principle of self-supervised learning; and Fig. 3 schematically shows the flow of the method for training a machine learning algorithm according to one embodiment.

[0066] In the figures of the drawings, the same reference symbols designate the same or functionally equivalent elements, parts or components, unless otherwise stated.

[0067] Fig. 1a and Fig. Figure 1b schematically illustrates the principle of supervised learning. In supervised learning, a machine learning algorithm, here the neural network 10, receives an input 12 comprising an object 14 and an annotation 16. The annotation 16 is linked to the object 14, for example, a description or a property of the object 14.

[0068] With each training run, the neural network 10 adapts more and more to assigning the same annotation 16 to similar objects.

[0069] When training is complete, the neural network 10 has found a way to assign the correct annotation 16 to the input objects 14 using its outputs 18. If unknown inputs 12' with an object 14' are now input into the neural network, it generates as output 18' with sufficient probability an annotation 16' for the object 14' that matches the annotation 14 if the object 16' has a certain similarity to the object 16. This is shown in Fig. 1b.

[0070] The disadvantage of supervised learning, however, is that a sufficient amount of annotated datasets must be available to train the neural network 10. Ideally, the probability of correct classification should be as high as possible. To achieve this, sufficient training data must be available for both classes.

[0071] Fig. Figure 2 schematically shows how a machine learning algorithm is trained in a self-supervised manner. In this representation, the machine learning algorithm comprises a convolutional neural network 20 followed by a multilayer perceptron 22.

[0072] In this embodiment, the contrastive learning method is applied to train the machine learning algorithm. In a first step, two images 24 and 26 of different objects are input to the machine learning algorithm. The images are then transformed using a transformation 28, which can be a different transformation each time. The transformation 28 can, for example, include selecting a section of the input image, compressing and / or rotating the input image, changing the colors of the image, or applying multiple transformations.

[0073] Images 24 and 26 are then processed by the convolutional neural network 20. The convolutional neural network generates a one-dimensional feature vector, which is in turn input to the multi-layer perceptron 22. The multi-layer perceptron 22 is trained to mark the transformed images of the same image source 24 or 26 as similar. This is represented by the solid-line arrows. The transformed images of different image sources 24 and 26 are marked as different, which is represented by the dashed-line arrows.

[0074] This creates a parameter space in which similar elements are close together and there is a certain distance between different elements.

[0075] Through transformation 28 of images 24 and 26, the machine learning algorithm trains itself independently, i.e., self-supervised, on the features that the transformed elements have in common after the transformation. Therefore, it is important to apply different transformations.

[0076] In Fig. Figure 3 schematically illustrates how a machine learning algorithm may be trained according to one embodiment.

[0077] The process for training the machine learning algorithm begins with step S10, in which one or more images are provided. The images can, for example, be shots of the surface of a product that has been manufactured. The images can then be used for quality control, attempting to find defects on the product's surface based on the images. This is particularly important for products where the surface plays a key role, such as aerodynamically shaped products designed to reduce air resistance or products where the surface is intended to provide protection.

[0078] In step S12, a segment mask is created for the image provided in step S10. The segment mask divides the image into segments, each of which is assigned a class, for example, "OK" and "NOK." Other classes can also be used. The class selection depends on the classification task that the trained machine learning algorithm is intended to perform.

[0079] In step S14, one of the classes stored in the segments is selected. All segments of the selected first class are used together to train the machine learning algorithm using self-supervised learning in step S16. For this purpose, contrastive learning can be used, for example, as described in Fig. 2 is shown.

[0080] In step S18, a second class is selected. The segments of this class are also used to train the self-supervised machine learning algorithm in step S20. Steps S14 and S18, or S16 and S20, can occur in any order, as shown in the flowchart. Fig. 3. It does not matter which of the classes is used first.

[0081] If more than two classes are used, the machine learning algorithm would have to be trained in a self-supervised manner for this class as well.

[0082] In step S22, the segments of all classes are used together to train the machine learning algorithm. This training is performed using supervised learning, such as in Fig. 1a. This allows the machine learning algorithm to correctly classify the images input to it.

[0083] Finally, in step S24, the trained machine learning algorithm is provided so that it can be used for automatic optical inspection. In embodiments, the machine learning algorithm can be retrained in use, provided feedback from the classified objects exists.

Claims

[1] Computer-implemented method for training a machine learning algorithm, wherein the machine learning algorithm is trained to classify an input image (24, 26), wherein the classification comprises assigning the image (24, 26) to one of at least two classes, the method comprising the following steps: - Providing at least one image (S10, 24, 26); - providing a segment mask (S12) for each image, wherein the segment mask divides the image (24, 26) into a plurality of segments and wherein each segment is assigned one of the at least two classes; - Selecting a first class (S14) and hiding all segments not assigned to the first class; - Performing self-supervised learning (S16) on the first-class segments to train the machine learning algorithm to detect first-class segments; - Selecting a second class (S18) and hiding all segments not assigned to the second class; - performing self-supervised learning (S20) on the second-class segments to train the machine learning algorithm to detect second-class segments; - performing supervised learning (S22) with all segments of the image to train the machine learning algorithm to assign the image (24, 26) to one of the classes; and - Providing the trained machine learning algorithm (S24). [2] The computer-implemented method of claim 1, wherein the machine learning algorithm comprises a convolutional neural network (22). [3] A computer-implemented method according to any one of the preceding claims, wherein the self-supervised learning is contrastive learning and the segments are transformed prior to the steps of performing the self-supervised learning. [4] A computer-implemented method according to any one of the preceding claims, wherein the transformation (28) comprises rotating, shearing, cutting out, color-adjusting, color-matching, and / or otherwise transforming the segments. [5] Computer-implemented method according to one of the preceding claims, wherein the image (24, 26) is an image of an object (14, 14') to be inspected, in particular the surface of the object (14, 14') to be inspected, and wherein one of the classes identifies the object (14, 14') as defect-free and another class identifies the object as defective. [6] A computer-implemented method according to any one of the preceding claims, wherein the supervised learning comprises calculating a classification loss function, and wherein the classification loss function is a cross-entropy loss function. [7] A computer-implemented method for classifying images, the method being carried out using a machine learning algorithm, the machine learning algorithm being trained by a method according to any one of claims 1 to 6. [8] A computer program comprising program code for carrying out a method according to any one of claims 1 to 6 and 7 when the computer program is executed on a computer. [9] A computer-readable data carrier comprising program code of a computer program for carrying out a method according to any one of claims 1 to 6 and 7 when the computer program is executed on a computer. [10] A system for training a machine learning algorithm, the system being configured to carry out a method according to any one of claims 1 to 6.

Images