Method for training a machine learning model to predict a geometric relationship between anomalies

Abstract

The invention relates to a method for training a machine learning model to predict a geometric relationship between anomalies. The invention further relates to a computer program, a device, and a storage medium for this purpose.

Description

[0001] The invention relates to a method for training a machine learning model to predict a geometric relationship between anomalies. The invention further relates to a computer program, a device, and a storage medium for this purpose. State of the art

[0002] In industrial applications, visual inspection often involves classifying images into different categories. Tasks can also include detecting subtle differences that lead to the different classification of images that appear similar in other aspects. In these scenarios, subtle variations in the relationship between image parts or the spatial arrangement of visual features can be crucial. For example, one task might be to distinguish between defective and non-defective components. Powerful deep learning techniques can be effectively employed, provided a robust model and sufficient data are available. Deep learning methods using neural networks typically require many annotated samples for each class.In real-world applications, these samples are often either expensive to obtain and annotate, or difficult to collect in sufficient quantities, as is the case with rare defect types. Therefore, visual inspection and detection tasks usually have to be solved with a limited number of samples. Disclosure of the invention

[0003] The invention relates to a method with the features of claim 1, a computer program with the features of claim 9, a device with the features of claim 10, and a computer-readable storage medium with the features of claim 11. Further features and details of the invention will become apparent from the respective dependent claims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program, the device, and the computer-readable storage medium according to the invention, and vice versa, so that mutual reference is always possible with regard to the disclosure of the invention.

[0004] The invention relates in particular to a method for training a machine learning model for predicting a geometric relationship of anomalies, comprising the following steps, wherein the steps can be repeated and / or performed sequentially.

[0005] In a first step, a first training dataset is preferably provided, wherein the first training dataset comprises synthetic samples, each representing the geometric relationship of the anomalies. The geometric relationship expresses, in particular, how the anomalies are arranged in relation to one another. In the first training dataset, the anomalies can, for example, be represented by geometric shapes in the synthetic samples, such that the geometric relationship in the first training dataset expresses, in particular, how these shapes are arranged in relation to one another.

[0006] The geometric relationship within the scope of the present invention is in particular selected from: - An overlap of anomalies, especially in a given synthetic or real sample, - A combination of anomalies, especially in a given synthetic or real sample, i.e., the anomalies touch but do not overlap, - A proximity of anomalies, especially in a given synthetic or real sample, whereby the proximity can be assessed in a more differentiated way, at least in terms of "near" and "far" or also on the basis of a distance in a unit of length, - An isolated anomaly, particularly in a given synthetic or real sample.

[0007] In a further step, the machine learning model is preferably pre-trained based on the initial training dataset to provide a pre-trained machine learning model. This pre-training aims to predict the geometric relationship of the anomalies. For this purpose, the synthetic samples can be assigned labels to perform the pre-training as supervised training. Possible labels include "Overlaid," "Connected," "Detached - Near," "Detached - Far," and "Standalone," with the labels specifically referring to the geometric relationship of the anomalies or shapes. Furthermore, the precision of the machine learning model can be monitored during training based on a corresponding loss function. For example, pre-training can then be terminated once a defined precision level is reached or after a defined number of training episodes.The anomalies can initially be represented as shapes, so that the machine learning model is specifically pre-trained to predict the geometric relationship of the shapes.

[0008] In a further step, a second training dataset is preferably provided, wherein the second training dataset comprises real samples resulting from the acquisition of at least one sensor, the real samples representing the geometric relationship of the anomalies. The at least one sensor is preferably a camera sensor, such that the real samples are, in particular, individual images from the camera sensor. However, it is also conceivable to use other sensors with which the geometric relationship of the anomalies can be represented, for example, a radar, LiDAR, infrared, or ultrasonic sensor.

[0009] In a further step, the pre-trained machine learning model is preferably trained on the basis of the second training dataset. This training aims to predict the geometric relationship between the anomalies. The training can be performed analogously to the pre-training, except that instead of the first training dataset, it is now based on the second.

[0010] The method according to the invention enables the efficient training of a machine learning model with only a few real samples, as it has already been pre-trained based on synthetic samples. The use of synthetic samples can thus advantageously reduce the need for a large, annotated dataset of real samples for training the machine learning model. Predicting geometric relationships is relevant for various applications. For example, a connection between anomalies in electronic components can indicate a short circuit. In materials science, the proximity of anomalies can be used, for instance, to detect microdefects.

[0011] Furthermore, it is conceivable that the synthetic samples comprise elementary geometric shapes, each representing anomalies. This offers the advantage that the pre-trained machine learning model can learn a basic understanding of the geometric relationships between the anomalies by using synthetic samples with simple geometric shapes. These elementary geometric shapes could be, for example, circles, ellipsoids, rectangles, squares, or triangles. This can subsequently enable faster and more efficient training of the machine learning model based on the second training dataset containing the real samples.

[0012] Furthermore, it is conceivable that the synthetic samples include shapes specific to a particular application of the machine learning model. In this way, the pre-trained machine learning model may possess a deeper understanding of the geometric relationships relevant to its subsequent application. For example, the machine learning model could later be used in a manufacturing process. The shapes specific to this application could, in this case, represent components or materials used in manufacturing, where anomaly detection is performed using the trained machine learning model. This can lead to improved accuracy and efficiency in classifying the geometric relationships of anomalies in real-world scenarios, i.e., in the subsequent application of the machine learning model.The use of application-specific shapes in the synthetic samples can enable a more effective adaptation of the machine learning model to the specific challenges of the respective application.

[0013] A further advantage is that the second training dataset comprises fewer samples than the first training dataset, and preferably fewer than 25, more preferably fewer than 10, real samples per category or in total. The method according to the invention enables efficient adaptation of the pre-trained machine learning model to the specific properties of the real samples. Advantageously, a high degree of accuracy in predicting geometric relationships can be achieved even with a limited number of real samples.

[0014] Preferably, the anomalies can be a crack and / or a cavity in a material, and the actual samples can represent the geometric relationship with respect to the crack and / or cavity in the material. This allows the machine learning model to be specifically trained to detect and classify a geometric relationship between cracks and cavities. This can be advantageous in the context of material analysis.

[0015] Furthermore, the pre-training and training phases can be designed to train the machine learning model, based on the first and second provided training datasets, to classify digital images based on image points and / or pixels, preferably edges or pixel attributes. These digital images can, for example, be digital images resulting from a recording during a manufacturing process. This recording is possible, for example, using at least one camera sensor within the manufacturing environment. The classification can be designed to detect anomalies in components or materials, as well as the geometric relationships between these anomalies within the environment depicted by the digital images. Based on the classification, and in particular on a result of the classification, a corresponding action can then be initiated and / or carried out.For example, a warning message could be issued or a production machine stopped if an anomaly with a specific geometric relationship is detected. As part of the training process, testing can also be performed by splitting the provided first and second training datasets into separate test and training data, and using the test data to verify the training progress.

[0016] It is also possible that the procedure includes the following further steps: - Providing sensor data, wherein the sensor data results from the acquisition of at least one sensor during a manufacturing process, - Using the trained pre-trained machine learning model in an inference phase within the manufacturing process to predict the geometric relationship of the anomalies based on the provided sensor data, in particular based on a classification of the sensor data based on image points and / or pixels, wherein the sensor data are preferably in the form of digital images.

[0017] The sensor data could, for example, be images from a camera sensor that captures components or materials as digital images during manufacturing. Using the machine learning model in the inference phase of production can thus enable the early identification of anomalies and their geometric relationships. This allows potential problems to be identified and corrected before they lead to major errors or production downtime.

[0018] Subsequently, depending on the result of the prediction, at least one action can be initiated. For example, a warning message can be issued or a machine in production can be stopped if a specific geometric relationship between the anomalies is found.

[0019] The invention also relates to a computer program, in particular a computer program product, comprising instructions which, when executed by a computer, cause the computer to execute the method according to the invention. Thus, the computer program according to the invention offers the same advantages as those described in detail with reference to a method according to the invention.

[0020] The invention also relates to a data processing device configured to execute the method according to the invention. The device can, for example, be a computer that executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. Alternatively, a non-volatile data storage device can be provided in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0021] The invention may also relate to a computer-readable storage medium which contains the computer program according to the invention and / or includes instructions which, when executed by a computer, cause the computer to execute the method according to the invention. The storage medium is, for example, designed as a data storage device such as a hard drive and / or non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0022] Furthermore, the method according to the invention can also be implemented as a computer-implemented method. Alternatively or additionally, at least one of the disclosed method steps can be computer-implemented and / or carried out automatically.

[0023] Further advantages, features, and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can each be essential to the invention individually or in any combination. The drawings show: Fig. 1 a schematic visualization of a method, a machine learning model, a sensor, a device, a storage medium and a computer program according to exemplary embodiments of the invention, Fig. 2 a schematic representation of a first training data set according to exemplary embodiments of the invention, Fig. 3 a schematic representation of a second training data set according to exemplary embodiments of the invention.

[0024] In Fig. Figure 1 shows a method 100, a machine learning model 50, a sensor 6, a device 10, a storage medium 15 and a computer program 20 according to exemplary embodiments of the invention.

[0025] Fig. Figure 1 shows in particular an embodiment of a method 100 for training a machine learning model 50 to predict a geometric relationship of anomalies 1. In a first step 101, a first training dataset 4 is provided, wherein the first training dataset 4 comprises synthetic samples 2, each representing the geometric relationship of the anomalies 1. In a second step 102, the machine learning model 50 is pre-trained on the basis of the first training dataset 4 to provide a pre-trained machine learning model 50, wherein the machine learning model 50 is pre-trained to predict the geometric relationship of the anomalies 1.In a third step 103, a second training dataset 5 is provided, wherein the second training dataset 5 comprises real samples 3 resulting from the acquisition of at least one sensor 6, and the real samples 3 represent the geometric relationship of the anomalies 1. In a fourth step 104, the pre-trained machine learning model 50 is trained on the basis of the second training dataset 5, wherein the pre-trained machine learning model 50 is trained to predict the geometric relationship of the anomalies 1.

[0026] Fig. Figure 2 shows a first training dataset 4 according to the exemplary embodiments. This dataset comprises several synthetic samples 2, each of which in turn exhibits elementary geometric shapes 11. The synthetic samples 2 can be divided into several categories of geometric relationships and labeled accordingly, for example, "Overlaid", "Connected", "Detached - near", "Detached - distant", and "Standalone", with the labels referring in particular to the geometric relationship of the elementary geometric shapes 11.

[0027] Fig.Figure 3 shows a second training dataset 5 according to exemplary embodiments. This dataset comprises several real samples 3, each of which exhibits anomalies 1. These anomalies are further subdivided into cracks 7 and voids 8 in a material 9. The real samples 3 can be divided into several categories of geometric relationships and labeled accordingly, for example, "Superimposed", "Connected", "Detached - near", "Detached - distant", and "Standalone", with the labels referring in particular to the geometric relationship of the anomalies 1, especially between the cracks 7 and voids 8 in the material 9.

[0028] One way to overcome the challenge of a small number of real-world samples in machine learning is "few-shot learning" (FSL). Few-shot classification (FSC) is particularly related to the more general field of meta-learning. Meta-learning preferably involves creating machine learning models with general learning capabilities, especially through pre-training, so that the machine learning model can subsequently learn more specific machine learning tasks. Humans, for example, possess an innate ability to learn new categories from a minimal number of samples in visual tasks. A crucial element in this process is the human capacity to conceptualize and rationalize concepts derived from images at a higher level.

[0029] In machine learning tasks related to visual understanding, the relationship between image parts can play an important role. For example, the arrangement of detected anomalies and their spatial geometric relationships can lead to different categorizations or classifications. An industrial application of classification based on geometric relationships involves, for example, the investigation of an arrangement of various anomalies or internal defects in the structure of a material 9. If the application allows, a 2D image section of the material 9 can be created and provided as a real sample 3, even using intrusive methods. Initially, the presence of anomalies 1 in such 2D sections can be identified. The subsequent steps of the investigation depend, in particular, on a logic that defines the defects, orAnomalies 1 are categorized according to their relative geometric arrangement. This analysis can also consider structural features near the defects, or anomalies 1. One possible type of anomaly 1 is a structural crack 7. Another type is a cavity 8, specifically an empty space or a piece of another material within material 9. The classification of a geometric relationship between such cavity and crack arrangements may depend on whether the cavity and crack overlap. If they do not overlap, their relative distance may also be significant.

[0030] The so-called few-shot classification, which is in particular an instantiation of meta-learning and few-shot learning, aims to provide machine learning models with classification capabilities that can be adapted to new classes with minimal effort, i.e., with only a few annotated real samples. According to embodiments of the present invention, meta-learning is used to create a machine learning model that understands the concepts of geometric relationships between shapes or anomalies.

[0031] In conventional few-shot classification, the samples of the new, or second, training dataset, used to test the performance of the machine learning model during application, belong to entirely new categories. An example would be new bird species to be classified or a new person to be identified as a new class. In contrast, according to the embodiments, a first training dataset 4 is constructed from synthetic samples 2 that belong to the categories used during the subsequent application of the machine learning model 50, i.e., in particular, categories of geometric relationship concepts or connections that, however, comprise visually different content or shapes. For example, this visually different content could be rectangles or circles, or synthetic versions of application-specific shapes, e.g.,They should be crack- and cavity-like forms arranged in a similar geometric context as in the actual application.

[0032] Examples of geometric relationships between shapes, or anomalies 1, are the following: - Overlaid, - Tied together, - Detached - close, - Detached - removed, - Standalone, i.e., no other anomaly or object representing an anomaly is depicted in an image (or part of an image).

[0033] Meta-learning algorithms comprise, in particular, a meta-training phase and a meta-testing phase. The meta-training phase serves, for example, to train a machine learning model so that it achieves a general learning capability. Specifically, the goal is to provide a machine learning model that is able to adapt to new (usually similar) tasks based on the learning capability acquired in the meta-training phase. This capability is then preferably tested in the meta-testing phase. A single execution of a machine learning task is specifically referred to as an episode (episodic learning). In few-shot classification, preferably in each episode, N classes are selected to form an N-fold classification task, and K sample images are randomly drawn to create N-fold K-shot classification tasks.One goal of few-shot learning is to train a machine learning model for a desired machine learning task, such as N-way classification, using a limited number of annotated samples, i.e., K instances or K shots. Therefore, a meta-training approach can be used to prepare the machine learning model for a machine learning task, such as N-way K-shot classification. Such a learning episode might involve, for example, a short training session on the few (K) annotated samples, followed by a performance evaluation. During meta-training, few-shot learning preferably performs numerous learning episodes with machine learning tasks and samples that are similar to, but not identical with, those used in the meta-testing phase. The main difference between an episode during meta-training and the meta-test lies specifically in the data classes and associated data examples actually used.The usual roles of Support Set, Query Set, BASE and NOVEL data classes and training episodes are described, for example, in the scientific literature (see, e.g., Wei-Yu Chen et al. A Closer Look at Few-shot Classification. openreview.net, Sept. 2018. url: https: / / openreview.net / forum?id=HkxLXnAcFQ).

[0034] According to exemplary embodiments of the invention, a novel application of meta-learning and the "few-shot" classification framework is described. In the meta-training phase of few-shot learning, synthetically generated samples 2 are used, in particular, to pre-train a machine learning model 50 that can predict geometric relationships between shapes, or shapes that represent anomalies 1. These synthetically generated samples 2 can be, for example, elementary geometric shapes 11 such as triangles, quadrilaterals, or circles, or shapes that are specific to an application of the machine learning model 50, i.e., a simulation of cracks 7 and cavities 8.

[0035] More complex and application-specific forms, resulting from the acquisition of data from at least one sensor 6, can be used to further train the pre-trained machine learning model 50 using a small number of real-world samples 3. The precision of the retrained machine learning model 50 can then be evaluated in a test phase, particularly a meta-testing phase.

[0036] The first training dataset 4 for pre-training according to the exemplary embodiments comprises, for example, synthetic samples 2 in the form of images belonging to categories used during the subsequent application of the trained machine learning model, i.e., in particular, categories of geometric relationships. However, the first training dataset 4 also includes other visual content, e.g., elementary geometric shapes 11 such as rectangles or circles, or synthetic versions of application-specific shapes, e.g., cracks 7 and cavity-like shapes 8, arranged in a similar geometric relationship to that in the actual application.

[0037] During the meta-training phase, the shapes used can thus differ advantageously from those used during the application. An example is meta-training on elementary geometric shapes 11 such as circles, ellipsoids, rectangles, squares, or triangles. However, during the meta-training phase, the shapes used can also be more similar to the domain data, i.e., the shapes specific to an application of the machine learning model 50. For example, a synthetic version of cracks 7 and cavities 8 can be used.

[0038] The synthetic samples 2 can be generated by a computer script, e.g., defined by a random set of shapes that follow certain rules. Alternatively, the synthetic samples 2 can be generated by generative machine learning methods.

[0039] One possible application of the method according to the invention is the filtering of crack-cavity arrangements. In this process, crack-cavity arrangements can be classified based on the geometric relationship between the two anomaly types. According to exemplary embodiments, various geometric relationships between these crack-cavity arrangements can be predicted and classified, for example, based on semantically segmented images.

[0040] The method according to the invention can further be applied in other areas where geometric relationships between possibly complex shapes are important, but only a few annotated samples per class are available.

[0041] The preceding explanation of the embodiments describes the present invention solely by way of examples. Naturally, individual features of the embodiments can be freely combined with one another, provided this is technically feasible, without departing from the scope of the present invention.

Claims

[1] Method (100) for training a machine learning model (50) for a prediction of a geometric relationship of anomalies (1), comprising the following steps: - Providing (101) a first training dataset (4) wherein the first training dataset (4) comprises synthetic samples (2) each representing the geometric relationship of the anomalies (1), - Pre-training (102) machine learning model (50) based on the first training data set (4) to provide a pre-trained machine learning model (50), wherein the machine learning model (50) is pre-trained to predict the geometric relationship of the anomalies (1), - Providing (103) a second training dataset (5), wherein the second training dataset (5) comprises real samples (3) resulting from a recording of at least one sensor (6), wherein the real samples (3) represent the geometric relationship of the anomalies (1), - Training (104) the pre-trained machine learning model (50) on the basis of the second training data set (5), wherein the pre-trained machine learning model (50) is trained to predict the geometric relationship of the anomalies (1). [2] Method (100) according to claim 1, characterized by , that the geometric relationship is selected from: - An overlap of anomalies (1), - A combination of anomalies (1), - A proximity of anomalies (1), - An isolated anomaly (1). [3] Method (100) according to any one of the preceding claims, characterized by, that the synthetic samples (2) comprise elementary geometric shapes (11), where the elementary geometric shapes (11) each represent anomalies (1). [4] Method (100) according to any one of the preceding claims, characterized by , that the synthetic samples (2) include forms that are specific for an application of the machine learning model (50). [5] Method (100) according to any one of the preceding claims, characterized by , that the second training dataset (5) contains fewer samples than the first training dataset (4) and preferably less than 25, further preferably less than 10, real samples (3) per category. [6] Method (100) according to any one of the preceding claims, characterized by, that the anomalies (1) are a crack (7) and / or a cavity (8) in a material (9) and the real samples (3) represent the geometric relationship with respect to the crack (7) and / or the cavity (8) in the material (9). [7] Method (100) according to any one of the preceding claims, characterized by , that the pre-training (102) and the training (104) are intended to train the machine learning model (50) on the basis of the first provided training data set (4) and the second provided training data set (5) to classify digital images on the basis of picture points and / or pixels, wherein the digital images are preferably digital images resulting from a recording in the context of a manufacturing process. [8] Method (100) according to any one of the preceding claims, characterized by , that the procedure (100) further comprises the following steps: - Providing sensor data, wherein the sensor data results from the acquisition of at least one sensor (6) during a manufacturing process, - Using the trained pre-trained machine learning model (50) in an inference phase during manufacturing to predict the geometric relationship of the anomalies (1) based on the provided sensor data. [9] Computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause it to execute the method (100) according to any of the preceding claims. [10] Device (10) for data processing which is configured to carry out the method (100) according to any one of claims 1 to 8. [11] Computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause it to perform the steps of the method (100) according to any one of claims 1 to 8.

Images