Method for providing combined training data set for machine learning model

Abstract

The present invention relates to a method for providing a combined training dataset for a machine learning model, comprising the steps of providing image data, where the image data comprises unannotated portions and annotated portions, training a base machine learning model based on the unannotated portions of the image data in order to provide a generic model, training a generic model based on annotated portions of the image data to provide a semantic segmentation model, analyzing the training dataset by means of the semantic segmentation model to provide semantics for the training dataset, analyzing the training dataset by means of a zero sample segmentation model to provide segmentation for the training dataset, a combined training dataset is provided based on a combination of the provided semantics and the provided segmentation of the training dataset. The invention also relates to a computer program, a device and a storage medium for this purpose.

Description

Technical Field

[0001] The present invention relates to a method for providing a combined training data set for a machine learning model. The present invention also relates to a computer program, a device and a storage medium for this purpose. Background Art

[0002] Automatic segmentation algorithms such as the foundation models of "Segment Anything" are known in the prior art. In particular, the models are trained, in particular with large datasets, in order to provide zero-shot segmentation. This enables: segmenting objects or features in an image without explicitly training the model with the aid of annotations or examples of the specific objects or features in advance.

[0003] In many cases, semantic segmentation can be a challenge if sometimes no annotations (labels in English) of the data exist. This is relevant, for example, in the case of automated optical inspection (AOI) tasks, such as inspecting components in production. Therefore, conventional solutions for semantic segmentation usually require that each pixel of the image be annotated, which can be very time-consuming, error-prone, and difficult. Summary of the invention

[0004] The subject matter of the invention is a method having the features of claim 1, a computer program having the features of claim 8, a device having the features of claim 9 and a computer-readable storage medium having the features of claim 10. Further features and details of the invention are derived from the respective dependent claims, the description and the drawings. Features and details described in the context of the method according to the invention obviously also apply in the context of the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, so that the disclosure concerning the various aspects of the invention always refers to or can refer to each other.

[0005] In particular, the subject of the present invention is a method for providing a combined training data set for a machine learning model. In particular, according to a first step, the method includes providing image data, wherein the image data includes an unannotated part (i.e., unannotated image data) and an annotated part (i.e., annotated image data). In addition, according to another step, the method may include training a basic machine learning model based on the unannotated part of the image data in order to provide a general model. Here, the unannotated part of the image data may also be annotated. In addition, the method may include training a general model based on the annotated part of the image data in order to provide a semantic segmentation model. Here, alternatively or additionally, image data annotated in the scope of training the basic machine learning model may be used. In addition, in the method, it may be provided to analyze the training data set with the help of a semantic segmentation model in order to provide semantics for the training data set. Then, the training data set may be analyzed with the help of a zero-shot segmentation model in order to provide segmentation for the training data set. In addition, it may be feasible to provide a combined training data set based on a combination of the provided semantics and the provided segmentation of the training data set. With the help of this method, the training data set can be advantageously supplemented with annotations (English: labels) and thus improved. For this purpose, only a part of the image data may have annotations for training the general model. In other words, the image data may have, for example, image information representing at least one object. In contrast to the unannotated parts, the annotated parts of the image data may also be annotated with image information of the annotated parts, for example with the class of the represented object. In particular, the effort and cost of manually annotating each individual image of a huge training dataset may be reduced by the method. Thus, high-quality annotations may be achieved with less effort without compromising the performance of the machine learning model to be trained.

[0006] The image data may be derived from detection by at least one sensor, such as a camera sensor, and / or may include synthetic image data. The training data set may include training image data, which may also be derived from detection by at least one sensor, such as a camera sensor, and / or may include synthetic image data. The image data and the training data set may include at least one individual image, in particular a plurality of individual images. In the scope of the present invention, a foundation machine learning model (English: foundation model) should be understood in particular as a large model of machine learning that has been trained on a large scale with a large amount of data, for example by self-supervised or semi-supervised learning. The foundation machine learning model can thereby be adapted to different downstream tasks.

[0007] In the context of the present invention, the generic model is in particular a basic machine learning model which is adapted, in particular trained, with respect to unannotated image data, wherein the training is preferably self-supervised training.

[0008] The semantic segmentation model is preferably a general model which is adapted, in particular trained, on the annotated image data, wherein the training is preferably at least semi-supervised training due to the presence of annotations.

[0009] Analyzing the training data set by means of a semantic segmentation model and analyzing the training data set by means of a zero-shot segmentation model are preferably corresponding inferences of the semantic segmentation model and the zero-shot segmentation model. Zero-shot segmentation is particularly related to the problem of segmenting an object or entity in image data, while previously annotated examples of the specific object or entity are not visible during training. Here, a type of additional information, such as a text description, an image-based description and / or a semantic attribute, can be used in a supporting manner. The description can be a user's input based on a prompt word. For example, the task in zero-shot segmentation is to bridge the semantic gap between known categories (seen during training) and new categories (not seen during training), without requiring complete pixel-level annotations for the new categories. By semantics, a segment with corresponding annotations can be provided for at least a portion of the training data set, in particular for at least a portion of each individual image of the training data set. Segmentation preferably provides segmentation for at least a portion of the training data set, in particular for the complete training data set. For example, a segmentation is provided in each image of the training data set. By combining, a segment with a semantically corresponding annotation can be linked to a segment of the segmentation. This can be advantageous because segmented segments can be expressed in more detail, more precise, and / or more complete, but without annotations, whereas semantics can have ambiguous and / or incomplete segments, albeit with annotations.

[0010] Optionally, it is feasible that providing the image data further comprises the following steps:

[0011] - provide segmentation for a portion of unannotated image data via a zero-shot segmentation model,

[0012] - Assign annotations to the provided segments to provide annotated portions of the image data.

[0013] Therefore, it is also conceivable that the image data initially have no annotations and that the annotated parts of the image data are provided by the previous steps. Advantageously, not all sub-elements of the training dataset need to be annotated, but only a small part thereof using the zero-shot segmentation model.

[0014] Advantageously, within the scope of the invention it can be provided that the annotation is assigned using at least one prompt word based input of the user, wherein the annotation is assigned to the provided segment by means of at least one prompt word based input. The prompt word based input is preferably an image based prompt word based input and comprises, for example, the determination of a position, in particular coordinates and / or a bounding box (English: boundingbox) in the image data. Thus, the annotation can be assigned based on the determined position and / or bounding box. For example, it can be considered that the user identifies a defect in a component in an image of the image data and assigns a corresponding annotation to the corresponding position of the defect in the image.

[0015] It is also advantageous that analyzing the training dataset with the aid of a semantic segmentation model comprises the following steps:

[0016] - segmenting the training dataset with the aid of a semantic segmentation model so as to provide first segmented regions in the training dataset,

[0017] - Assigning corresponding features to the corresponding first segmented regions through a semantic segmentation model to provide semantics for the training dataset.

[0018] The corresponding characteristics can also be annotations. For example, it can be considered that the corresponding characteristics describe a component or assembly, a region of a component or assembly, or a defect in a component or assembly. Therefore, the meaning of each first segmented area can be advantageously provided by the corresponding characteristics of at least a part of the training data set through semantics.

[0019] Furthermore, it can be provided within the scope of the present invention that the analysis of the training data set by means of the zero-shot segmentation model comprises the following steps:

[0020] - segmenting the training data set with the aid of the zero-shot segmentation model to provide a segmentation for the training data set based on second segmented regions in the training data set.

[0021] It can be provided that the zero-shot segmentation model performs segmentation without additional information, such as an input based on a prompt word. The segmentation particularly provides a second segmentation region, which is preferably a segmentation for the training data set. Segmentation with the help of the zero-shot segmentation model can advantageously achieve a very detailed, precise and / or complete segmentation.

[0022] Advantageously, it may be proposed in the present invention that providing a combined training data set comprises the following steps:

[0023] - comparing a first segmented region of the training data set with a second segmented region of the training data set in order to determine at least one consistent region.

[0024] Based on the at least one determined consistent region, at least one property of the first segmented region can be assigned to the second segmented region in order to combine the provided semantics of the training data set and the provided segmentation. In other words, the locations where there is an overlap between the first segmented region and the second segmented region are compared so that the property of the first segmented region can be assigned to the second segmented region based on the overlap. It can be considered that before the combination is performed, the consistency or overlap must exceed a certain threshold and / or relative threshold with respect to at least one of the segmented regions.

[0025] It is also possible to train a machine learning model based on a combined training data set for classification and / or detection based on image information, wherein the image information is in particular pixels of an image record and / or represents at least one recorded object, wherein the detection preferably includes the detection of defective components in production, wherein the detection is preferably performed based on semantic segmentation and / or pixel-based classification. For example, a defective component may have production errors that should be detected. For example, here, faulty components and / or defective soldering points or conductor tracks and / or scratches may be present as production errors of the component. In this case, a combined training data set can be advantageous, since in particular in production a large amount of image data may be present and here only a small amount of image data with production errors may be present. Therefore, advantageously, the extensive data set does not have to be fully manually annotated in order to be able to be used for training a machine learning model.

[0026] The image data can be derived from detection with the aid of a camera sensor. In this case, the environment can be represented by the values ​​of image points, preferably pixels, of the image data. By means of a classification based on the values ​​and preferably an image classification, objects of the environment, such as defective components, can be detected if necessary. In this case, the classification and image classification can also be provided in the form of semantic segmentation (i.e. pixel-by-pixel or region-by-region classification) and / or object detection. The image data can also be, for example, images of a radar sensor and / or an ultrasonic sensor and / or a lidar sensor and / or a thermal imager. Correspondingly, the image can also be configured as a radar image and / or an ultrasonic image and / or a thermal image and / or a lidar image.

[0027] The subject of the invention is also a computer program, in particular a computer program product, which comprises instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to the invention. The computer program according to the invention therefore brings with it the same advantages as described in detail with reference to the method according to the invention.

[0028] The subject of the invention is also a device for data processing, which is designed to carry out the method according to the invention. For example, a computer can be provided as the device, which executes the computer program according to the invention. The computer can have at least one processor, in particular at least one graphics processing unit (GPU), for executing the computer program. A non-volatile data memory can also be provided, in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0029] The subject matter of the present invention may also be a computer-readable storage medium, which has a computer program according to the present invention and / or comprises instructions which, when executed by a computer, cause the computer to carry out the method according to the present invention. The storage medium is designed, for example, as a data memory, such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into a computer.

[0030] Furthermore, the method according to the invention can also be designed as a computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Further advantages, features and details of the invention are apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and the description may be essential to the invention individually or in any combination.

[0032] in:

[0033] Figure 1 Schematic visualizations showing methods, components, users, devices, storage media, and computer programs according to embodiments of the present invention,

[0034] Figure 2 A schematic diagram showing a method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] exist Figure 1 , a method 100, a component 8, a user 9, a device 10, a storage medium 15 and a computer program 20 according to an embodiment of the present invention are schematically shown.

[0036] Figure 1A method 100 for providing a combined training data set 4' for a machine learning model according to an embodiment of the present invention is shown. In a first step 101, image data are provided, wherein the image data comprises an unannotated part 1 and an annotated part 1'. In a second step 102, a basic machine learning model is trained based on the unannotated part 1 of the image data in order to provide a general model 2. The general model 2 can thus be a basic machine learning model adapted, in particular trained, with respect to the unannotated image data. The training is preferably performed as self-supervised training.

[0037] In a third step 103, the general model 2 is trained based on the annotated part 1' of the image data to provide a semantic segmentation model 3. The semantic segmentation model 3 is therefore preferably a general model 2 that has been adapted, in particular trained, on the annotated image data. It is advantageous here that the training is performed as at least semi-supervised training, since the annotations are present.

[0038] In a fourth step 104, the training data set 4 is analyzed with the aid of the semantic segmentation model 3 in order to provide semantics 5 for the training data set 4. In a fifth step 105, the training data set 4 is analyzed with the aid of the zero-shot segmentation model 7 in order to provide segmentation 6 for the training data set 4. In a sixth step 106, a combined training data set 4' is provided based on the combination of the provided semantics 5 and the provided segmentation 6 of the training data set 4. Here, a comparison is preferably performed at locations where there is an overlap of the semantics 5 and the segmentation 6, so that properties of the semantics 5, i.e., in particular annotations, can be assigned to corresponding regions or segments of the segmentation 6 based on the overlap. In other words, the semantics 5 can be combined with the segmentation 6 by determining the overlapping regions of the semantics 5 and the segmentation 6 in the training data set. Within the segmentation 6, the corresponding properties of the segments of the semantics 5 are preferably assigned to the individual segments of the segmentation 6 with the maximum overlap.

[0039] Figure 2 A further possible method according to an exemplary embodiment of the present invention is schematically shown, to which reference is made in the following description.

[0040] One aspect of an embodiment according to the present invention is to improve semantic segmentation when the annotation part of the training data set 4 is scarce. The improvement of the training data set 4 is performed in particular in multiple steps. First, the general capabilities of the basic machine learning model can be improved by training with the help of unannotated image data 1. In addition, the input based on the prompt words can be used to annotate a part of the image data, in particular for fine-tuning. Then, reasoning can be performed to obtain the semantics 5 of the training data 4 and combine it with the result of the zero-shot segmentation 6 (i.e., in particular the segmentation 6 of the training data 4).

[0041] Semantic segmentation is often particularly needed in order to obtain image regions with semantics. In machine learning models, especially neural networks, weights can be initialized or pre-trained according to a random principle. Since so-called low-sample segmentation should preferably be performed according to an embodiment of the present invention, knowledge about a specific data set or context, such as automatic optical inspection (AOI), can be introduced, for example. In order to achieve this, a basic machine learning model can first be trained with the help of self-supervised learning techniques, such as contrastive learning or similar techniques, to provide a general model 2.

[0042] Once the generic model 2 exists, the input request can be performed with the help of a zero-shot segmentation model, such as "Segment Anything", in order to obtain image segments. Preferably, annotations are assigned to the segments for a small number of images of the image data 1, for example 10-100 images.

[0043] The prompt word based input of the user 9 can be used to reduce the annotation cost. Once there is a trained semantic segmentation model 3, the prompt word based input is preferably no longer needed. The prompt word based input can be purely image-based rather than text-based. The prompt word based input can be, for example, a position (x, y coordinates) and / or a bounding box (English: BoundingBox).

[0044] Some of the semantic segmentation results, i.e. some of the annotated parts 1' of the image data, can be used to train a general model 2 to provide a semantic segmentation model 3. Then, inference of the semantic segmentation model 3 can be performed based on the training data set 4 to provide semantics 5. In this way, in particular weakly segmented images with corresponding semantics are obtained.

[0045] Furthermore, zero-shot segmentation can be performed based on the training dataset 4 through a zero-shot segmentation model to obtain semantic-free high-quality segmentation 6.

[0046] In particular, the above results, i.e. semantics 5 and segmentation 6, can be combined by two models through a coordination scheme within each segment. There are various possibilities for performing the combination. One possible strategy is to check the overlapping weak segmentation mask, i.e. the first segmentation area, i.e., the semantics 5, in particular, in the strong segmentation mask, i.e., the second segmentation area, i.e., the segmentation 6. Within the strong segmentation mask, properties such as classes are preferably assigned to the weak segmentation mask that has the largest overlap with the strong segmentation mask.

[0047] The above explanations of the embodiments describe the invention only within the scope of examples. Obviously, the individual features of the embodiments can be combined freely with each other as long as it is technically reasonable, without departing from the scope of the invention.

Claims

1. A method (100) for providing a combined training data set (4') for a machine learning model, comprising the following steps: - providing (101) image data, wherein the image data comprises an unannotated portion (1) and an annotated portion (1'), - training (102) a base machine learning model based on the unannotated portion (1) of the image data so as to provide a general model (2), - training (103) said general model (2) based on the annotated portion (1') of said image data so as to provide a semantic segmentation model (3), - analyzing (104) a training data set (4) by means of the semantic segmentation model (3) in order to provide semantics (5) for the training data set (4), - analyzing (105) the training data set (4) by means of a zero-shot segmentation model (7) in order to provide a segmentation (6) for the training data set (4), - providing (106) a combined training data set (4') based on a combination of the provided semantics (5) and the provided segmentation (6) of said training data set (4).

2. The method (100) according to claim 1, It is characterized in that Providing (101) image data further comprises the following steps: - providing segmentation for a portion of the unannotated image data (1) by means of the zero-shot segmentation model (7), - assigning annotations to the provided segments to provide an annotated portion (1') of said image data.

3. The method (100) according to claim 2, It is characterized in that The annotation is assigned using at least one cue-word based input of a user (9), wherein the annotation is assigned to the provided segment by the at least one cue-word based input.

4. The method (100) according to any one of the preceding claims, It is characterized in that Analyzing (104) the training data set (4) by means of the semantic segmentation model (3) comprises the following steps: - segmenting the training data set (4) by means of the semantic segmentation model (3) so as to provide a first segmented region in the training data set (4), - assigning corresponding features to corresponding first segmented regions by means of the semantic segmentation model (3) so as to provide the semantics (5) for the training data set (4).

5. The method (100) according to any one of the preceding claims, It is characterized in that Analyzing (105) the training data set (4) by means of the zero-shot segmentation model comprises the following steps: - segmenting the training data set (4) by means of the zero-shot segmentation model in order to provide a segmentation (6) for the training data set (4) based on second segmented regions in the training data set (4).

6. The method (100) according to claims 4 and 5, It is characterized in that Providing (105) a combined training data set (4') comprises the following steps: - comparing a first segmented region of the training data set (4) with a second segmented region of the training data set (4) in order to determine at least one consistent region, Based on at least one determined consistent region, at least one property of the first segmented region is assigned to the second segmented region in order to combine the provided semantics (5) and the provided segmentation (6) of the training data set (4).

7. The method (100) according to any one of the preceding claims, It is characterized in that The machine learning model is trained based on a combined training data set (4') for classification and / or detection based on image information, wherein the image information is in particular pixels recorded in the image and / or represents at least one recorded object, wherein the detection preferably includes the detection of defective components (8) in production, wherein the detection is preferably performed based on semantic segmentation and / or pixel-based classification.

8. A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to perform the method (100) according to any one of the preceding claims.

9. A device (10) for data processing, the device being designed to: execute the method (100) according to any one of claims 1 to 7.

10. A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer to perform the steps of the method (100) according to any one of claims 1 to 7.

Images