Computer-implemented methods and systems for generating datasets

Abstract

The present invention relates to a computer-implemented method and system for generating a dataset for training and / or validating a first machine learning algorithm. [Solution] The method includes the steps of: supplying an input dataset (S10) which includes a set of multiple images, wherein objects to be classified are mapped onto the images, and each image is associated with at least one class having a class label; generating a cutout of each image (S12) which includes the object to be classified (S12.1); inputting the image and the generated cutout into a second machine learning algorithm (S12.2) which generates a label for the cutout; and supplying a dataset (S14) which includes a set of multiple images, with each image associated with the respective generated label.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method and system for generating a dataset for training and / or validating a first machine learning algorithm.

Background Art

[0002] Prior Art For machine learning models, especially for segmentation and object recognition, training data plays a central role. Such data often consists of a plurality of images or videos with class names assigned to represent objects in the images. The quality and accuracy of such class names are critically important as they have a significant impact on the performance of the trained model. The class names are used as text labels associated with visual information in a segmentation dataset. This enables the model to form an association between the visually presented object and its semantic reference. Such an association forms the basis for later identifying and classifying objects in new, unobserved data.

[0003] In conventional machine learning, such class names are defined with a fixed vocabulary. This often results in a limitation in representational ability, especially in scenarios using an open vocabulary where the model must be able to identify new or rare objects that may not be present in the training data. If the class names in the training data are inaccurate or overly general, an incorrect association between the visual information and the text labels is caused, which further reduces the model accuracy. This is relevant not only for the training phase but also for later model evaluation, because inaccurate or incorrect classifications can lead to incorrect results.

[0004] In the training process of a segmentation model, the neural network is trained to identify patterns in training images and associate these patterns with corresponding labels, or class names. The training process involves iteratively adjusting the model to minimize the difference between the model's predictions and the actual labels. In this case, a loss function is used to measure how well the model's predictions match the actual class names. If inaccurate or inconsistent class names are used, the model may learn incorrect patterns or overgeneralize, which will reduce the accuracy of subsequent segmentation or object recognition.

[0005] In particular, in segmentation environments using open vocabulary, where the model must be able to identify a large number of objects with high accuracy, accurate and consistent class names are critically important. Here, a multimodal model capable of processing both visual and textual information opens up new possibilities. By integrating a large-scale language model (LLM) such as GPT, this model can not only access visual information but also grasp the semantic context of class names. This can reduce discrepancies between visual and textual information by improving not only the learning of visual features but also the understanding of objects and their relationships in text.

[0006] Therefore, the precise use and definition of class names is a crucial aspect of data creation and a vital component of the training process for models developed for precise segmentation tasks. Insufficient precision in class names within the training data can lead to irreversible errors in the model, which directly impact the generalized object recognition capabilities. [Overview of the project] [Problems that the invention aims to solve]

[0007] Therefore, the problem on which the present invention is based is to provide a method and system for improving the quality of class names when generating training data and / or validation data, thereby enabling better training and testing of segmentation and object recognition using open vocabulary. [Means for solving the problem]

[0008] The above issues are resolved by the scope of each independent claim.

[0009] Disclosure of the invention According to a first aspect of the present invention, the above problem is solved by a computer-implemented method for generating a dataset for training and / or validating a first machine learning algorithm, The method involves the following steps, namely: - A step of supplying an input dataset comprising a set of multiple images, wherein objects to be classified are mapped onto the images, and each image is associated with at least one class having a class label. -For each image, • Generate a cutout of the image containing the object to be classified. The image and the generated cutout are input into a second machine learning algorithm. • Generate a label for the cutout. Steps and - A step of supplying a dataset that includes a set of multiple images, and each image is associated with a generated label, It is resolved by methods including the following.

[0010] According to a second aspect of the present invention, the above problem is solved by a system for generating a dataset for training and / or validating a first machine learning algorithm.

[0011] The system comprises a supply unit configured to supply an input dataset consisting of a set of multiple images, wherein objects to be classified are mapped onto the images, and each image is associated with at least one class having a class label.

[0012] The system further applies to each image: a. Generate a cutout of the image containing the object to be classified. b. The image and the generated cutout are added to the second machine learning algorithm. c. Generate a label for the cutout. It is equipped with a computing unit configured in such a way.

[0013] Furthermore, the system includes an output unit configured to supply a dataset containing a set of multiple images, each with a corresponding generated label associated with it.

[0014] Large Language Models (LLMs) process language by using complex and deep neural networks trained to identify patterns and dependencies in text. These models typically rely on a transformer architecture capable of detecting both local and global dependencies within the text. At the heart of this process are so-called tokens, which represent the smallest linguistic units, such as words or word parts. Tokenization transforms the input text into a sequence of numerical vectors, which are then fed into the model.

[0015] During training, LLM learns to interpret tokens by analyzing the context in which they appear. This is done by using a self-attention-like mechanism that allows the model to focus on relevant information across the entire sequence, regardless of its position in the text. Context here is critically important to the model's ability to detect the meaning of words and phrases, particularly in relation to handling ambiguity and multiple meanings. In the case of class names, this means that LLM can not only treat the name as an isolated label, but also understand the semantic relationships between various class names and their meanings in a given context.

[0016] Processing class names using LLM requires an accurate understanding of the semantic nuances behind those names. LLM learns this by analyzing large amounts of data on how the class names are used in various contexts. This aggregation of information allows the model to derive generalizations that enable the flexible application of class names to new objects and situations, even when the class names conflict with concepts that do not explicitly appear in the training dataset.

[0017] The essential aspect of language processing in LLM is the so-called embedding layer, which transfers the semantic meaning of words and phrases into a continuous vector space. In such a vector space, semantically similar words are located in close proximity to each other, allowing the model to learn the relationships between various class names. For example, the model can learn that the concepts of "dog" and "cat" have similar characteristics in many contexts, while the concepts of "car" and "airplane" represent different object classes and therefore exist in different semantic spaces. This ability for semantic generalization plays a crucial role when LLM is used in segmentation or object recognition applications using open vocabulary.

[0018] Here, the class names in the data used for training such a model are used as anchors for the semantic understanding of the model. Since the LLM can learn from multi-modal data that includes both text information and image information, the meaning of these class names can be associated with visual features. Such multi-modal capabilities enable the model not only to associate class names with text descriptions but also to transfer them to visual objects displayed by images or videos. Such an association between language information and visual information greatly expands the applicability of the LLM, especially enabling more accurate object recognition and segmentation in real-world scenarios within the environment using an open vocabulary.

[0019] In the framework of the present invention, an input data set including a set of multiple images is supplied. The images may well include, for example, real captured images, i.e., for example, those including photographs or X-ray images, or may also include, for example, synthetic images from a virtual environment.

[0020] For the method, preferably, a machine learning algorithm different from the machine learning algorithm to be trained is used. Thus, let the machine learning algorithm to be trained be referred to as the first machine learning algorithm, and in contrast, the machine learning algorithm used according to the present invention be referred to as the second machine learning algorithm.

[0021] The first machine learning algorithm is trained using the data generated according to the present invention. The task of the first machine learning algorithm may basically have any nature when the first machine learning algorithm uses image data as input. The first machine learning algorithm does not necessarily have to be able to process text.

[0022] The second machine learning algorithm receives the images of the input data set. Then, each image is processed. First, for this purpose, cutouts of the images containing the objects to be classified are generated. Usually, a machine learning algorithm is not required for this task. This task can also be solved by known image processing methods when the parameters necessary for cropping are supplied, for example. Then, the cutouts thus generated and the entire image used as the context of the cutouts in subsequent processing processes are input into the second machine learning algorithm.

[0023] The second machine learning algorithm generates new labels associated with the cutouts in response to the input. This process is repeated for all images in the input data set, whereby new labels are associated with each cutout of each image. By mechanically processing the image data, a certain degree of consistency is obtained in the allocation of labels and the class attributes of the accompanying image data, thereby improving the quality of the training data for the first machine learning algorithm. At the same time, since label generation is performed flexibly, labels can be generated to fit the input image data. In this way, the present invention solves its task.

[0024] In one embodiment, the second machine learning algorithm includes a visual language model.

[0025] A visual language model is a machine learning algorithm that receives both language and graphic data, particularly images or image data, as inputs and can process them together. Such a model is also referred to as a multimodal model.

[0026] In one embodiment, the instructions for outputting the labels are described in natural language.

[0027] Furthermore, the machine learning algorithm receives natural language input, so-called prompts. These prompts instruct the machine learning algorithm to generate labels for image cutouts. The prompts may also include further instructions, for example, that a second machine learning algorithm should behave as if it were an expert in the field of image classification or class naming. Specifically, the prompts might include: "Assuming you are an expert, rename the segments in the image to improve the quality of the naming. The first image is the interest cutout segment, which was originally labeled [Placeholder]. The second image is a context image that uses a red border frame to show where the cutout is located within the image." Your objective is to find a name similar to the original name "[Placeholder]," for example, a synonym or subcategory, or a name corresponding to the earliest common usage describing the cutout in this case. Briefly explain your considerations and end your response with "A:new_name," where new_name is your chosen name. If the segment name is incorrect or you cannot find a new name, respond with "A:NA." It will look like this.

[0028] Large-scale language models can receive concepts derived from text. Differences in word choice or formulation play a secondary role in processing prompts. In other words, prompts that are similar in semantic content can be used similarly. The [placeholder] shown in the prompt may be occupied by a class label assigned to the original image or image cutout.

[0029] In one embodiment, the label is selected from a list consisting of synonyms or subcategories of the first label.

[0030] The list can be supplied to a second machine learning algorithm, which may be user-defined or proposed by another machine learning algorithm. This list can be used to limit the number of possible class labels to a defined size. This, in particular, prevents each image from obtaining a unique label.

[0031] In one embodiment, the method further includes: - A step in which each image is input into a third machine learning algorithm along with instructions for verifying the label, - Steps to verify the generated labels, Includes.

[0032] The instructions for validating labels can be, in particular, natural language instructions, i.e., prompts. For example, a prompt is: "Assuming you are an expert, you're asked to rename a segment in an image to improve its naming quality. Another expert says the segment should be called [Generated Label]. The original name is [Placeholder]. Which do you think is better, the original name or the new name?" It will look like this.

[0033] Here, the label generated by the second machine learning algorithm is used for [Generated Label], and [Placeholder] is replaced with the class label set in the input dataset.

[0034] Furthermore, the third machine learning algorithm may be another machine learning algorithm that receives the output of the second machine learning algorithm. In particular, the first and second machine learning algorithms can be configured to communicate with each other as so-called agents, so that the output of the second machine learning algorithm is used directly as input for the third machine learning algorithm.

[0035] Alternatively, a model is available that uses two independent threads, where one thread handles the task of a second machine learning algorithm and the other thread handles the task of a third machine learning algorithm.

[0036] In one embodiment, if the output of a third machine learning algorithm indicates that a pre-generated label has not been validated, a different label is generated.

[0037] If the third machine learning algorithm is unable to validate the generated label and holds the so-called view that the generated label does not fit the cutout, then another label is generated. This can be done by the second machine learning algorithm by receiving a new instruction to generate a label. Alternatively, the other label can also be generated by the third machine learning algorithm.

[0038] Theoretically, this basic approach continues until the second and third machine learning algorithms match the label. However, in practice, the process can be shortened by using the second or xth label generated in the event of a mismatch.

[0039] In one embodiment, the verification includes comparing a label generated by a third machine learning algorithm with another label.

[0040] A third machine learning algorithm can validate the generated labels by examining how well they linguistically fit to the objects read from the image cutouts. This validation is performed by computation within the third machine learning algorithm, using complex expressions, but the result is the output for the validation result.

[0041] In one embodiment, a label generates a natural language text-based rationale for that label, and the label is validated using this rationale.

[0042] Validation using a third machine learning algorithm based solely on labels can sometimes yield inaccurate results. This is because the amount of data used as input to the third machine learning algorithm is limited. When evidence is attached, the third machine learning algorithm can similarly examine the data, thereby achieving a better classification of the labels generated by the second machine learning algorithm. In other words, the amount of input data increases, ultimately leading to a more accurate output.

[0043] In one embodiment, the first machine learning algorithm includes an algorithm for identifying traffic signs, road surface and lane integrity, pedestrians, and / or vehicles.

[0044] In another embodiment, the present invention relates to a computer program that includes program code for carrying out the above-described method when executed on a computer.

[0045] In another embodiment, the present invention relates to a computer-readable data carrier including program code of a computer program for carrying out the above-described method when executed on a computer.

[0046] In another embodiment, the present invention relates to a system for generating a dataset for training and / or validating a first machine learning algorithm, wherein the system is configured to perform the method described above.

[0047] In summary, the present invention proposes a method for generating a dataset for training and / or validating a first machine learning algorithm, a computer program, a computer-readable data carrier including program code, and a system for generating a dataset for training and / or validating a first machine learning algorithm.

[0048] The configurations and variations described can be combined in any way.

[0049] Other possible configurations, variations, and realizations of the present invention include combinations of features of the present invention that are not expressly listed above or below in relation to the examples.

[0050] The attached drawings are for further understanding of embodiments of the present invention. The drawings illustrate embodiments and are used in connection with the description of the basic form and concept of the present invention.

[0051] In relation to the drawings, many other embodiments and the advantages described above can be obtained. The elements shown in the drawings are not necessarily drawn to scale relative to each other. The drawings show the following: [Brief explanation of the drawing]

[0052] [Figure 1] This is a schematic diagram showing a flow of the method according to the present invention according to one embodiment. [Figure 2] This figure shows a system according to one embodiment of the present invention.

[0053] In each drawing, elements, components, or parts having the same or equivalent function are denoted by the same reference numeral unless otherwise explicitly stated. [Modes for carrying out the invention]

[0054] Figure 1 schematically shows the flow of the method according to one embodiment. The method can be considered as two nested processes, namely an outer process and an inner process.

[0055] The outer process begins with step S10, which involves supplying the input dataset. If the first machine learning algorithm is to be used, for example, for monitoring surrounding traffic, and the input data for that machine learning algorithm includes camera images, then many images of the traffic situation can be used as the input dataset.

[0056] An image may depict one or more objects, which is indicated in the input dataset by the use of class labels representing individual images or objects mapped onto those images.

[0057] Next, the images are processed separately by a second machine learning algorithm and a third machine learning algorithm in step S12. Step S12 includes several substeps that map the internal processes.

[0058] In step S12, once all images have been processed, in step S14, the images, the cutouts generated from those images, and the labels generated for those cutouts are combined into a single training dataset, which is then supplied for training the first machine learning algorithm.

[0059] The following describes the internal processes performed by the second and third machine learning algorithms in these embodiments. These internal processes are performed on each image in the input dataset, and therefore the set of images supplied for training the first machine learning algorithm is not reduced.

[0060] From each image, in step S12.1, a cutout is first generated showing the object and any elements that are not important to or even detrimental to label generation. If the image shows, for example, traffic conditions and the first machine learning algorithm is trained to identify traffic signs from camera images, areas of the image that do not show traffic signs can be removed. In such images, there are typically empty areas, road areas, and road surrounds that are not necessary or, at best, unnecessary in terms of context for finding a suitable label.

[0061] In step S12.2, the generated cutouts, images, and natural language instructions, or prompts, are input to a second machine learning algorithm. The prompts include instructions for the second machine learning algorithm, which form context for the general task of generating labels for image cutouts and include specific requests for generating labels for the input cutouts.

[0062] Context is important for the second machine learning algorithm because it sets what kind of labels should be generated. If the cutout, for example, represents a traffic sign, the context can indicate that the second machine learning algorithm should output the type of traffic sign as the label.

[0063] In step S12.3, labels are generated in response to instructions from the prompt. These labels can essentially be used for the training set or training of the first machine learning algorithm. Test results have also shown that when a third machine learning algorithm examines the generated labels, the labels become more accurate.

[0064] For validation, the original image, the generated cutout, the original label, the label generated for the cutout, and, if applicable, the rationale for the generated label created by a second machine learning algorithm, are input to a third machine learning algorithm. The third machine learning algorithm performs validation, for example, by comparing the original label with the generated label and performing a semantic evaluation. In this case, graphic information from the cutout, graphic information from the context of the original image, and the rationale can all be considered.

[0065] If the third machine learning algorithm validates the generated label, i.e., if it evaluates it to be better than the original label, the inner process restarts from the beginning with the next image. If the generated label could not be validated, another label is generated in step S12.5. Optionally, this other label can also be validated, in which case preferably the machine learning algorithm that did not generate the other label performs the validation.

[0066] Figure 2 shows a system 100 for generating a dataset for training and / or validating a first machine learning algorithm.

[0067] The system includes a supply unit 102 configured to supply an input dataset S10, which includes a set of multiple images, where objects to be classified are mapped onto the images, and each image is associated with at least one class having a class label.

[0068] The system further performs S12 for each image. d. Generate an image cutout containing the object to be classified in S12.1. e. The image and the generated cutout are added to the second machine learning algorithm in S12.2. f. Generate labels for cutouts It includes a computing unit 104 configured as such.

[0069] Furthermore, the system includes an output unit (106) configured to supply a dataset (S14) containing a set of multiple images, each associated with a generated label.

Claims

1. A computer-implemented method for generating a dataset for training and / or validating a first machine learning algorithm, The above method involves the following steps, namely: - Step (S10) of supplying an input dataset including a set of multiple images, wherein objects to be classified are mapped onto the images, and each image is associated with at least one class having a class label, - For each image (S12), - Generate a cutout of the image containing the object to be classified (S12.1), - The image and the generated cutout are input into the second machine learning algorithm (S12.2), - Generate a label for the cutout. Steps and - Step (S14) of supplying a dataset that includes a set of multiple images, and each image is associated with a corresponding label that has been generated. Methods that include...

2. The computer-implemented method according to claim 1, wherein the second machine learning algorithm includes a visual language model.

3. The computer-implemented method according to claim 1 or 2, wherein the instructions for outputting the label are written in natural language.

4. The computer-implemented method according to any one of claims 1 to 3, wherein the label is selected from a list consisting of synonyms or subcategories of the first label.

5. The above method further, - A step (S12.4) in which each image is input to a third machine learning algorithm along with an instruction to verify the label, - Steps to verify the generated labels, A computer-implemented method according to any one of claims 1 to 4, including the method described in any one of claims 1 to 4.

6. The computer-implemented method according to claim 5, wherein if the output of the third machine learning algorithm indicates that a pre-generated label has not been validated, another label is generated and the other label replaces the pre-generated label.

7. The computer-implemented method according to claim 5 or 6, wherein the verification includes comparing the label generated by the third machine learning algorithm with the other label.

8. The computer-implemented method according to claim 6 or 7, wherein the label generates a natural language text basis for the label, and the verification of the label is performed using the basis.

9. The computer-implemented method according to any one of claims 1 to 8, wherein the first machine learning algorithm includes an algorithm for identifying traffic signs, road surfaces and lane integrity, pedestrians and / or vehicles.

10. A computer program comprising program code for carrying out the method described in any one of claims 1 to 9 when the computer program is executed on a computer.

11. A computer-readable data carrier comprising program code of a computer program for carrying out the method according to any one of claims 1 to 9 when the computer program is executed on a computer.

12. A system (100) for generating a dataset for training and / or validating a first machine learning algorithm, The aforementioned system, - A supply unit (102) configured to supply an input dataset (S10) which includes a set of multiple images, wherein objects to be classified are mapped onto the images, and each image is associated with at least one class having a class label, - For each image (S12), a. Generate a cutout of the image containing the object to be classified (S12.1), b. The image and the generated cutout are added to the second machine learning algorithm (S12.2), c. Generate a label for the cutout. A computing unit (104) configured as follows, - An output unit (106) is configured to supply a dataset (S14) which includes a set of multiple images, and each image is associated with a corresponding label that has been generated. A system (100) comprising the above.

Images