Method for generating training data for training machine learning models and apparatus for generating training data using the same
The combination of vision language and foundation models addresses the issue of inaccurate auto-labeling by generating high-quality training data through captioning and filtering, facilitating efficient and cost-effective dataset creation.
Patent Information
- Application Number
- JP2025151406
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2025-08-25
- Filing Date
- 2025-09-11
- Publication Date
- 2026-05-18
- Estimated Expiration
- 2045-09-11
AI Technical Summary
Conventional auto-labeling methods for generating training data in machine learning models often result in inaccurate initial labeling due to small or low-quality initial data, leading to the accumulation of false positives and noise, especially in large-scale and specialized domains, which can lead to incorrect pattern learning and are difficult to manually verify.
A method combining a vision language model and a vision foundation model to generate training data through captioning, open vocabulary object detection, and filtering, using alignment scores and bounding box criteria to create high-quality pseudo-labels, thereby preventing incorrect label accumulation.
Enables the rapid construction of large-scale, high-quality multi-domain datasets without human intervention, reducing costs and time, and ensuring accurate label generation.
Smart Images
Figure 0007860647000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for generating training data for training a machine learning model and a training data generation device using the same, and more particularly, to a method for generating training data through auto-labeling by combining a vision language model (VLM) and a vision foundation model (VFM), and a training data generation device using the same.
Background Art
[0002] Generally, labeling methods for generating training data include a manual labeling method in which a labeler manually generates a label for an image, and an auto-labeling method in which a learned machine learning model automatically generates a label for an image. Recently, with the improvement in the performance of machine learning models, the auto-labeling method is widely used for generating training data.
[0003] The process of generating training data using the auto-labeling method involves first training an object detection model such as a VLM (Vision Language Model) or VFM (Vision Foundation Model) using a small amount of manually labeled, high-quality training data. Then, objects are predicted on an unlabeled dataset through the trained object detection model. In other words, pseudo-labels or auto-labels are generated by automatically assigning information such as object location and class to unlabeled images using the trained object detection model (VLM or VFM), and these pseudo-labels or auto-labels are annotated onto the unlabeled dataset to generate the training dataset. The training dataset generated by auto-labeling is then used again to train the object detection model, and the quality of the labels is improved as needed through quality verification and misslabel detection algorithms. By repeating this process, the object detection model that performs auto-labeling can be applied to more data and diverse environments, and it will evolve on its own.
[0004] However, with such conventional auto-labeling methods, if the initial manually labeled training data is small or of poor quality, the initial auto-labeling results will be inaccurate. In subsequent iterative training, false positives / failures of the object detection model will accumulate, and incorrect pseudo-labels or auto-labels may be included in the iterative training data. In this case, there is a risk that the object detection model will learn incorrect patterns. In large-scale auto-labeling processes, mislabels and noise are easily introduced, and it is difficult to verify them one by one manually. Furthermore, in specialized domains such as industrial settings, there are many details and rare events that general VLM or VFM may miss, requiring additional quality control.
[0005] Therefore, the applicant seeks to propose a novel method for constructing large-scale, high-quality training datasets. [Overview of the project] [Problems that the invention aims to solve]
[0006] The purpose of this invention is to solve all of the problems of the prior art described above.
[0007] Another objective of the present invention is to enable the generation of large-scale, high-quality training data by combining a vision language model and a vision foundation model.
[0008] Furthermore, another objective of the present invention is to enable the prevention of the accumulation of false labels through filtering and validation of the outputs of the vision language model and the vision foundation model.
[0009] Furthermore, the present invention aims to enable the rapid construction of high-quality, large-scale multi-domain datasets without human intervention, thereby reducing the cost and time required for data construction in actual industrial and research settings. [Means for solving the problem]
[0010] According to one embodiment of the present invention, in a method for generating training data for training a machine learning model, (a) when a source image is acquired, a training data generation device performs captioning on the source image to generate an image caption for the source image, extracts at least one noun phrase from the image caption, performs open vocabulary object detection on the source image by referring to the at least one noun phrase to generate at least one first pseudo label including at least one first category name and at least one first bounding box, and a subprocess for generating at least one pseudo label corresponding to at least one object from the source image A method is provided which includes (b) a subprocess of extracting at least one proposal, generating at least one region description corresponding to the at least one proposal, and generating at least one second pseudolabel including at least one second category name and at least one second bounding box; and (b) the training data generation device filtering the at least one first pseudolabel and the at least one second pseudolabel according to preset filtering conditions to generate an integrated pseudolabel, and annotating the original image with the integrated pseudolabel to generate training data.
[0011] In one example, in step (b), the training data generation device calculates an image-text alignment score for each of the at least one first pseudo-label and the at least one second pseudo-label, including a first alignment score between the first crop image corresponding to the first bounding box and the first category name, and a second alignment score between the second crop image corresponding to the second bounding box and the second category name, and a first filtering condition for removing at least one pseudo-label corresponding to a specific image-text alignment score that is less than or equal to a preset threshold score from among the image-text alignment scores, at least The at least one first pseudolabel and the at least one second pseudolabel are filtered by at least one filtering condition, which includes a second filtering condition that removes at least one pseudolabel whose bounding box size is less than or equal to a preset threshold size for each of the at least one first pseudolabel and the at least one second pseudolabel, and a third filtering condition that removes at least one pseudolabel whose IOU between the first bounding box and the second bounding box is greater than or equal to a preset threshold IOU for each of the at least one first pseudolabel and the at least one second pseudolabel.
[0012] In one example, in step (b), the training data generation device calculates the image-text alignment score using one of the following models under the first filtering condition: CLIP (Contrastive Language-Image Pre-training) model, SigLIP (Sigmoid Loss for Language Image Pre-training)-2 model, GLIP (Grounded Language-Image Pre-training) model, or BLIP (Bootstrapping Language-Image Pre-training)-2 model.
[0013] In one example, in step (b), when the training data generation device filters out pseudolabels corresponding to any one of a specific first bounding box and a specific second bounding box whose IOU is greater than or equal to the preset threshold IOU, it refers to a specific first category name corresponding to the specific first bounding box and a specific second category name corresponding to the specific second bounding box to include in the integrated pseudolabel any one specific pseudolabel (any one specific pseudolabel is either the specific first bounding box or the specific second bounding box) that matches the task of the machine learning model (the machine learning model is a model that intends to learn using the training data), and removes any other specific pseudolabel corresponding to any other specific bounding box (any other specific pseudolabel is any other of the specific first bounding box or the specific second bounding box).
[0014] In one example, in step (a), when the training data generation device executes a subprocess for generating the at least one first pseudo-label, it extracts the image caption for the original image through a first vision language model, extracts the at least one noun phrase from the image caption through a natural language processing model, inputs the at least one noun phrase and the original image into an open vocabulary object detection model, and uses the open vocabulary object detection model to detect the at least one object corresponding to the at least one noun phrase from the original image.
[0015] In one example, in step (a), when the training data generation device executes a subprocess for generating the at least one second pseudo-label, it inputs the original image into an object proposal model to use the object proposal model to extract the at least one proposal corresponding to the at least one object from the original image, and inputs the at least one proposal and the original image into a second vision language model to use the second vision language model to generate the at least one region description for the at least one proposal.
[0016] According to one embodiment of the present invention, a learning data generation device for generating learning data for training a machine learning model includes: a memory storing instructions for generating learning data for training a machine learning model; and a processor that generates learning data for training a machine learning model according to the instructions stored in the memory, wherein the processor (i) when a source image is acquired, performs captioning on the source image to generate an image caption for the source image; extracts at least one noun phrase from the image caption; and performs open vocabulary object detection from the source image by referring to the at least one noun phrase to generate at least one first category name and at least one first bounding box. (II) A learning data generation device is provided that performs a subprocess for generating two first pseudo labels, and a subprocess for extracting at least one proposal corresponding to at least one object from the source image, generating at least one region description corresponding to the at least one proposal, and generating at least one second pseudo label including at least one second category name and at least one second bounding box, and (II) a process for filtering the at least one first pseudo label and the at least one second pseudo label according to a preset filtering condition to generate an integrated pseudo label, and annotating the source image with the integrated pseudo label to generate learning data.
[0017] In one example, the processor calculates an image-text alignment score for each of the at least one first pseudo-label and the at least one second pseudo-label, including a first alignment score between the first crop image corresponding to the first bounding box and the first category name, and a second alignment score between the second crop image corresponding to the second bounding box and the second category name, and a first filtering condition for removing at least one pseudo-label corresponding to a specific image-text alignment score that is less than or equal to a preset threshold score from among the image-text alignment scores, The at least one first pseudolabel and the at least one second pseudolabel are filtered by at least one filtering condition, selected from a second filtering condition that removes at least one pseudolabel whose bounding box size is less than or equal to a preset threshold size, and a third filtering condition that removes at least one pseudolabel whose IOU between the first bounding box and the second bounding box is greater than or equal to a preset threshold IOU, for each of the at least one first pseudolabel and the at least one second pseudolabel.
[0018] In one example, the processor calculates the image-text alignment score in the (II) process using one of the following models in the first filtering condition: CLIP (Contrastive Language-Image Pre-training) model, SigLIP (Sigmoid Loss for Language Image Pre-training)-2 model, GLIP (Grounded Language-Image Pre-training) model, or BLIP (Bootstrapping Language-Image Pre-training)-2 model.
[0019] In one example, in the (II) process, when the processor filters out pseudolabels corresponding to any one of a specific first bounding box and a specific second bounding box whose IOU is greater than or equal to the preset threshold IOU, according to the third filtering condition, it refers to a specific first category name corresponding to the specific first bounding box and a specific second category name corresponding to the specific second bounding box to include in the integrated pseudolabel any one specific pseudolabel (any one specific pseudolabel is either the specific first bounding box or the specific second bounding box) that matches the task of the machine learning model (the machine learning model is a model that intends to learn using the machine learning data), and removes any other specific pseudolabel corresponding to any other specific bounding box (any other specific pseudolabel is any other of the specific first bounding box or the specific second bounding box).
[0020] In one example, the processor, in process (I), executes a subprocess for generating the at least one first pseudo-label, extracting an image caption for the source image through a first Vision Language Model, extracting the at least one noun phrase from the image caption through a natural language processing model, inputting the at least one noun phrase and the source image into an open vocabulary object detection model, and using the open vocabulary object detection model to detect the at least one object corresponding to the at least one noun phrase from the source image.
[0021] In one example, when the processor executes a sub-process of generating the at least one second pseudo label in the (I) process, the processor inputs the original image into an object proposal model, and uses the object proposal model to extract the at least one proposal corresponding to the at least one object from the original image, inputs the at least one proposal and the original image into a second vision language model, and uses the second vision language model to generate the at least one region description for the at least one proposal.
Advantages of the Invention
[0022] According to the present invention, by combining a vision language model and a vision foundation model, it is possible to generate large-scale and high-quality learning data.
[0023] According to the present invention, it is possible to prevent incorrect labels from accumulating through filtering and verification of the outputs of the vision language model and the vision foundation model.
[0024] According to the present invention, it is possible to quickly construct a high-quality and large-scale multi-domain dataset without manual intervention, and reduce the costs and time required for data construction in actual industrial and research sites.
Brief Description of the Drawings
[0025] The following drawings attached for use in the description of the embodiments of the present invention are only a part of the embodiments of the present invention, and those with ordinary knowledge in the technical field to which the present invention belongs (hereinafter referred to as "ordinary technicians") can obtain other drawings based on these drawings without performing inventive work.
[0026] [Figure 1]It is a diagram schematically showing a learning data generation device for generating learning data for learning a machine learning model according to an embodiment of the present invention. [Figure 2] It is a diagram schematically showing a method for generating learning data for learning a machine learning model according to an embodiment of the present invention. [Figure 3] It is a diagram exemplarily showing a first pseudo label and a second pseudo label generated for a source image in a method for generating learning data for learning a machine learning model according to an embodiment of the present invention. [Figure 4] It is a diagram schematically showing a state where first filtering is performed using an image - text alignment score from the first pseudo label and the second pseudo label in FIG. 3 in a method for generating learning data for learning a machine learning model according to an embodiment of the present invention. [Figure 5] It is a diagram schematically showing a state where second filtering is performed using a bounding box size from the first pseudo label and the second pseudo label in FIG. 3 in a method for generating learning data for learning a machine learning model according to an embodiment of the present invention. [Figure 6] It is a diagram schematically showing a state where third filtering is performed using IOU from the first pseudo label and the second pseudo label in FIG. 3 in a method for generating learning data for learning a machine learning model according to an embodiment of the present invention. [Figure 7] It is a diagram schematically showing an integrated pseudo label obtained by performing first to third filtering from the first pseudo label and the second pseudo label in FIG. 3 in a method for generating learning data for learning a machine learning model according to an embodiment of the present invention.
Modes for Carrying Out the Invention
[0027] The detailed description of the present invention, as described below, refers to the accompanying drawings illustrating specific embodiments in which the present invention may be carried out. These embodiments are described in sufficient detail to enable a person of the ordinary skill to carry out the present invention. It should be understood that the various embodiments of the present invention are different from one another but do not need to be mutually exclusive. For example, certain shapes, structures and characteristics described herein can be realized by modifying one embodiment to another without departing from the spirit and scope of the present invention. It should also be understood that the position or arrangement of individual components within each embodiment can be modified without departing from the spirit and scope of the present invention. Therefore, the detailed description described below should not be taken as restrictive, and the scope of the present invention should be understood to encompass the scope claimed in the claims and all equivalent scopes thereto. In the drawings, similar reference numerals indicate identical or similar components in various aspects.
[0028] In the following, several preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that a person with ordinary skill in the art to which the present invention pertains can easily implement the present invention.
[0029] Figure 1 is a schematic diagram of a learning data generation device that generates training data for training a machine learning model according to one embodiment of the present invention. Referring to Figure 1, the learning data generation device 1000 may include a memory 1100 that stores instructions for generating training data for training a machine learning model, and a processor 1200 that generates training data for training a deep learning model according to the instructions stored in the memory 1100.
[0030] Specifically, the learning data generation device 1000 may, but is not limited to, achieve desired system performance by utilizing a combination of a typical computing device (e.g., a device that may include computer processors, memory, storage, input and output devices, and other conventional computing device components; electronic communication devices such as routers and switches; and electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that enable the computing device to function in a particular manner).
[0031] Furthermore, the processor 1200 of the learning data generation device 1000 may include hardware components such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and a data bus. The computing device may also further include an operating system and software components for applications that perform a specific purpose.
[0032] However, this does not exclude the case in which the learning data generation device 1000 includes an integrated processor in which a medium, processor, and memory for carrying out the present invention are integrated.
[0033] On the other hand, the processor 1200 of the learning data generation device 1000 can execute a process in accordance with the instructions stored in the memory 1100. When a source image is acquired, it can execute a subprocess that performs captioning on the source image to generate an image caption for the source image, extracts at least one noun phrase from the image caption, and performs open vocabulary object detection on the source image by referring to at least one noun phrase to generate at least one first pseudo label including at least one first category name and at least one first bounding box. It can also execute a subprocess that extracts at least one proposal corresponding to at least one object from the source image, generates at least one region description corresponding to at least one proposal, and generates at least one second pseudo label including at least one second category name and at least one second bounding box. The processor 1200 of the learning data generation device 1000 can then execute a process to generate training data by filtering at least one first pseudolabel and at least one second pseudolabel according to the instructions stored in the memory 1100 and the pre-set filtering conditions, generating an integrated pseudolabel, and annotating the original image with the integrated pseudolabel.
[0034] The method for generating training data for training a machine learning model in the training data generation device 1000 configured in this way will be explained in more detail with reference to Figure 2, as follows.
[0035] First, the training data generation device 1000 can acquire the source images necessary for generating training data (S100).
[0036] For example, the source image may be obtained from an unlabeled dataset containing unlabeled images collected for the generation of training data. In this case, the unlabeled dataset may be stored in a storage device that works in conjunction with the training data generation device 1000 or in cloud storage, but the present invention is not limited thereto, and may be stored in various storage environments capable of recording data.
[0037] The learning data generation device 1000 can then perform captioning on the original image (S211) to generate image captions for the original image.
[0038] As an example, the training data generation device 1000 can extract image captions for the original images through a first vision-language model. In this case, the image caption can be assumed to be, for example, "there is a lot of man playing an acoustic guitar," which is used to derive the category names "man" and "acoustic guitar" corresponding to the solid-line frame in Figure 3. The first vision-language model can include CLIP (Contrastive Language-Image Pre-training), BLIP (Bootstrapping Language-Image Pre-training), InternVL (large-scale vision-language foundation model), etc., but the present invention is not limited thereto and can include various multimodal models for performing image captioning.
[0039] Subsequently, the learning data generation device 1000 can extract at least one noun phrase from the image caption (S212).
[0040] As an example, the learning data generation device 1000 can extract at least one noun phrase from an image caption through a natural language processing model, and the extracted noun phrase may be a coarse-grained noun phrase. In this case, the extracted noun phrase may be, for example, "man" or "acoustic guitar" corresponding to the solid line frame in Figure 3. The natural language processing model may include SpaCy, WordNet, etc., but the present invention is not limited to these and may include various models for processing natural language. Furthermore, the noun phrase may include nouns representing objects, object attributes such as color, shape, and material.
[0041] The learning data generation device 1000 can then perform a subprocess to generate at least one first pseudo label (S214) that includes at least one first category name and at least one first bounding box by referencing at least one noun phrase and performing open vocabulary object detection from the original image (S213). In this case, the first category name and the first bounding box may be, for example, the text and solid frame corresponding to the solid frame in Figure 3.
[0042] As an example, the learning data generation device 1000 inputs at least one noun phrase and the original image into an open vocabulary object detection model, and uses the open vocabulary object detection model to detect at least one object corresponding to at least one noun phrase from the original image, that is, to predict the position of at least one object corresponding to at least one noun phrase, for example, the bounding box, thereby generating at least one first pseudo-label that includes at least one first category name which is at least one noun phrase and at least one corresponding first bounding box. In this case, the open vocabulary object detection model may include at least a part of a Grounding-DINO model, a zero-shot model, etc., but the present invention is not limited thereto and may include various models that perform object detection in an open vocabulary manner.
[0043] Furthermore, the training data generation device 1000, which has acquired the original image (S100), can extract at least one proposal corresponding to at least one object from the original image (S221).
[0044] As an example, the training data generation device 1000 can input the original image into an object proposal model and use the object proposal model to extract at least one proposal corresponding to at least one object from the original image. In this case, the proposal may be, for example, the dotted frame in Figure 3. The object proposal model may include a UPN (Universal Proposal Network), and the UPN can extract coarse-grained proposals corresponding to instance-level objects and fine-grained proposals corresponding to part-level objects.
[0045] The learning data generation device 1000 can then execute a subprocess to generate at least one second pseudo-label (S223) that includes at least one second category name and at least one second bounding box, by generating at least one region description (S222) corresponding to at least one proposal. In this case, the region description may be, for example, "old man" or "brown acoustic guitar" corresponding to the dotted frame in Figure 3, or it may be a description of an object on the region corresponding to the proposal.
[0046] As an example, the training data generation device 1000 can input at least one proposal and a source image into a second vision language model and use the second vision language model to generate at least one region description for at least one proposal, thereby generating at least one second pseudo-label that includes at least one bounding box which is at least one proposal and at least one second category name which is at least one region description corresponding to it. In this case, the second vision language model may include a ChatRex model. The region description output from ChatRex may be a fine-grained phrase, which is a more detailed and contextual sentence, for each proposal.
[0047] Once at least one first pseudo-label and at least one second pseudo-label are generated for the original image in the same manner as described above, the training data generation device 1000 can filter the at least one first pseudo-label and at least one second pseudo-label according to preset filtering conditions to generate an integrated pseudo-label (S300).
[0048] At this time, the learning data generation device 1000 calculates an image-text alignment score for each of at least one first pseudo-label and at least one second pseudo-label, which includes a first alignment score between the first crop image corresponding to the first bounding box and the first category name, and a second alignment score between the second crop image corresponding to the second bounding box and the second category name, for example, the first alignment score indicating how well the first category name describes the object on the first crop image, and the second alignment score indicating how well the second category name describes the object on the second crop image, and selects from the image-text alignment scores that are less than or equal to a preset threshold score. At least one of the following filtering conditions can be used to filter at least one first pseudolabel and at least one second pseudolabel: a first filtering condition that removes at least one pseudolabel corresponding to a specific image-text alignment score; a second filtering condition that removes at least one pseudolabel for each of the at least one first pseudolabel and at least one second pseudolabel whose bounding box size is less than or equal to a preset threshold size; and a third filtering condition that removes at least one pseudolabel for each of the at least one first pseudolabel and at least one second pseudolabel whose IOU between the first bounding box and the second bounding box is greater than or equal to a preset threshold IOU.
[0049] As an example, referring to Figures 3 to 7, the process by which the learning data generation device 1000 filters at least one first pseudolabel and at least one second pseudolabel according to the first to third filtering conditions to generate an integrated pseudolabel is described below. For reference, in Figures 3 to 7, solid lines may represent the first pseudolabel, and dotted lines may represent the second pseudolabel.
[0050] As shown in Figure 3, with the original image automatically labeled and a first pseudo-label and a second pseudo-label generated, as shown in Figure 4, the training data generator 1000 calculates an image-text alignment score for each of the first and second pseudo-labels through one of the following models: CLIP (Contrastive Language-Image Pre-training) model, SigLIP (Sigmoid Loss for Language Image Pre-training)-2 model, GLIP (Grounded Language-Image Pre-training) model, or BLIP (Bootstrapping Language-Image Pre-training)-2 model, according to the first filtering condition. The pseudo-labels "partially visible car" (11), "small wooden table" (12), "black sandals" (13), and "black and white sneakers" (14) can be removed if their image-text alignment scores are below a set threshold score.
[0051] Then, as shown in Figure 5, the learning data generation device 1000 can check the size of the bounding boxes of the first pseudo-label and the second pseudo-label according to the second filtering condition, and can remove the pseudo-label "mouth" (21) whose bounding box size is confirmed to be less than or equal to the threshold size.
[0052] Subsequently, as shown in Figure 6, the learning data generator 1000 checks for pairs of first and second pseudo-labels that overlap with each other, i.e., whose bounding box IOUs are greater than or equal to a set threshold IOU, such as "acoustic guitar" (31) and "brown acoustic guitar" (31'), and "man" (32) and "older man" (32'), and removes the pseudo-labels for "brown acoustic guitar" (31') and "older man" (32'). As shown in Figure 7, the learning data generator 1000 can then generate an integrated pseudo-label that has been filtered to remove at least one first pseudo-label and at least one second pseudo-label.
[0053] At this time, the learning data generation device 1000, in accordance with the third filtering condition, filters out pseudolabels corresponding to any one of a specific first bounding box and a specific second bounding box whose IOU is greater than or equal to a set threshold IOU. In doing so, it can refer to a specific first category name corresponding to a specific first bounding box and a specific second category name corresponding to a specific second bounding box to include one specific pseudolabel corresponding to any one specific bounding box that matches the task of the machine learning model to be learned into an integrated pseudolabel (the one specific pseudolabel is any one of the specific first bounding box and the specific second bounding box), and remove another specific pseudolabel corresponding to another specific bounding box (the other specific pseudolabel is any other one of the specific first bounding box and the specific second bounding box).
[0054] In other words, as explained with reference to Figure 6, in the case of the pairs of first and second pseudo-labels that overlap each other, namely "acoustic guitar" (31) and "brown acoustic guitar" (31'), and "man" (32) and "older man" (32'), if the machine learning model to be trained simply needs the object's class, the second pseudo-labels "brown acoustic guitar" (31') and "older man" (32') can be removed and the first pseudo-labels "acoustic guitar" (31) and "man" (32) can be included in the unified pseudo-label. If the machine learning model to be trained needs the object's attributes, the first pseudo-labels "acoustic guitar" (31) and "man" (32) can be removed and the second pseudo-labels "brown acoustic guitar" (31') and "older man" (32') can be included in the unified pseudo-label.
[0055] Furthermore, referring to Figure 2, the training data generation device 1000 can generate training data (S400) by annotating the original image with integrated pseudo-labels.
[0056] Subsequently, the model used to generate the training data can be retrained using the generated training data, and the process of auto-labeling the newly acquired unlabeled dataset can be repeated.
[0057] The embodiments of the present invention described above are implemented in the form of program instructions that can be executed through various computer components and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or they may be known and available to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.
[0058] Although the present invention has been described above with specific details such as concrete components, and with limited embodiments and drawings, these are provided only to aid in a more overall understanding of the invention, and the invention is not limited to the above embodiments. A person with ordinary skill in the art to which the invention pertains can make various modifications and variations from this description.
[0059] Therefore, the concept of the present invention should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims described below shall also fall within the scope of the concept of the present invention. [Explanation of Symbols]
[0060] 1000 Training Data Generator 1100 memory 1200 processors
Claims
1. In a method for generating training data for training a machine learning model, (a) When a source image is acquired, the learning data generation device performs a subprocess to generate an image caption for the source image by performing captioning on the source image, extract at least one noun phrase from the image caption, perform open vocabulary object detection on the source image by referring to the at least one noun phrase to generate at least one first pseudo (pseudo) label including at least one first category name and at least one first bounding box, and extract at least one proposal corresponding to at least one object from the source image, generate at least one region description corresponding to the at least one proposal to generate at least one second pseudo (pseudo) label including at least one second category name and at least one second bounding box, (b) The learning data generation device generates a learning data by filtering the at least one first pseudolabel and the at least one second pseudolabel according to a preset filtering condition to generate an integrated pseudolabel, and annotating the original image with the integrated pseudolabel. A method that includes this.
2. In step (b) above, The training data generation device calculates an image-text alignment score for each of the at least one first pseudo-label and the at least one second pseudo-label, including a first alignment score between the first crop image corresponding to the first bounding box and the first category name, and a second alignment score between the second crop image corresponding to the second bounding box and the second category name, and a first filtering condition for removing at least one pseudo-label corresponding to a specific image-text alignment score that is less than or equal to a preset threshold score from the image-text alignment scores, and the at least one first pseudo-label and The method according to claim 1, wherein the at least one first pseudo-label and the at least one second pseudo-label are filtered by at least one filtering condition from among a second filtering condition that removes at least one pseudo-label whose bounding box size is less than or equal to a preset threshold size for each of the at least one second pseudo-label, and a third filtering condition that removes at least one pseudo-label whose IOU between the first bounding box and the second bounding box is greater than or equal to a preset threshold IOU for each of the at least one first pseudo-label and the at least one second pseudo-label.
3. In step (b) above, The method according to claim 2, wherein the training data generation device calculates the image-text alignment score through one of the following models in the first filtering condition: CLIP (Contrastive Language-Image Pre-training) model, SigLIP (Sigmoid Loss for Language Image Pre-training)-2 model, GLIP (Grounded Language-Image Pre-training) model, and BLIP (Bootstrapping Language-Image Pre-training)-2 model.
4. In step (b) above, The method according to claim 2, wherein the learning data generation device, in accordance with the third filtering condition, filters pseudolabels corresponding to any one of a specific first bounding box and a specific second bounding box whose IOU is greater than or equal to the preset threshold IOU, by referring to a specific first category name corresponding to the specific first bounding box and a specific second category name corresponding to the specific second bounding box, and includes in the integrated pseudolabel any one specific pseudolabel corresponding to any one specific bounding box necessary for the task of the machine learning model (the machine learning model is a model that intends to learn using the learning data) (the one specific pseudolabel is any one of the specific first bounding box and the specific second bounding box), and removes another specific pseudolabel corresponding to another specific bounding box (the other specific pseudolabel is any other one of the specific first bounding box and the specific second bounding box).
5. In step (a) above, The method according to claim 1, wherein the learning data generation device, when executing a subprocess for generating the at least one first pseudo-label, extracts the image caption for the original image through a first vision language model, extracts the at least one noun phrase from the image caption through a natural language processing model, inputs the at least one noun phrase and the original image into an open vocabulary object detection model, and uses the open vocabulary object detection model to detect the at least one object corresponding to the at least one noun phrase from the original image.
6. In step (a) above, The method according to claim 1, wherein the training data generation device, when executing a subprocess for generating the at least one second pseudo-label, inputs the original image into an object proposal model, and uses the object proposal model to extract the at least one proposal corresponding to the at least one object from the original image, and inputs the at least one proposal and the original image into a second vision language model, and uses the second vision language model to generate the at least one region description for the at least one proposal.
7. In a training data generation device that generates training data for training machine learning models, Memory containing instructions for generating training data for training a machine learning model, A processor that generates training data for training a machine learning model according to the instructions stored in the memory, Includes, The processor is a learning data generation device that performs the following processes: (I) when a source image is acquired, it performs captioning on the source image to generate an image caption for the source image, extracts at least one noun phrase from the image caption, performs open vocabulary object detection on the source image by referring to the at least one noun phrase to generate at least one first pseudo (pseudo) label including at least one first category name and at least one first bounding box; and extracts at least one proposal corresponding to at least one object from the source image, generates at least one region description corresponding to the at least one proposal to generate at least one second pseudo label including at least one second category name and at least one second bounding box; and (II) filters the at least one first pseudo label and the at least one second pseudo label according to a preset filtering condition to generate an integrated pseudo label, and annotates the source image with the integrated pseudo label to generate learning data.
8. The aforementioned processor, In the process described in (II) above, for each of the at least one first pseudolabel and the at least one second pseudolabel, an image-text alignment score is calculated including a first alignment score between the first crop image corresponding to the first bounding box and the first category name, and a second alignment score between the second crop image corresponding to the second bounding box and the second category name, and a first filtering condition is used to remove at least one pseudolabel corresponding to a specific image-text alignment score that is less than or equal to a preset threshold score from the image-text alignment scores, the at least one first pseudolabel and the The learning data generation device according to claim 7, wherein the at least one first pseudo-label and the at least one second pseudo-label are filtered by at least one filtering condition from among a second filtering condition that removes at least one pseudo-label whose bounding box size is less than or equal to a preset threshold size for each of the at least one second pseudo-label, and a third filtering condition that removes at least one pseudo-label whose IOU between the first bounding box and the second bounding box is greater than or equal to a preset threshold IOU for each of the at least one first pseudo-label and the at least one second pseudo-label.
9. The aforementioned processor, The learning data generation device according to claim 8, wherein in the process (II) above, the image-text alignment score is calculated in the first filtering condition using one of the following models: CLIP (Contrastive Language-Image Pre-training) model, SigLIP (Sigmoid Loss for Language Image Pre-training)-2 model, GLIP (Grounded Language-Image Pre-training) model, or BLIP (Bootstrapping Language-Image Pre-training)-2 model.
10. The aforementioned processor, In the process of (II) described above, when filtering pseudolabels corresponding to any one of a specific first bounding box and a specific second bounding box whose IOU is greater than or equal to the preset threshold IOU, according to the third filtering condition, the learning data generation apparatus according to claim 8, includes in the integrated pseudolabel any one specific pseudolabel corresponding to any one specific bounding box necessary for the task of the machine learning model (the machine learning model is a model that intends to learn using machine learning data) (the one specific pseudolabel is any one of the specific first bounding box and the specific second bounding box), and removes another specific pseudolabel corresponding to another specific bounding box (the other specific pseudolabel is any other of the specific first bounding box and the specific second bounding box).
11. The aforementioned processor, The learning data generation device according to claim 7, wherein in the process of (I) above, when executing a subprocess for generating the at least one first pseudo-label, the device extracts an image caption for the original image through a first Vision Language Model, extracts the at least one noun phrase from the image caption through a natural language processing model, inputs the at least one noun phrase and the original image into an open vocabulary object detection model, and uses the open vocabulary object detection model to detect the at least one object corresponding to the at least one noun phrase from the original image.
12. The aforementioned processor, The learning data generation apparatus according to claim 7, wherein in the process (I) above, when executing the subprocess for generating the at least one second pseudo-label, the original image is input to an object proposal model so that the object proposal model can extract the at least one proposal corresponding to the at least one object from the original image, and the at least one proposal and the original image are input to a second vision language model so that the second vision language model can generate the at least one region description for the at least one proposal.