A method for selecting training data for training a deep learning model and a training data selection device using the same.

Abstract

This provides a method for selecting training data for training deep learning models. [Solution] The method includes the steps of: a learning data selection device generating at least one individual type corresponding to at least one object contained in each of a large number of learning images stored in a data pool, and generating a binary graph matching each of the large number of learning images with the individual type; and a learning data selection device referring to the binary graph and using an optimization algorithm to select a specific subset with the fewest number of learning images from a subset consisting of a predetermined number of learning images containing all the individual types, calculating the remaining learning images by removing a specific number of learning images contained in the specific subset, and repeating the process of selecting at least one other specific subset consisting of a predetermined number of learning images containing all the individual types from the remaining learning images using an optimization algorithm until n learning images have been selected.

Description

[Technical Field]

[0001] The present invention relates to a method for uniformly selecting training data for training a deep learning model from all training data stored in a data pool, without bias or variability in the data, and a training data selection device utilizing this method. [Background technology]

[0002] Generally, deep learning models recognize complex patterns in images, text, sound, and other data to generate accurate insights and predictions, and are applied in various fields such as computer vision, speech recognition, autonomous vehicles, robotics, natural language processing, and medical image analysis.

[0003] In order for such deep learning models to accurately perform their intended tasks, they must be trained using a large amount of training data.

[0004] Conventional methods for selecting training data for training deep learning models from a collected data pool include random sampling, which selects a target number of training data from all the training data stored in the data pool, and vector quantization, which clusters and groups the vectors representing each of the training data generated by embedding extraction, and then selects representative values ​​for each group of grouped vectors.

[0005] For example, Patent Document 1 discloses a method and apparatus for generating learning data required to train an animated character based on deep learning, and Patent Document 2 discloses a similarity-based clustering apparatus and method utilizing deep learning learning techniques. Furthermore, Patent Document 3 discloses a training apparatus and method for a deep learning classification model, and Patent Document 4 discloses a system and method for training a machine learning model using active learning.

[0006] However, conventional methods for selecting training data have the problem of bias and variability in data types.

[0007] For example, if a data pool contains 1 million training images, with 70% related to sunny weather, 20% related to cloudy weather, 5% related to foggy weather, and 5% related to snowy and / or rainy weather, then randomly sampling 10,000 training images would result in only about 500 images being selected from a total of 50,000 images related to snowy and / or rainy weather. This would lead to a bias and variability in the selection of training images based on weather type.

[0008] Furthermore, while using vector quantization to select training images can somewhat mitigate the bias and variability in the types of training images selected by embedding extraction and clustering, it cannot fundamentally prevent problems related to data bias and variability.

[0009] Therefore, the applicant aims to propose a method that enables the selection of training data for training a deep learning model from all training data stored in a data pool, categorized by type and selected evenly without bias or variability. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] U.S. Patent Publication No. 11106942 [Patent Document 2] Korean Published Patent No. 10-2023-0068941 [Patent Document 3] Patent No. 7225614 [Patent Document 4] U.S. Patent Publication No. 11663409 [Overview of the project] [Problems that the invention aims to solve]

[0011] The purpose of this invention is to solve all of the problems described above.

[0012] Another objective of this invention is to enable the selection of training data for training a deep learning model from a data pool, without bias towards any particular type, and to allow for even selection by type.

[0013] Another objective of this invention is to enable the equal selection of training data from a data pool that is suitable for various scenarios according to the type of training data. [Means for solving the problem]

[0014] According to one embodiment of the present invention, a method for selecting training data for training a deep learning model is provided, comprising the steps of: (a) a training data selection device generating at least one attribute corresponding to each at least one object contained in each of a large number of training images stored in a data pool, and generating a binary graph matching each of the large number of training images with the attribute; and (b) the training data selection device (i) referring to the binary graph, through an optimization algorithm, selects a specific subset of training images consisting of a predetermined number of training images containing all the attribute, which has the fewest number of training images, calculates the remaining training images excluding a specific number of training images contained in the specific subset, and (ii) repeating the process of selecting at least one other specific subset of the remaining training images consisting of a predetermined number of training images containing all the attribute, through the optimization algorithm, until n training images for training the deep learning model (where n is the target number of training images for training the deep learning model, and is an integer representing a plurality of numbers) are selected.

[0015] In one example, in step (b) above, the learning data selection device uses linear programming. Using programming, the process of selecting at least one other specific subset from the selection goodness-of-fit variables of the Q learning images in each of the P individual types and Q learning images in the binary graph is repeated, and the product of the matrix between the PxQ binary matrix corresponding to the P individual types and Q learning images in the binary graph and the Q-dimensional vector representing the selection goodness-of-fit variable for each of the Q learning images in each of the P individual types is calculated to generate a P-dimensional vector (the P-dimensional vector represents the sum of the goodness-of-fit variables for the Q learning images belonging to each of the P individual types) is calculated so that the sum of the goodness-of-fit variables in the P-dimensional vector is 1 or more, and the selection goodness-of-fit variables in the Q-dimensional vector are between 0 and 1. A specific subset is selected from the selection goodness-of-fit variables of the Q-dimensional vector that have the minimum sum of the selection goodness-of-fit variables. The remaining learning images are calculated by removing the specific learning images included in the specific subset from the Q learning images. The process of selecting at least one other specific subset from the remaining learning images using linear programming is repeated so that the number of selected learning images is n or more.

[0016] In one example, in step (b), the training data selection device selects the particular subset using a dual linear programming method which applies at least one of the following constraints in linear programming: merging, separating, and sign changing.

[0017] In one example, in the step (b), the learning data selection device uses integer programming to calculate the product of a PxQ binary matrix corresponding to P individual types and Q learning images in the bipartite graph and a Q-dimensional vector representing the selection variables of each of the Q learning images in each of the P individual types, and generates a P-dimensional vector (the P-dimensional vector represents the selection quantity of the learning images belonging to each of the P individual types). Among the selection variables of the Q-dimensional vector that satisfy that the selection quantity in the P-dimensional vector is 1 or more and the selection variables in the Q-dimensional vector are 0 or 1, the learning data selection device selects a specific subset including a specific learning image corresponding to a specific selection variable having the minimum sum of the selection variables, calculates the remaining learning images obtained by removing the specific learning image included in the specific subset from the Q learning images, and repeats the process of selecting at least one other specific subset for the remaining learning images using the integer programming until the number of selected learning images becomes n or more.

[0018] In one example, in the step (b), the learning data selection device selects the specific subset using a dual integer programming method to which at least one of constraint merging, separation, and sign change in the integer programming is applied.

[0019] In one example, in the step (a), the learning data selection device transmits the learning images to a labeler terminal, and uses a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each object included in each of the learning images.

[0020] In one example, in the step (a), the learning data selection device executes object detection for each of the learning images to detect at least one object from each of the learning images, generates a crop image by cropping a region corresponding to the bounding box of each of the detected objects in each of the learning images, executes an embedding operation on each of the crop images to generate an object vector corresponding to each of the crop images, clusters the object vectors to generate an object cluster, and generates the individual type corresponding to the learning image by referring to the object cluster.

[0021] In one example, in the step (a), the learning data selection device generates a crop image by cropping a region corresponding to the bounding box of each object from each of the learning images by referring to the respective ground truth information included in each of the learning images, executes an embedding operation on each of the crop images to generate an object vector corresponding to each of the crop images, clusters the object vectors to generate an object cluster, and generates the individual type corresponding to the learning image by referring to the object cluster.

[0022] In one example, in the step (a), the learning data selection device checks the respective meta information included in each of the learning images, further refers to the respective shooting times included in the respective meta information, and generates the individual type corresponding to each of the learning images.

[0023] In one example, in step (b), the specific individual type includes a 1_1 specific individual type to a 1_x specific individual type corresponding to the object (where x is an integer of 1 or more), and a 2_1 specific individual type to a 2_y specific individual type corresponding to the shooting time (where y is an integer of 1 or more). The learning data selection device selects the n learning images such that the number of 1_1 specific individual types to 1_x specific individual types corresponding to the object and the number of 2_1 specific individual types to 2_y specific individual types corresponding to the shooting time that match the n learning images are within the threshold deviation, the number of 1_1 specific individual types to the number of 1_x specific individual types are within the first threshold deviation, and the number of 2_1 specific individual types to the number of 2_y specific individual types are within the second threshold deviation.

[0024] According to one embodiment of the present invention, a training data selection device for selecting training data for training a deep learning model includes: a memory storing instructions for selecting training data for training a deep learning model; and a processor that performs operations for selecting training data for training a deep learning model according to the instructions stored in the memory, wherein the processor includes (I) a process of generating at least one attribute corresponding to each of the at least one object contained in each of the numerous training images stored in a data pool, and generating a binary graph matching each of the numerous training images with the attribute, and (I I) (i) Referencing the binary graph, select a specific subset consisting of a predetermined number of training images containing all the individual types, using an optimization algorithm, to obtain the minimum number of training images, and calculate the remaining training images after removing a specific number of training images included in the specific subset; and (ii) Repeat the process of selecting at least one other specific subset consisting of a predetermined number of training images containing all the individual types from the remaining training images using the optimization algorithm, until n training images for training the deep learning model (where n is the target number of training images for training the deep learning model, and is an integer representing a number of multiple images) are selected.

[0025] In one example, the processor uses linear programming in the (II) process. Using a programming method, the following steps are performed to generate a P-dimensional vector (where the P-dimensional vector represents the sum of the fitness of the Q learning images belonging to each of the P individual types) by calculating the product of a PxQ binary matrix corresponding to the P individual types and Q learning images in the binary graph and a Q-dimensional vector representing the selection fitness variable for each of the Q learning images in each of the P individual types. The sum of the fitness of the P-dimensional vector is 1 or more, and the selection fitness variable in the Q-dimensional vector is between 0 and 1. From among the selection fitness variables of the Q-dimensional vector, a specific subset is selected that includes a specific learning image corresponding to a specific selection fitness variable whose sum of the selection fitness variables is the minimum. The remaining learning images are calculated by removing the specific learning image included in the specific subset from the Q learning images. The process of selecting at least one other specific subset from the remaining learning images using the linear programming method is repeated until the number of selected learning images is n or more.

[0026] In one example, the processor selects the particular subset in the (II) process using a dual linear programming method which applies at least one of the constraint merging, separation, and sign changing methods in linear programming.

[0027] In one example, the processor, in process (II), uses integer programming to calculate the product of a PxQ binary matrix corresponding to P individual types and Q training images in the binary graph and a Q-dimensional vector representing the selection variable for each of the Q training images in each of the P individual types to generate a P-dimensional vector (the P-dimensional vector represents the selection quantity of training images belonging to each of the P individual types) in which the selection quantity is 1 or more, and the selection variable in the Q-dimensional vector is 0 or 1, and selects a specific subset from among the selection variables of the Q-dimensional vector that has the minimum sum of the selection variables, calculates the remaining training images by removing the specific training images included in the specific subset from the Q training images, and repeats the process of selecting at least one other specific subset from the remaining training images using linear programming so that the number of selected training images is n or more.

[0028] In one example, the processor selects the particular subset in the (II) process using a dual integer programming method which applies at least one of the following constraints in integer programming: merging, separating, and sign-changing.

[0029] In one example, the processor transmits the training images to a labeler terminal in process (I) and uses a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the objects contained in each of the training images.

[0030] In one example, the processor performs object detection on each of the training images in process (I) to detect at least one object from each of the training images, generates a cropped image by cropping the region corresponding to the bounding box of each detected object in each of the training images, performs an embedding operation on each of the cropped images to generate an object vector corresponding to each of the cropped images, clusters the object vectors to generate an object cluster, and generates the individual types corresponding to the training images by referring to the object cluster.

[0031] In one example, the processor, in process (I), refers to the ground truth information contained in each of the training images to generate a cropped image by cropping the region corresponding to the bounding box of each object from each of the training images, performs an embedding operation on each of the cropped images to generate an object vector corresponding to each of the cropped images, clusters the object vectors to generate an object cluster, and refers to the object cluster to generate the individual type corresponding to the training image.

[0032] In one example, the processor, in process (I), checks the metadata contained in each of the training images, and further refers to the shooting time contained in each of the metadata to generate the individual type corresponding to each of the training images.

[0033] In one example, the processor, in process (II), selects the n training images such that the number of the 1_1 specific individual types to the 1_x specific individual types corresponding to the object (where x is an integer of 1 or more) and the number of the 2_1 specific individual types to the 2_y specific individual types corresponding to the shooting time (where y is an integer of 1 or more) that match the n training images are within the threshold deviation, the number of the 1_1 specific individual types to the 1_x specific individual types corresponding to the object and the number of the 2_1 specific individual types to the 2_y specific individual types corresponding to the shooting time are within the threshold deviation, the number of the 1_1 specific individual types to the number of the 1_x specific individual types are within the first threshold deviation, and the number of the 2_1 specific individual types to the number of the 2_y specific individual types are within the second threshold deviation. [Effects of the Invention]

[0034] This invention makes it possible to select training data for training a deep learning model from a data pool evenly by type, without bias towards any particular type.

[0035] This invention enables the uniform selection of training data from a data pool that is suitable for various scenarios depending on the type of training data. [Brief explanation of the drawing]

[0036] The following drawings, attached for use in describing embodiments of the present invention, represent only a portion of the embodiments, and a person with ordinary skill in the art to which the present invention pertains (hereinafter referred to as "ordinary art") can obtain other drawings based on these drawings without performing any inventive work.

[0037] [Figure 1] Figure 1 is a schematic diagram showing a training data selection device for selecting training data for training a deep learning model according to one embodiment of the present invention. [Figure 2]Figure 2 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to the first embodiment of the present invention. [Figure 3] Figure 3 is a schematic diagram illustrating an example of generating individual types of training data in the first embodiment of the present invention. [Figure 4] Figure 4 is a schematic diagram illustrating another example of generating individual types of training data in the first embodiment of the present invention. [Figure 5] Figure 5 is a diagram illustrating a binary graph obtained by matching each of the training data with an individual type in the first embodiment of the present invention. [Figure 6a] Figure 6a is a schematic diagram illustrating the process of selecting training data by referring to a binary graph in the first embodiment of the present invention. [Figure 6b] Figure 6b is a schematic diagram illustrating the process of selecting training data by referring to a binary graph in the first embodiment of the present invention. [Figure 6c] Figure 6c is a schematic diagram illustrating the process of selecting training data by referring to a binary graph in the first embodiment of the present invention. [Figure 6d] Figure 6d is a schematic diagram illustrating the process of selecting training data by referring to a binary graph in the first embodiment of the present invention. [Figure 7] Figure 7 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to a second embodiment of the present invention. [Figure 8] Figure 8 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to a third embodiment of the present invention. [Figure 9] Figure 9 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to a fourth embodiment of the present invention. [Modes for carrying out the invention]

[0038] The detailed description of the present invention described below refers to the accompanying drawings illustrating specific embodiments in which the present invention may be carried out. These embodiments are described in sufficient detail to enable a person of the ordinary skill to carry out the present invention. It should be understood that the various embodiments of the present invention are different from one another but do not necessarily have to be mutually exclusive. For example, certain shapes, structures and characteristics described herein can be realized in other embodiments in relation to one embodiment without departing from the spirit and scope of the invention. It should also be understood that the position or arrangement of individual components in each disclosed embodiment can be modified without departing from the spirit and scope of the invention. Therefore, the detailed description described below should not be taken as restrictive, and the scope of the present invention is limited only by the accompanying claims, along with all equivalents to those claimed by those claims, if appropriately described. In the drawings, similar reference numerals refer to identical or similar functions across various aspects.

[0039] In the following, several preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that a person with ordinary skill in the art to which the present invention pertains can easily implement the present invention.

[0040] Figure 1 schematically shows a training data selection device for selecting training data for training a deep learning model according to one embodiment of the present invention. The training data selection device 100 may include a memory 110 that stores instructions for selecting training data for training a deep learning model, and a processor 120 that performs operations for selecting training data for training a deep learning model according to the instructions stored in the memory 110.

[0041] Specifically, the learning data sorting device 100 may, but is not limited to, achieve desired system performance by utilizing a combination of a typical computing device (e.g., a device that may include computer processors, memory, storage, input and output devices, and other conventional computing device components; electronic communication devices such as routers and switches; and electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions for using the computing device in a specific manner).

[0042] Furthermore, the processor 120 of the learning data sorting device 100 may include hardware components such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. The network controller 100 may also further include an operating system and software configurations for applications performing specific purposes.

[0043] However, this does not exclude the case in which the learning data sorting device 100 includes an integrated processor, which is a medium, processor, and memory integrated into one form for carrying out the present invention.

[0044] On the other hand, the processor 120 of the training data selection device 100 can execute a process to generate at least one individual type (attribute) corresponding to each of the many training data contained in all the training data stored in the data pool, according to instructions stored in the memory 110, and to generate a binary graph matching each of the many training data contained in all the training data with the individual type. Then, according to instructions stored in the memory 110, the processor 120 of the training data selection device 100 can refer to the binary graph and execute a process to select n training data for training the deep learning model such that the number of individual types matching each of the n training data is within a predetermined threshold deviation. The n is the target number of training data for training the deep learning model, and may be an integer representing multiple numbers.

[0045] A method for selecting training data for training a deep learning model will be described below, referring to the training data selection device according to one embodiment of the present invention configured as described above.

[0046] [First Embodiment] Figure 2 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to the first embodiment of the present invention. Referring to Figure 2, the first embodiment of the present invention will be described as follows.

[0047] First, the training data selection device 100 can acquire (S110) at least one individual type (attribute) corresponding to each of the many training data contained in all the training data stored in the data pool.

[0048] In this case, the individual types may include 1_1 individual types to 1_x individual types corresponding to the 1st type possessed by each of the numerous training data, and 2_1 individual types to 2_y individual types corresponding to the 2nd type possessed by each of the numerous training data. Each of x and y may be an integer of 1 or more. However, the present invention is not limited thereto, and may have three or more type types depending on the characteristics of the training data.

[0049] For example, if the learning data is text data, it can be composed of various individual types corresponding to various other types, such as type types related to genres corresponding to individual types like essays, novels, reports, travelogues, and letters; type types related to publication locations corresponding to individual types like blogs, newspapers, and social media posts; type types related to languages ​​corresponding to individual types like Korean, English, and German; and type types related to fields corresponding to individual types like economic terms, social terms, and mathematical terms.

[0050] As another example, if the training data is audio data, it can be composed of various individual types corresponding to various other types, such as type types related to time of day corresponding to individual types like day, night, dawn, and evening; type types related to location corresponding to individual types like city, countryside, mountains, and the sea; and type types related to the subject of the sound corresponding to individual types like car sounds, human sounds, bird songs, and wind sounds.

[0051] As yet another example, if the training data is image data, it can be composed of various individual types corresponding to various individual types, such as type types related to time of day corresponding to individual types such as day, night, evening, and dawn; type types related to weather corresponding to individual types such as sunny, cloudy, rainy, and foggy; and type types related to objects corresponding to individual types such as cars, people, and animals.

[0052] Furthermore, each of the numerous training data sets has at least one unique attribute, which can be determined by a labeler labeling each of the numerous training data sets against a predefined attribute, or by automatically extracting the unique attribute present in each of the numerous training data sets using various techniques such as AI (Artificial Intelligence) and clustering.

[0053] As an example, the learning data sorting device 100 can transmit a large amount of learning data to a labeler terminal, and use a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the large amount of learning data.

[0054] As another example, the training data selection device 100 can automatically generate various individual types corresponding to the various types of images that each of the training images, which constitute all the training data, possesses.

[0055] In other words, referring to Figure 3, the training data selection device 100 can perform a process to generate a first scene vector corresponding to each training image by performing a first embedding operation 10_1 on each training image and then clustering the first scene vectors 20 to generate a first scene cluster, or a process to generate a kth scene vector corresponding to each training image by performing a kth embedding operation 10_k on each training image and then clustering the kth scene vectors 20 to generate a kth scene cluster. The value of k may be an integer of 1 or more. The training data selection device 100 can then generate individual types corresponding to the training images by referring to the first scene cluster or the kth scene cluster.

[0056] In this case, the training data selection device 100 can utilize visual foundation models such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), and DINOv2 for the first to kth embedding operations. Each of the first embedding model for the first embedding operation and each of the kth embedding models for the kth embedding operation may be different embedding models that generate scene vectors corresponding to each of the first to kth types, which are different types of data. Furthermore, the training data selection device 100 can use various clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift to cluster the first to kth scene vectors.

[0057] Furthermore, while the above method involved embedding each training image and performing clustering to generate at least one unique type for each training image, it is also possible to generate unique types using metadata contained in each training image. For example, if k is 1, that is, if one embedding operation generates only unique types for one type corresponding to each training image, then it is also possible to use the metadata of each training image to generate even more unique types for type types different from the one type generated by the embedding operation.

[0058] In other words, it is possible to examine the metadata contained in each training image, and then refer to the shooting time contained in each metadata to generate a unique type corresponding to each training image.

[0059] For example, in order to generate individual categories based on time zone types such as daytime, nighttime, evening, and dawn, the learning data selection device 100 can refer to the shooting time of each learning image, group the learning images according to a predetermined time zone, and set individual categories related to the time zone for each group of grouped learning images.

[0060] In addition, when the learning data selection device 100 further references the capture time of each learning image to generate individual types corresponding to each learning image, it can also use only one specific embedding model, which is one of the first to kth embedding models, to generate individual types corresponding to one type using that specific embedding model, and individual types corresponding to another type using metadata.

[0061] In other words, the training data selection device 100 can perform a specific embedding operation on each training image to generate a specific scene vector corresponding to each training image, and then cluster the specific scene vectors to generate a specific scene cluster. The training data selection device 100 can then refer to the metadata contained in each training image to confirm the shooting time of each training image. Subsequently, the training data selection device 100 can also refer to the specific scene cluster and shooting time to generate an individual type corresponding to each training image.

[0062] As yet another example, the training data sorting device 100 can generate individual types corresponding to various type types for the entire image, but it can also generate individual types corresponding to objects contained within the image.

[0063] In other words, referring to Figure 4, the training data selection device 100 performs object detection 1 on each of the training images, which are all training data, to detect at least one object from each training image, generates a cropped image by cropping 2 the region corresponding to the bounding box of each detected object in each training image, performs an embedding operation 10 on each cropped image to generate an object vector corresponding to each cropped image, clusters the object vectors 20 to generate an object cluster, and generates individual types corresponding to the training images by referring to the object cluster. At this time, in order to perform object detection on each of the training images, the training data selection device 100 can use open-world object detection models such as Grounding DINO, OWL-ViT (short for Vision Transformer for Open-World Localization), and YOLO (You Only Look Once)-World.

[0064] In this case, the training data selection device 100 can also generate individual types corresponding to the training images by further referring to the scene clusters generated based on the above description, in addition to the object clusters.

[0065] In other words, the training data selection device 100 can perform a process to generate a first scene cluster by performing a first embedding operation on each training image to generate a first scene vector corresponding to each training image, and then clustering the first scene vectors. Alternatively, it can perform a process to generate a kth scene cluster by performing a kth embedding operation on each training image to generate a kth scene vector corresponding to each training image, and then clustering the kth scene vectors. The training data selection device 100 can then perform object detection on each training image to detect at least one object from each training image, generate a cropped image by cropping the region corresponding to the bounding box of each detected object in each training image, perform an embedding operation on each cropped image to generate an object vector corresponding to each cropped image, and then cluster the object vectors to generate an object cluster. Subsequently, the training data selection device 100 can generate individual types corresponding to the training images by referring to the first scene clusters to the kth scene clusters and the object clusters.

[0066] On the other hand, while the above method generated a cropped image after performing object detection on each of the training images, if each training image is labeled with its own ground truth, it is also possible to generate a cropped image by referencing the ground truth of each training image.

[0067] In other words, the training data selection device 100 can refer to the ground truth information contained in each training image to generate a cropped image by cropping the region corresponding to the bounding box of each object from each training image, perform an embedding operation on each cropped image to generate an object vector corresponding to each cropped image, cluster the object vectors to generate an object cluster, and refer to the object cluster to generate individual types corresponding to the training images.

[0068] In this case, the training data selection device 100 can also generate individual types corresponding to the training images by further referring to the scene clusters generated based on the above description, in addition to the object clusters.

[0069] In other words, the training data selection device 100 can perform a process to generate a first scene cluster by performing a first embedding operation on each training image to generate a first scene vector corresponding to each training image, and then clustering the first scene vectors. Alternatively, it can perform a process to generate a kth scene cluster by performing a kth embedding operation on each training image to generate a kth scene vector corresponding to each training image, and then clustering the kth scene vectors. The training data selection device 100 can then refer to the ground truth information contained in each training image to generate cropped images by cropping the region corresponding to the bounding box of each object from each training image, perform an embedding operation on each cropped image to generate an object vector corresponding to each cropped image, and then cluster the object vectors to generate an object cluster. Subsequently, the training data selection device 100 can refer to the first scene clusters, the kth scene clusters, and the object clusters to generate individual types corresponding to the training images.

[0070] Next, referring again to Figure 2, the learning data selection device 100 can generate a binary graph (S120) that matches each of the numerous learning data included in the total learning data with an individual type.

[0071] As an example, referring to Figure 5, the bisection graph matches the relationship between the training data group and the individual type group. It can be understood that training data 1 has individual type 1 and individual type 6, training data 2 has individual type 1, individual type 5, and individual type 6, training data 3 has individual type 2 and individual type 7, training data 4 has individual type 1, training data 5 has individual type 3, individual type 6, and individual type 8, training data 6 has individual type 4, and training data 7 has individual type 2, individual type 5, individual type 7, and individual type 8. On the other hand, Figure 5 exemplifies some of the training data and some of the individual types in the bisection graph.

[0072] Next, referring to Figure 2 again, the training data selection device 100, by referring to a binary graph, can select n training data for training the deep learning model (S130) such that the number of individual types matching the n training data is within a predetermined threshold deviation when selecting n training data from all training data to match individual types. The n is the target number of training data for training the deep learning model, and may be an integer representing multiple numbers.

[0073] In this case, if the individual types include 1st_1 individual types to 1st_x individual types corresponding to the first type each of the numerous training data have, and 2nd_1 individual types to 2nd_y individual types corresponding to the second type each of the numerous training data have, the training data selection device 100 can select n training data so that the number of 1st_1 individual types to 1st_x individual types corresponding to the first type and the number of 2nd_1 individual types to 2nd_y individual types corresponding to the second type that match the n training data are within a threshold deviation. In addition, it can select n training data so that the number of 1st_1 individual types or the number of 1st_x individual types are within a first threshold deviation, and the number of 2nd_1 individual types or the number of 2nd_y individual types are within a second threshold deviation. In other words, the training data selection device 100 can select training data so that the number of training data for each individual type is as equal as possible for all individual types of each type.

[0074] On the other hand, the training data selection device 100 can use at least some of the heuristic algorithms, optimization algorithms, etc., to select training data so that the number of data for each individual type is as equal as possible.

[0075] First, the process by which the training data selection device 100 selects training data using a heuristic algorithm is described below.

[0076] The training data selection device 100 can refer to a binary graph to check the number of corresponding individual types that match each of all training data, and select the specific training data with the largest number of corresponding individual types.

[0077] The learning data selection device 100 can then check the number of remaining individual types that match each of the total learning data, excluding the corresponding individual types, and perform the process of selecting other specific learning data with the largest number of remaining individual types.

[0078] In this way, by repeating the process of selecting the training data with the largest number of remaining individual types from the individual types that have been removed from the selected individual types, until all individual types have been selected, the training data selection device 100 can execute a cycle of selecting a portion of the training data that matches all individual types.

[0079] Subsequently, the training data selection device 100 can repeat the process of executing the cycle from the remaining training data, after excluding a portion of the training data selected to match all individual types from the total training data, until n training data are selected.

[0080] As an example, referring to Figures 6a to 6d, in Figure 6a, since training data 7 matches the four individual types with the most data, the training data selection device 100 can select training data 7 from training data 1 to training data 7.

[0081] Then, in the bisection graph of Figure 6a, if we remove individual types 2, 5, 7, and 8, which match the training data 7, individual types 1, 3, 4, and 6 remain, as shown in Figure 6b.

[0082] As shown in Figure 6b, since training data 1, training data 2, and training data 5 are matched to the two individual types with the highest number of occurrences, the training data selection device 100 can select training data 1, which is one of training data 1, training data 2, and training data 5. In this case, if there are multiple training data sets with the highest number of occurrences of the corresponding individual type, such as training data 1, training data 2, and training data 5, the training data selection device 100 can select one of the multiple training data sets, training data 1, training data 2, and training data 5, according to the first criterion, and select training data 1, which is the specific training data set. The first criterion can be set in various ways, such as selecting one training data set from among multiple training data sets using a random selection algorithm, or selecting one training data set based on the sequential ranking of multiple training data sets.

[0083] Then, in the bisection graph of Figure 6b, if we remove individual types 1 and 6, which match training data 1, individual types 3 and 4 remain, as shown in Figure 6c.

[0084] As a result, in Figure 6c, since training data 5 and training data 6 are matched to the single individual type with the largest number of occurrences, the training data selection device 100 can select one of the training data 5 from training data 5 and training data 6. In this case, if there are multiple remaining training data, such as training data 5 and training data 6, with the largest number of occurrences of the remaining individual type, the training data selection device 100 can select one of the multiple remaining training data, training data 5 and training data 6, according to the second criterion, and select the specific training data 5. The second criterion can be set in various ways, such as selecting one of the training data from multiple training data using a random selection algorithm, or selecting one of the training data based on the sequential ranking of multiple training data. The second criterion can be set in the same way as the first criterion described above, or it can be set differently.

[0085] Then, in the bisection graph of Figure 6c, if we remove individual type 3, which matches training data 5, individual type 4 remains as shown in Figure 6d.

[0086] As shown in Figure 6d, since training data 6 matches the single individual type with the most data, the training data selection device 100 selects training data 6, thereby enabling it to execute one cycle of selecting a portion of the training data that matches all of the individual types. Subsequently, the training data selection device 100 repeats the above cycle from the remaining training data, excluding the portion of training data selected from all training data—training data 1, training data 5, and training data 7—to select the desired number of n training data.

[0087] On the other hand, while the above described selecting some training data that matches all individual types, it is also possible to select training data by further referring to the type to which each individual type belongs. In other words, it is possible to select some training data that matches all individual types belonging to all type categories. For example, when selecting some training data from daytime and nighttime type categories and car and person object type categories, it is also possible to select some training data that matches equally when matching daytime individual types and nighttime individual types, matches equally when matching car individual types and person individual types within daytime individual types, and matches equally when matching both car individual types and person individual types within nighttime individual types.

[0088] Next, the process by which the training data selection device 100 selects training data using an optimization algorithm will be described as follows.

[0089] The training data selection device 100 refers to a binary graph and, through an optimization algorithm, selects a specific subset of training data consisting of a predetermined number of training data that includes all individual types, which has the fewest number of training data. It then calculates the remaining training data by removing a specific number of training data included in the specific subset, and repeats the process of selecting at least one other specific subset of the remaining training data consisting of a predetermined number of training data that includes all individual types, so that the number of selected training data is n or more. In this case, ensuring that the number of selected training data is n or more does not mean unconditionally ensuring that the number of selected training data is n or more, but rather that the total number of training data ultimately selected does not fall below n, depending on the number of specific subsets selected last. In other words, if the number of training data points selected so far is less than n, and the current total number of training data points, which is the sum of the numbers of the currently selected specific subsets, becomes n or greater, the process of selecting training data can be terminated, and the current total number of training data points can be generated as a training dataset for training the deep learning model.

[0090] As an example, the training data selection device 100 uses linear programming as its optimization algorithm to calculate the product of a PxQ binary matrix corresponding to P individual types and Q training data in a binary graph with a Q-dimensional vector representing the selection goodness-of-fit variable for each of the Q training data in each of the P individual types. The device then selects a specific subset from among the selection goodness-of-fit variables of the Q-dimensional vector that satisfy the minimum value of the sum of the selection goodness-of-fit variables, and the specific training data corresponding to a specific selection goodness-of-fit variable whose sum of the selection goodness-of-fit variables is 1 or more, and the selection goodness-of-fit variables in the Q-dimensional vector are between 0 and 1. The device then calculates the remaining training data by removing the specific training data included in the specific subset from the Q training data. The P-dimensional vector can represent the sum of the goodness-of-fit variables for each of the P individual types. The training data selection device 100 can then repeat the process of selecting at least one other specific subset from the remaining training data using linear programming, so that the number of selected training data is n or more.

[0091] In this case, linear programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, 0 ≤ x ≤ 1

[0092] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether or not the training data i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection goodness-of-fit variable for the training data i. Ax is the matrix product of A and x, and Ax[j] can represent the sum of the goodness-of-fit values ​​of the training data belonging to individual type j.

[0093] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the linear programming method.

[0094] The dual linear programming method, which is a modification of the linear programming method described above, can be expressed as follows, but the present invention is not limited thereto, and the dual linear programming method can be applied in various ways by modifying the constraints in the linear programming method described above.

[0095] Purpose: min 1 T x Condition: Bx ≧ b, x ≥ 0

[0096] At this time, B=[A;-I], (P+Q)xQ binary matrix, B[j,i]=A[j,i], B[P+i,i]= -1 for all i ∈ [1,2,···,Q], B[P+i,j]= 0 for all j≠i b=[1 P ;-1 P ], (P+Q)-dimensional vector, b[j]= 1 for all j ∈ [1,2,···,P], b[P+i]= -1 for all i ∈ [1,2,···,Q] It can be expressed as follows.

[0097] As another example, the training data selection device 100 uses integer programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training data in a binary graph and a Q-dimensional vector representing each of the selection variables for each of the Q training data in each of the P individual types. From the selection variables of the Q-dimensional vector that satisfy a selection quantity of 1 or more and a selection variable in the Q-dimensional vector being 0 or 1, the device can select a specific subset containing specific training data corresponding to a specific selection variable whose sum of selection variables is the minimum value, and calculate the remaining training data by removing the specific training data included in the specific subset from the Q training data. The P-dimensional vector can represent the selection quantity of training data belonging to each of the P individual types. The training data selection device 100 can then repeat the process of selecting at least one other specific subset from the remaining training data using integer programming until the number of selected training data is n or more.

[0098] In this case, integer programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, x[i] ∈ {0,1} for all i

[0099] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether or not the training data i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of the training data i. Ax is the matrix product of A and x, and Ax[j] can represent the number of training data selected that belong to individual type j.

[0100] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the integer programming method.

[0101] According to the first embodiment of the present invention, it becomes possible to select training data for each type so that they are as evenly distributed as possible for all individual types. Furthermore, if ground truth is not labeled on all training data stored in the data pool, ground truth only needs to be labeled on the training data that has been selected as evenly as possible for each type, thereby reducing the cost of generating training datasets.

[0102] [Second Example] Figure 7 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to a second embodiment of the present invention. Referring to Figure 7, the second embodiment of the present invention will be described as follows. In the following description, detailed explanations will be omitted for parts that can be easily understood from the description of the first embodiment.

[0103] First, the training data selection device 100 can acquire at least one individual type corresponding to each of the many training images stored in the data pool (S210).

[0104] In this case, the individual types may include 1_1 individual types to 1_x individual types corresponding to the first type type possessed by each of the many training images, and 2_1 individual types to 2_y individual types corresponding to the second type type possessed by each of the many training images. Each of x and y may be an integer of 1 or more. However, the present invention is not limited thereto, and may have three or more type types depending on the characteristics of the training images.

[0105] For example, it can be composed of various individual types corresponding to various other types, such as type types related to time of day corresponding to individual types such as day, night, evening, and dawn; type types related to weather corresponding to individual types such as sunny, cloudy, rainy, and foggy; and type types related to location corresponding to individual types such as city, countryside, and highway.

[0106] Furthermore, each of the numerous training images has at least one unique attribute, which can be determined by a labeler labeling each of the numerous training images against a predefined attribute, or by automatically extracting the unique attribute present in each of the numerous training images using various techniques such as AI (Artificial Intelligence) and clustering.

[0107] As an example, the training data sorting device 100 can transmit a large number of training images to a labeler terminal, and the labeler corresponding to the labeler terminal can generate at least one individual type corresponding to each of the large number of training images.

[0108] As another example, the training data selection device 100 can automatically generate various individual types corresponding to the various type types of the overall image that each of the many training images possesses.

[0109] In other words, referring again to Figure 3, the training data selection device 100 can perform a process to generate a first scene vector corresponding to each training image by performing a first embedding operation 10_1 on each training image and clustering the first scene vectors 20 to generate a first scene cluster, or a process to generate a kth scene vector corresponding to each training image by performing a kth embedding operation 10_k on each training image and clustering the kth scene vectors 20 to generate a kth scene cluster. The value of k may be an integer of 1 or more. The training data selection device 100 can then generate individual types corresponding to the training images by referring to the first scene cluster or the kth scene cluster.

[0110] In this case, the training data selection device 100 can utilize visual foundation models such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), and DINOv2 for the first to kth embedding operations. Each of the first embedding model for the first embedding operation and each of the kth embedding models for the kth embedding operation may be different embedding models that generate scene vectors corresponding to each of the first to kth types, which are different types of data. Furthermore, the training data selection device 100 can use various clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift to cluster the first to kth scene vectors.

[0111] Furthermore, while the above method involved embedding each training image and performing clustering to generate at least one unique type for each training image, it is also possible to generate unique types using metadata contained in each training image. For example, if k is 1, that is, if one embedding operation generates only unique types for one type corresponding to each training image, then it is also possible to use the metadata of each training image to generate even more unique types for type types different from the one type generated by the embedding operation.

[0112] In other words, it is possible to examine the metadata contained in each training image, and then refer to the shooting time contained in each metadata to generate a unique type corresponding to each training image.

[0113] For example, in order to generate individual categories based on time zone types such as daytime, nighttime, evening, and dawn, the learning data selection device 100 can refer to the shooting time of each learning image, group the learning images according to a predetermined time zone, and set individual categories related to the time zone for each group of grouped learning images.

[0114] In addition, when the learning data selection device 100 further references the capture time of each learning image to generate individual types corresponding to each learning image, it can also use only one specific embedding model, which is one of the first to kth embedding models, to generate individual types corresponding to one type using that specific embedding model, and individual types corresponding to another type using metadata.

[0115] In other words, the training data selection device 100 can perform a specific embedding operation on each training image to generate a specific scene vector corresponding to each training image, and then cluster the specific scene vectors to generate a specific scene cluster. The training data selection device 100 can then refer to the metadata contained in each training image to confirm the shooting time of each training image. Subsequently, the training data selection device 100 can also refer to the specific scene cluster and shooting time to generate an individual type corresponding to each training image.

[0116] Next, referring again to Figure 7, the learning data selection device 100 can generate a binary graph (S220) that matches each of the numerous learning images with an individual type.

[0117] Next, the training data selection device 100 can refer to a binary graph and execute a cycle (S230) to select a portion of the training images from among the many training images that match all individual types.

[0118] In other words, the training data selection device 100 can refer to a binary graph to check the number of corresponding individual types that match each of the many training images, and select a specific training image with the largest number of corresponding individual types.

[0119] The learning data selection device 100 can then check the number of remaining individual types that match each of the many learning images, excluding the corresponding individual types, and select the other specific learning image with the largest number of remaining individual types.

[0120] In this way, by repeating the process of selecting the training image with the largest number of remaining individual types from the individual types that have been removed from the selected individual types, until all individual types have been selected, the training data selection device 100 can execute a cycle of selecting a portion of training images that match all individual types.

[0121] In this case, if there are multiple training images with the largest number of corresponding individual types, the training data selection device 100 can select one of the multiple training images according to the first criterion to select a specific training image. If there are multiple remaining training images with the largest number of the remaining individual types, the device can select one of the multiple remaining training images according to the second criterion to select another specific training image.

[0122] Subsequently, the training data selection device 100 can repeat the process of executing the cycle from the remaining training images, excluding a portion of the selected training images that match all individual types from the large number of training images, until n training images have been selected (S240).

[0123] In this case, if the individual types include 1_1 individual types to 1_x individual types corresponding to the first type each of the many training images possesses, and 2_1 individual types to 2_y individual types corresponding to the second type each of the many training images possesses, the training data selection device 100 can ensure that the number of 1_1 individual types to 1_x individual types corresponding to the first type and the number of 2_1 individual types to 2_y individual types corresponding to the second type that match the n training images are within a threshold deviation. In addition, it can ensure that the number of 1_1 individual types to the number of 1_x individual types are within a first threshold deviation, and the number of 2_1 individual types to the number of 2_y individual types are within a second threshold deviation, by selecting n training images. In other words, the training data selection device 100 can select training images such that the number of training images for each individual type is as equal as possible for all individual types of each type. For example, when selecting some training images from daytime and nighttime type categories and car and person object type categories, it is also possible to select some training images that match equally when matching to individual daytime and nighttime type categories, match equally when matching to individual car and person type categories within the daytime type category, and match equally when matching to both individual car and person type categories within the nighttime type category.

[0124] [Third Embodiment] Figure 8 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to the third embodiment of the present invention. Referring to Figure 8, the third embodiment of the present invention will be described as follows. In the following description, detailed explanations will be omitted for parts that can be easily understood from the description of the first embodiment.

[0125] First, the training data selection device 100 can acquire at least one individual type corresponding to each of the many training images stored in the data pool (S310).

[0126] In this case, the individual types may include 1_1 individual types to 1_x individual types corresponding to the first type type possessed by each of the many training images, and 2_1 individual types to 2_y individual types corresponding to the second type type possessed by each of the many training images. Each of x and y may be an integer of 1 or more. However, the present invention is not limited thereto, and may have three or more type types depending on the characteristics of the training images.

[0127] For example, it can be composed of various individual types corresponding to various other types, such as type types related to time of day corresponding to individual types such as day, night, evening, and dawn; type types related to weather corresponding to individual types such as sunny, cloudy, rainy, and foggy; and type types related to location corresponding to individual types such as city, countryside, and highway.

[0128] Furthermore, each of the numerous training images has at least one unique attribute, which can be determined by a labeler labeling each of the numerous training images against a predefined attribute, or by automatically extracting the unique attribute present in each of the numerous training images using various techniques such as AI (Artificial Intelligence) and clustering.

[0129] As an example, the training data sorting device 100 can transmit a large number of training images to a labeler terminal, and the labeler corresponding to the labeler terminal can generate at least one individual type corresponding to each of the large number of training images.

[0130] As another example, the training data selection device 100 can automatically generate various individual types corresponding to the various type types of the overall image that each of the many training images possesses.

[0131] In other words, referring again to Figure 3, the training data selection device 100 can perform a process to generate a first scene vector corresponding to each training image by performing a first embedding operation 10_1 on each training image and clustering the first scene vectors 20 to generate a first scene cluster, or a process to generate a kth scene vector corresponding to each training image by performing a kth embedding operation 10_k on each training image and clustering the kth scene vectors 20 to generate a kth scene cluster. The value of k may be an integer of 1 or more. The training data selection device 100 can then generate individual types corresponding to the training images by referring to the first scene cluster or the kth scene cluster.

[0132] In this case, the training data selection device 100 can utilize visual foundation models such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), and DINOv2 for the first to kth embedding operations. Each of the first embedding model for the first embedding operation and each of the kth embedding models for the kth embedding operation may be different embedding models that generate scene vectors corresponding to each of the first to kth types, which are different types of data. Furthermore, the training data selection device 100 can use various clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift to cluster the first to kth scene vectors.

[0133] Furthermore, while the above method involved embedding each training image and performing clustering to generate at least one unique type for each training image, it is also possible to generate unique types using metadata contained in each training image. For example, if k is 1, that is, if one embedding operation generates only unique types for one type corresponding to each training image, then it is also possible to use the metadata of each training image to generate even more unique types for type types different from the one type generated by the embedding operation.

[0134] In other words, it is possible to examine the metadata contained in each training image, and then refer to the shooting time contained in each metadata to generate a unique type corresponding to each training image.

[0135] For example, in order to generate individual categories based on time zone types such as daytime, nighttime, evening, and dawn, the learning data selection device 100 can refer to the shooting time of each learning image, group the learning images according to a predetermined time zone, and set individual categories related to the time zone for each group of grouped learning images.

[0136] In addition, when the learning data selection device 100 further references the capture time of each learning image to generate individual types corresponding to each learning image, it can also use only one specific embedding model, which is one of the first to kth embedding models, to generate individual types corresponding to one type using that specific embedding model, and individual types corresponding to another type using metadata.

[0137] In other words, the training data selection device 100 can perform a specific embedding operation on each training image to generate a specific scene vector corresponding to each training image, and then cluster the specific scene vectors to generate a specific scene cluster. The training data selection device 100 can then refer to the metadata contained in each training image to confirm the shooting time of each training image. Subsequently, the training data selection device 100 can also refer to the specific scene cluster and shooting time to generate an individual type corresponding to each training image.

[0138] Next, referring again to Figure 8, the learning data selection device 100 can generate a binary graph (S320) that matches each of the numerous learning images with an individual type.

[0139] Next, the training data selection device 100 can refer to a binary graph and, through an optimization algorithm, select a specific subset from a predetermined number of training images that include all individual types, which has the fewest number of training images, and calculate the remaining training images after removing a specific number of training images included in the specific subset (S330).

[0140] As an example, the training data selection device 100 uses linear programming as an optimization algorithm to calculate the product of a PxQ binary matrix corresponding to P individual types and Q training images in a binary graph with a Q-dimensional vector representing the selection goodness-of-fit variable for each of the Q training images in each of the P individual types. The device then selects a specific subset from among the selection goodness-of-fit variables of the Q-dimensional vector that satisfy the condition that the sum of the selection goodness-of-fit variables in the Q-dimensional vector is 1 or greater, and that the selection goodness-of-fit variables in the Q-dimensional vector are between 0 and 1. The device then selects a specific subset that includes a specific training image corresponding to a specific selection goodness-of-fit variable whose sum of the selection goodness-of-fit variables is the minimum value, and calculates the remaining training images by excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector can represent the sum of the goodness-of-fit variables for each of the Q training images belonging to each of the P individual types.

[0141] In this case, linear programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, 0 ≤ x ≤ 1

[0142] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether the training image i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection goodness-of-fit variable for the training image i. Ax is the matrix product of A and x, and Ax[j] can represent the sum of the fitness scores of the training images belonging to individual type j.

[0143] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the linear programming method.

[0144] The dual linear programming method transformed from the above linear programming method is expressed as follows. However, the present invention is not limited thereto, and the dual linear programming method can be applied by various methods of transforming the constraints in the linear programming method.

[0145] Objective: min 1 T x Constraints: Bx ≧ b, x ≧ 0

[0146] At this time, B = [A; -I], (P + Q)×Q binary matrix, B[j,i] = A[j,i], B[P + i,i] = -1 for all i ∈ [1,2,···,Q], B[P + i,j] = 0 for all j≠i b = [1 P ;-1 P , (P + Q)-dimensional vector, b[j] = 1 for all j ∈ [1,2,···,P], b[P + i] = -1 for all i ∈ [1,2,···,Q] can be expressed as follows.

[0147] As another example, the training data selection device 100 uses integer programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in a binary graph and a Q-dimensional vector representing each of the selection variables for each of the Q training images in each of the P individual types. From among the selection variables in the Q-dimensional vector that satisfy a selection quantity of 1 or more and a selection variable in the Q-dimensional vector is 0 or 1, the device can select a specific subset that includes a specific training image corresponding to a specific selection variable whose sum of selection variables is the minimum value, and calculate the remaining training images by excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector can represent the selection quantity of training images belonging to each of the P individual types.

[0148] In this case, integer programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, x[i] ∈ {0,1} for all i

[0149] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether the training image i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of the training image i. Ax is the matrix product of A and x, and Ax[j] can represent the number of selected training images belonging to individual type j.

[0150] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the integer programming method.

[0151] Subsequently, the training data selection device 100 can repeat the process of selecting at least one other specific subset consisting of a predetermined number of training images containing all individual types from the remaining training images through an optimization algorithm until n training images for training the deep learning model have been selected (S340).

[0152] In this case, if the individual types include 1_1 individual types to 1_x individual types corresponding to the first type each of the many training images possesses, and 2_1 individual types to 2_y individual types corresponding to the second type each of the many training images possesses, the training data selection device 100 can ensure that the number of 1_1 individual types to 1_x individual types corresponding to the first type and the number of 2_1 individual types to 2_y individual types corresponding to the second type that match the n training images are within a threshold deviation. In addition, it can ensure that the number of 1_1 individual types to the number of 1_x individual types are within a first threshold deviation, and the number of 2_1 individual types to the number of 2_y individual types are within a second threshold deviation, by selecting n training images. In other words, the training data selection device 100 can select training images such that the number of training images for each individual type is as equal as possible for all individual types of each type. For example, when selecting some training images from daytime and nighttime type categories and car and person object type categories, it is also possible to select some training images that match equally when matching to individual daytime and nighttime type categories, match equally when matching to individual car and person type categories within the daytime type category, and match equally when matching to both individual car and person type categories within the nighttime type category.

[0153] [Fourth embodiment] Figure 9 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to the fourth embodiment of the present invention. Referring to Figure 9, the fourth embodiment of the present invention will be described as follows. In the following description, detailed explanations will be omitted for parts that can be easily understood from the description of the first embodiment.

[0154] First, the training data selection device 100 can acquire (S410) at least one individual type (attribute) corresponding to at least one object contained in each of the many training images stored in the data pool.

[0155] In this case, each object has at least one unique attribute, which can be determined by a labeler labeling each of the numerous training images against a predefined attribute, or by automatically extracting the unique attributes present in each of the numerous training images using various techniques such as AI (Artificial Intelligence) and clustering.

[0156] As an example, the training data sorting device 100 can transmit a large number of training images to a labeler terminal, and the labeler corresponding to the labeler terminal can generate at least one individual type corresponding to each object contained in each of the large number of training images.

[0157] As another example, the training data sorting device 100 can automatically generate at least one distinct type corresponding to each object contained in each of the numerous training images.

[0158] In other words, referring again to Figure 4, the training data selection device 100 performs object detection 1 on each training image to detect at least one object from each training image, generates a cropped image by cropping 2 the region corresponding to the bounding box of each detected object in each training image, performs an embedding operation 10 on each cropped image to generate an object vector corresponding to each cropped image, clusters the object vectors 20 to generate an object cluster, and generates individual types corresponding to the training images by referring to the object cluster. At this time, in order to perform object detection on each training image, the training data selection device 100 can use open-world object detection models such as Grounding DINO, OWL-ViT (short for Vision Transformer for Open-World Localization), and YOLO (You Only Look Once)-World.

[0159] In this case, the training data selection device 100 can also, in addition to object clusters, check the metadata contained in each training image, and further refer to the shooting time contained in each metadata to generate individual types corresponding to each training image.

[0160] On the other hand, while the above method generated a cropped image after performing object detection on each of the training images, if each training image is labeled with its own ground truth, it is also possible to generate a cropped image by referencing the ground truth of each training image.

[0161] In other words, the training data selection device 100 can refer to the ground truth information contained in each training image to generate a cropped image by cropping the region corresponding to the bounding box of each object from each training image, perform an embedding operation on each cropped image to generate an object vector corresponding to each cropped image, cluster the object vectors to generate an object cluster, and refer to the object cluster to generate individual types corresponding to the training images.

[0162] Next, referring again to Figure 9, the learning data selection device 100 can generate a binary graph (S420) that matches each of the numerous learning images with an individual type.

[0163] Next, the training data selection device 100 can refer to a binary graph and, through an optimization algorithm, select a specific subset from a predetermined number of training images that include all individual types, which has the fewest number of training images, and calculate the remaining training images after removing a specific number of training images included in the specific subset (S430).

[0164] As an example, the training data selection device 100 uses linear programming as an optimization algorithm to calculate the product of a PxQ binary matrix corresponding to P individual types and Q training images in a binary graph with a Q-dimensional vector representing the selection goodness-of-fit variable for each of the Q training images in each of the P individual types. The device then selects a specific subset from among the selection goodness-of-fit variables of the Q-dimensional vector that satisfy the condition that the sum of the selection goodness-of-fit variables in the Q-dimensional vector is 1 or greater, and that the selection goodness-of-fit variables in the Q-dimensional vector are between 0 and 1. The device then selects a specific subset that includes a specific training image corresponding to a specific selection goodness-of-fit variable whose sum of the selection goodness-of-fit variables is the minimum value, and calculates the remaining training images by excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector can represent the sum of the goodness-of-fit variables for each of the Q training images belonging to each of the P individual types.

[0165] In this case, linear programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, 0 ≤ x ≤ 1

[0166] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether the training image i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection goodness-of-fit variable for the training image i. Ax is the matrix product of A and x, and Ax[j] can represent the sum of the fitness scores of the training images belonging to individual type j.

[0167] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the linear programming method.

[0168] The dual linear programming method, which is a modification of the linear programming method described above, can be expressed as follows, but the present invention is not limited thereto, and the dual linear programming method can be applied in various ways by modifying the constraints in the linear programming method described above.

[0169] Purpose: min 1 T x Condition: Bx ≧ b, x ≥ 0

[0170] At this time, B=[A;-I], (P+Q)xQ binary matrix, B[j,i]=A[j,i], B[P+i,i]= -1 for all i ∈ [1,2,···,Q], B[P+i,j]= 0 for all j≠i b=[1 P ;-1 P ], (P+Q)-dimensional vector, b[j]= 1 for all j ∈ [1,2,···,P], b[P+i]= -1 for all i ∈ [1,2,···,Q] It can be expressed as follows.

[0171] As another example, the training data selection device 100 uses integer programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in a binary graph and a Q-dimensional vector representing the selection variable for each of the Q training images in each of the P individual types. From the selection variables of the Q-dimensional vector that satisfy a selection quantity of 1 or more and a selection variable of 0 or 1, the device can select a specific subset that includes a specific training image corresponding to a specific selection variable whose sum of selection variables is the minimum value, and calculate the remaining training images by excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector can represent the selection quantity of training images belonging to each of the P individual types.

[0172] In this case, integer programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, x[i] ∈ {0,1} for all i

[0173] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether the training image i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of the training image i. Ax is the matrix product of A and x, and Ax[j] can represent the number of selected training images belonging to individual type j.

[0174] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the integer programming method.

[0175] Subsequently, the training data selection device 100 can repeat the process of selecting at least one other specific subset consisting of a predetermined number of training images containing all individual types from the remaining training images through an optimization algorithm until n training images for training the deep learning model have been selected (S440).

[0176] The embodiments of the present invention described above are implemented in the form of program instructions that can be executed through various computer components and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or they may be known and available to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0177] Although the present invention has been described above with specific details such as concrete components, and with limited embodiments and drawings, these are provided only to aid in a more overall understanding of the invention, and the invention is not limited to the above embodiments. A person with ordinary skill in the art to which the invention pertains can make various modifications and variations from this description.

[0178] Therefore, the concept of the present invention shall not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims described below shall also fall within the scope of the concept of the present invention. [Explanation of symbols]

[0179] 100: Training data sorting device 110: Memory 120: Processor

Claims

1. In a method for selecting training data for training a deep learning model, (a) A learning data selection device generates at least one attribute corresponding to each of the at least one object contained in each of the numerous learning images stored in the data pool, and generates a binary graph matching each of the numerous learning images with the attribute; and (b) The learning data selection device (i) refers to the binary graph and, through an optimization algorithm, selects a specific subset consisting of a predetermined number of learning images that include all of the individual types, which has the fewest number of learning images, calculates the remaining learning images after removing a specific number of learning images included in the specific subset, and (ii) from the remaining learning images, through the optimization algorithm, selects at least one other specific subset consisting of a predetermined number of learning images that include all of the individual types, until n learning images for training the deep learning model (where n is the target number of learning images for training the deep learning model and is an integer representing a number) are selected; A method that includes this.

2. In step (b) above, The training data selection device uses linear programming to calculate the product of a PxQ binary matrix corresponding to the P individual types and Q training images in the binary graph and a Q-dimensional vector representing the selection goodness-of-fit variable for each of the Q training images in each of the P individual types. The device generates a P-dimensional vector (the P-dimensional vector represents the sum of the goodness-of-fit variables for the Q training images belonging to each of the P individual types) where the sum of the goodness-of-fit variables is 1 or more, and the selection goodness-of-fit variable in the Q-dimensional vector is between 0 and 1. From among the selection goodness-of-fit variables of the Q-dimensional vector, the device selects a specific subset that includes a specific training image corresponding to a specific selection goodness-of-fit variable whose sum of the selection goodness-of-fit variables has the minimum value. The device then calculates the remaining training images by removing the specific training image included in the specific subset from the Q training images. The method according to claim 1, wherein the process of selecting at least one other specific subset using the linear programming method with respect to the remaining training images is repeated so that the number of selected training images is n or more.

3. In step (b) above, The method according to claim 2, wherein the training data selection device selects the specific subset using a dual linear programming method that applies at least one of the following constraints in linear programming: merging, separating, and sign changing.

4. In step (b) above, The training data selection device uses integer programming to calculate the product of a PxQ binary matrix corresponding to the P individual types and Q training images in the binary graph and a Q-dimensional vector representing the selection variable for each of the Q training images in each of the P individual types. The device generates a P-dimensional vector (the P-dimensional vector represents the selection quantity of training images belonging to each of the P individual types) in which the selection quantity is 1 or more, and the selection variable in the Q-dimensional vector is 0 or 1. From among the selection variables of the Q-dimensional vector, the device selects a specific subset that includes a specific training image corresponding to a specific selection variable whose sum of the selection variables is the minimum value. The device then calculates the remaining training images by removing the specific training image included in the specific subset from the Q training images. The method according to claim 1, wherein the process of selecting at least one other specific subset from the remaining training images using the integer programming method is repeated so that the number of selected training images is n or more.

5. In step (b) above, The method according to claim 4, wherein the training data selection device selects the particular subset using a dual integer programming method to which at least one of merging, separating, and sign-changing constraints in integer programming is applied.

6. In step (a) above, The method according to claim 1, wherein the learning data selection device transmits the learning images to a labeler terminal, and a labeler corresponding to the labeler terminal generates at least one individual type corresponding to each of the objects contained in each of the learning images.

7. In step (a) above, The method according to claim 1, wherein the training data selection device performs object detection on each of the training images to detect at least one object from each of the training images, generates a cropped image by cropping the region corresponding to the bounding box of each detected object in each of the training images, performs an embedding operation on each of the cropped images to generate an object vector corresponding to each of the cropped images, clusters the object vectors to generate an object cluster, and generates the individual type corresponding to the training image by referring to the object cluster.

8. In step (a) above, The method according to claim 1, wherein the training data selection device refers to the ground truth information contained in each of the training images to generate a cropped image obtained by cropping the region corresponding to the bounding box of each object from each of the training images, performs an embedding operation on each of the cropped images to generate an object vector corresponding to each of the cropped images, clusters the object vectors to generate an object cluster, and refers to the object cluster to generate the individual type corresponding to the training image.

9. In step (a) above, The method according to claim 1, wherein the learning data selection device checks the metadata contained in each of the learning images, and further refers to the shooting time contained in each of the metadata to generate the individual type corresponding to each of the learning images.

10. In step (b) above, The aforementioned specific individual type includes a first_1 specific individual type to a first_x specific individual type corresponding to the object (where x is an integer of 1 or more), and a second_1 specific individual type to a second_y specific individual type corresponding to the time of shooting (where y is an integer of 1 or more), The method according to claim 9, wherein the learning data selection device selects the n learning images such that the number of the first_1 specific individual type to the first_x specific individual type corresponding to the object that matches the n learning images and the number of the second_1 specific individual type to the second_y specific individual type corresponding to the shooting time are within the threshold deviation, the number of the first_1 specific individual type to the number of the first_x specific individual type are within the first threshold deviation, and the number of the second_1 specific individual type to the second_y specific individual type are within the second threshold deviation.

11. In a training data selection device for selecting training data for training a deep learning model, Memory containing instructions for selecting training data for training a deep learning model; and A processor that performs an operation to select training data for training the deep learning model in accordance with the instructions stored in the memory; Includes, The processor is a learning data selection device that performs the following processes until n learning images for training the deep learning model are selected (i) each at least one attribute corresponding to each at least one object contained in each of a large number of learning images stored in a data pool, and a binary graph matching each of the large number of learning images with the attribute; and (ii) by referring to the binary graph, an optimization algorithm selects a specific subset with the fewest number of learning images from a subset consisting of a predetermined number of learning images containing all the attribute types, calculates the remaining learning images excluding a specific number of learning images contained in the specific subset, and (ii) the optimization algorithm selects at least one other specific subset from the remaining learning images consisting of a predetermined number of learning images containing all the attribute types, and repeats this process until n learning images for training the deep learning model are selected (where n is the target number of learning images for training the deep learning model, and is an integer representing a plurality of numbers).

12. The aforementioned processor, In the process described in (II) above, using linear programming, the product of a PxQ binary matrix corresponding to the P individual types and Q training images in the binary graph and a Q-dimensional vector representing the selection goodness-of-fit variable for each of the Q training images in each of the P individual types is calculated to generate a P-dimensional vector (the P-dimensional vector represents the sum of the goodness-of-fit for the Q training images belonging to each of the P individual types). The sum of the goodness-of-fit in the P-dimensional vector is 1 or more, and the selection goodness-of-fit variable in the Q-dimensional vector is 0 or more and 1 or less. From among the selection goodness-of-fit variables of the Q-dimensional vector, a specific subset is selected that includes a specific training image corresponding to a specific selection goodness-of-fit variable whose sum of the selection goodness-of-fit variables has the minimum value. The remaining training images are calculated by removing the specific training image included in the specific subset from the Q training images. The learning data selection device according to claim 11, wherein the process of selecting at least one other specific subset using the linear programming method with respect to the remaining learning images is repeated so that the number of selected learning images is n or more.

13. The aforementioned processor, The training data sorting apparatus according to claim 12, wherein in the process of (II) above, the particular subset is sorted using a dual linear programming method which applies at least one of the constraint merging, separation, and sign changing in linear programming.

14. The aforementioned processor, In the process described in (II) above, using integer programming, the product of a PxQ binary matrix corresponding to the P individual types and Q training images in the binary graph and a Q-dimensional vector representing the selection variable for each of the Q training images in each of the P individual types is calculated to generate a P-dimensional vector (the P-dimensional vector represents the selection quantity of training images belonging to each of the P individual types), and the selection quantity in the P-dimensional vector is 1 or more, and the selection variable in the Q-dimensional vector is 0 or 1. From among the selection variables of the Q-dimensional vector, a specific subset is selected that includes a specific training image corresponding to a specific selection variable whose sum of the selection variables has the minimum value, and the remaining training images are calculated by removing the specific training image included in the specific subset from the Q training images. The learning data selection device according to claim 11, wherein the process of selecting at least one other specific subset using the linear programming method with respect to the remaining learning images is repeated so that the number of selected learning images is n or more.

15. The aforementioned processor, The training data sorting apparatus according to claim 14, wherein in the process of (II), the particular subset is sorted using a dual integer programming method which applies at least one of the following constraints in integer programming: merging, separating, and sign-changing.

16. The aforementioned processor, The learning data sorting apparatus according to claim 11, wherein in the process of (I) above, the learning images are transmitted to a labeler terminal, and a labeler corresponding to the labeler terminal generates at least one individual type corresponding to each of the objects contained in each of the learning images.

17. The aforementioned processor, The learning data sorting apparatus according to claim 11, wherein in the process of (I) above, object detection is performed on each of the learning images to detect at least one object from each of the learning images, a cropped image is generated by cropping the region corresponding to the bounding box of each of the detected objects in each of the learning images, an embedding operation is performed on each of the cropped images to generate an object vector corresponding to each of the cropped images, the object vectors are clustered to generate an object cluster, and the individual types corresponding to the learning images are generated by referring to the object cluster.

18. The aforementioned processor, The learning data sorting device according to claim 11, wherein in the process of (I) above, the device refers to the ground truth information contained in each of the learning images to generate a cropped image obtained by cropping the region corresponding to the bounding box of each object from each of the learning images, performs an embedding operation on each of the cropped images to generate an object vector corresponding to each of the cropped images, clusters the object vectors to generate an object cluster, and refers to the object cluster to generate the individual type corresponding to the learning images.

19. The aforementioned processor, The learning data sorting apparatus according to claim 11, wherein in the process of (I) above, the metadata contained in each of the learning images is checked, and the shooting time contained in each of the metadata is further referenced to generate the individual type corresponding to each of the learning images.

20. The aforementioned processor, In the process described in (II) above, a first_1 specific individual type or a first_x specific individual type (where x is an integer of 1 or more) corresponding to the object, and a second_1 specific individual type or a second_y specific individual type (where y is an integer of 1 or more) corresponding to the time of capture, A learning data selection device according to claim 19, which selects the n learning images such that the number of the first_1 specific individual type to the first_x specific individual type corresponding to the object that matches the n learning images and the number of the second_1 specific individual type to the second_y specific individual type corresponding to the shooting time are within the threshold deviation, the number of the first_1 specific individual type to the first_x specific individual type are within the first threshold deviation, and the number of the second_1 specific individual type to the second_y specific individual type are within the second threshold deviation.

Images