A method for selecting training data for training a deep learning model and a training data selection device using the same.

Abstract

This provides a method for selecting training data for training deep learning models. [Solution] The method includes the steps of: a learning data selection device generating a binary graph matching each of a large number of learning images with an individual type; and the learning data selection device referring to the binary graph to confirm the number of corresponding individual types that match each learning image from among the individual types, selecting a specific learning image with the largest number of corresponding individual types, confirming the number of remaining corresponding individual types that match each learning image from the remaining individual types after removing the corresponding individual types, selecting another specific learning image with the largest number of remaining corresponding individual types, repeating this process to select some learning images that match all individual types, and repeating the process of executing the cycle from the remaining learning images after removing the selected learning images from a large number of learning images until n learning images are selected.

Description

[Technical Field]

[0001] The present invention relates to a method for uniformly selecting training data for training a deep learning model from all training data stored in a data pool, without bias or variability in the data, and a training data selection device utilizing this method. [Background technology]

[0002] Generally, deep learning models recognize complex patterns in images, text, sound, and other data to generate accurate insights and predictions, and are applied in various fields such as computer vision, speech recognition, autonomous vehicles, robotics, natural language processing, and medical image analysis.

[0003] In order for such deep learning models to accurately perform their intended tasks, they must be trained using a large amount of training data.

[0004] Conventional methods for selecting training data for training deep learning models from a collected data pool include random sampling, which selects a target number of training data from all the training data stored in the data pool, and vector quantization, which clusters and groups the vectors representing each of the training data generated by embedding extraction, and then selects representative values ​​for each group of grouped vectors.

[0005] For example, Patent Document 1 discloses a method for preparing cognitive data for training a deep learning model, and Patent Document 2 discloses a similarity-based clustering device and method using deep learning learning techniques. Furthermore, Patent Document 3 discloses a training device and method for a deep learning classification model, and Patent Document 4 discloses a system and method for training a machine learning model using active learning.

[0006] However, conventional methods for selecting training data have the problem of bias and variability in data types.

[0007] For example, if a data pool contains 1 million training images, with 70% related to sunny weather, 20% related to cloudy weather, 5% related to foggy weather, and 5% related to snowy and / or rainy weather, then randomly sampling 10,000 training images would result in only about 500 images being selected from a total of 50,000 images related to snowy and / or rainy weather. This would lead to a bias and variability in the selection of training images based on weather type.

[0008] Furthermore, while using vector quantization to select training images can somewhat mitigate the bias and variability in the types of training images selected by embedding extraction and clustering, it cannot fundamentally prevent problems related to data bias and variability.

[0009] Therefore, the applicant aims to propose a method that enables the selection of training data for training a deep learning model from all training data stored in a data pool, categorized by type and selected evenly without bias or variability. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] U.S. Patent No. 11475335 [Patent Document 2] Korean Published Patent No. 10-2023-0068941 [Patent Document 3] Patent No. 7225614 [Patent Document 4] U.S. Patent Publication No. 11663409 [Overview of the project] [Problems that the invention aims to solve]

[0011] The purpose of this invention is to solve all of the problems described above.

[0012] Another objective of this invention is to enable the selection of training data for training a deep learning model from a data pool, without bias towards any particular type, and to allow for even selection by type.

[0013] Another objective of this invention is to enable the equal selection of training data from a data pool that is suitable for various scenarios according to the type of training data. [Means for solving the problem]

[0014] According to one embodiment of the present invention, in a method for selecting training data for training a deep learning model, (a) a training data selection device generates at least one attribute corresponding to each of a large number of training images stored in a data pool, and generates a binary graph matching each of the large number of training images with the attribute; and (b) the training data selection device (i) refers to the binary graph to confirm the number of corresponding attributes that match each of the training images, and selects a specific training image with the largest number of corresponding attributes, (i i) A method is provided which includes the step of checking the number of each of the remaining individual types that match each of the training images from the remaining individual types excluding the corresponding individual types, and repeating the process of selecting another specific training image with the largest number of the remaining individual types to perform a cycle of selecting some training images that match all of the individual types, and repeating the process of performing the cycle from the remaining training images excluding the selected some training images from the large number of training images until n training images for training the deep learning model (wherein n is the target number of training images for training the deep learning model and is an integer representing a number) are selected.

[0015] In one example, in the step (b), the individual types include the first_1 individual type to the first_x individual type (where x is an integer greater than or equal to 1) corresponding to the first type that each of the learning images has, and the second_1 individual type to the second_y individual type (where y is an integer greater than or equal to 1) corresponding to the second type that each of the learning images has. The learning data selection device selects the n learning images such that the number of the first_1 individual type to the first_x individual type corresponding to the first type that matches the n learning images and the number of the second_1 individual type to the second_y individual type corresponding to the second type are within the threshold deviation, and such that the number of the first_1 individual type to the first_x individual type is within the first threshold deviation and the number of the second_1 individual type to the second_y individual type is within the second threshold deviation.

[0016] In one example, in the step (b), when there are multiple learning images with the largest number of corresponding individual types, the learning data selection device selects any one of the multiple learning images according to the first criterion so that the specific learning image is selected. When there are multiple remaining learning images with the largest number of the remaining individual types, the learning data selection device selects any one of the multiple remaining learning images according to the second criterion so that the other specific learning image is selected.

[0017] In one example, in the step (a), the learning data selection device transmits the learning images to a labeler terminal, and uses the labeler corresponding to the labeler terminal to generate at least one respective individual type corresponding to each of the learning images.

[0018] In one example, in the step (a), the learning data selection device executes a first embedding operation on each of the learning images to generate a first scene vector corresponding to each of the learning images, clusters the first scene vectors to generate a first scene cluster, or executes a k-th embedding operation (where k is an integer greater than or equal to 1) on each of the learning images to generate a k-th scene vector corresponding to each of the learning images, clusters the k-th scene vectors to generate a k-th scene cluster, and generates the individual type corresponding to the learning image by referring to the first scene cluster to the k-th scene cluster.

[0019] In one example, in the step (a), the learning data selection device checks each piece of meta information included in each of the learning images, further refers to each shooting time included in each piece of meta information, and generates the individual type corresponding to each of the learning images.

[0020] In one example, in the step (a), the learning data selection device (i) executes a specific embedding operation on each of the learning images to generate a specific scene vector corresponding to each of the learning images, clusters the specific scene vectors to generate a specific scene cluster, (ii) refers to each piece of meta information included in each of the learning images to check the shooting time of each of the learning images, and (iii) generates the individual type corresponding to each of the learning images by referring to the specific scene cluster and the shooting time.

[0021] According to one embodiment of the present invention, a training data selection device for selecting training data for training a deep learning model includes: a memory storing instructions for selecting training data for training a deep learning model; and a processor that performs operations for selecting training data for training a deep learning model in accordance with the instructions stored in the memory, wherein the processor (I) generates at least one attribute corresponding to each of a large number of training images stored in a data pool, and generates a binary graph matching each of the large number of training images with the attribute; and (II) (i) referring to the binary graph, the corresponding attribute that matches each of the attribute for each of the training images A learning data selection device is provided that performs a process of selecting a portion of learning images that match all of the individual types, by (ii) checking the number of each of the corresponding individual types and selecting a specific learning image that has the largest number of the corresponding individual types, and by repeating the process of checking the number of each of the remaining individual types that match each of the learning images and selecting another specific learning image that has the largest number of the remaining individual types, thereby performing a cycle to select a portion of learning images that match all of the individual types, and repeating the process of performing the cycle from the remaining learning images after removing the portion of learning images selected from the large number of learning images until n learning images for training the deep learning model (where n is the target number of learning images for training the deep learning model and is an integer representing a number of items) are selected.

[0022] In one example, the processor selects the n learning images in the (II) process such that the individual types include a 1_1 individual type to a 1_x individual type (where x is an integer of 1 or more) corresponding to a first type type that each of the learning images has, and a 2_1 individual type to a 2_y individual type (where y is an integer of 1 or more) corresponding to a second type type that each of the learning images has, and that the number of 1_1 individual types to 1_x individual types corresponding to the first type and the number of 2_1 individual types to 2_y individual types corresponding to the second type that match the n learning images are within the threshold deviation, the number of 1_1 individual types to the number of 1_x individual types are within the first threshold deviation, and the number of 2_1 individual types to the number of 2_y individual types are within the second threshold deviation.

[0023] In one example, in the (II) process, if there are multiple training images that have the largest number of the corresponding individual types, the processor selects one of the multiple training images according to the first criterion to select the specific training image, and if there are multiple remaining training images that have the largest number of the remaining individual types, the processor selects one of the multiple remaining training images according to the second criterion to select the other specific training image.

[0024] In one example, the processor transmits the learning images to a labeler terminal in process (I) and uses a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the learning images.

[0025] In one example, the processor performs a process in process (I) in which it performs a first embedding operation on each of the training images to generate a first scene vector corresponding to each of the training images and clusters the first scene vectors to generate a first scene cluster, or a process in which it performs a kth embedding operation (where k is an integer of 1 or more) on each of the training images to generate a kth scene vector corresponding to each of the training images and clusters the kth scene vectors to generate a kth scene cluster, and then generates the individual types corresponding to the training images by referring to the first scene cluster or the kth scene cluster.

[0026] In one example, the processor, in process (I), checks the metadata contained in each of the training images, and further refers to the shooting time contained in each of the metadata to generate the individual type corresponding to each of the training images.

[0027] In one example, the processor, in process (I), (i) performs a specific embedding operation on each of the training images to generate a specific scene vector corresponding to each of the training images, clusters the specific scene vectors to generate a specific scene cluster, (ii) refers to the respective metadata contained in each of the training images to confirm the shooting time of each of the training images, and (iii) refers to the specific scene cluster and the shooting time to generate the individual type corresponding to each of the training images. [Effects of the Invention]

[0028] This invention makes it possible to select training data for training a deep learning model from a data pool evenly by type, without bias towards any particular type.

[0029] This invention enables the uniform selection of training data from a data pool that is suitable for various scenarios depending on the type of training data. [Brief explanation of the drawing]

[0030] The following drawings, attached for use in describing embodiments of the present invention, represent only a portion of the embodiments, and a person with ordinary skill in the art to which the present invention pertains (hereinafter referred to as "ordinary art") can obtain other drawings based on these drawings without performing any inventive work.

[0031] [Figure 1] Figure 1 is a schematic diagram showing a training data selection device for selecting training data for training a deep learning model according to one embodiment of the present invention. [Figure 2] Figure 2 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to the first embodiment of the present invention. [Figure 3] Figure 3 is a schematic diagram illustrating an example of generating individual types of training data in the first embodiment of the present invention. [Figure 4] Figure 4 is a schematic diagram illustrating another example of generating individual types of training data in the first embodiment of the present invention. [Figure 5] Figure 5 is a diagram illustrating a binary graph obtained by matching each of the training data with an individual type in the first embodiment of the present invention. [Figure 6a] Figure 6a is a schematic diagram illustrating the process of selecting training data by referring to a binary graph in the first embodiment of the present invention. [Figure 6b] Figure 6b is a schematic diagram illustrating the process of selecting training data by referring to a binary graph in the first embodiment of the present invention. [Figure 6c] Figure 6c is a schematic diagram illustrating the process of selecting training data by referring to a binary graph in the first embodiment of the present invention. [Figure 6d] Figure 6d is a schematic diagram illustrating the process of selecting training data by referring to a binary graph in the first embodiment of the present invention. [Figure 7]Figure 7 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to a second embodiment of the present invention. [Figure 8] Figure 8 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to a third embodiment of the present invention. [Figure 9] Figure 9 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to a fourth embodiment of the present invention. [Modes for carrying out the invention]

[0032] The detailed description of the present invention described below refers to the accompanying drawings illustrating specific embodiments in which the present invention may be carried out. These embodiments are described in sufficient detail to enable a person of the ordinary skill to carry out the present invention. It should be understood that the various embodiments of the present invention are different from one another but do not necessarily have to be mutually exclusive. For example, certain shapes, structures and characteristics described herein can be realized in other embodiments in relation to one embodiment without departing from the spirit and scope of the invention. It should also be understood that the position or arrangement of individual components in each disclosed embodiment can be modified without departing from the spirit and scope of the invention. Therefore, the detailed description described below should not be taken as restrictive, and the scope of the present invention is limited only by the accompanying claims, along with all equivalents to those claimed by those claims, if appropriately described. In the drawings, similar reference numerals refer to identical or similar functions across various aspects.

[0033] In the following, several preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that a person with ordinary skill in the art to which the present invention pertains can easily implement the present invention.

[0034] Figure 1 schematically shows a training data selection device for selecting training data for training a deep learning model according to one embodiment of the present invention. The training data selection device 100 may include a memory 110 that stores instructions for selecting training data for training a deep learning model, and a processor 120 that performs operations for selecting training data for training a deep learning model according to the instructions stored in the memory 110.

[0035] Specifically, the learning data sorting device 100 may, but is not limited to, achieve desired system performance by utilizing a combination of a typical computing device (e.g., a device that may include computer processors, memory, storage, input and output devices, and other conventional computing device components; electronic communication devices such as routers and switches; and electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions for using the computing device in a specific manner).

[0036] Furthermore, the processor 120 of the learning data sorting device 100 may include hardware components such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. The network controller 100 may also further include an operating system and software configurations for applications performing specific purposes.

[0037] However, this does not exclude the case in which the learning data sorting device 100 includes an integrated processor, which is a medium, processor, and memory integrated into one form for carrying out the present invention.

[0038] On the other hand, the processor 120 of the training data selection device 100 can execute a process to generate at least one individual type (attribute) corresponding to each of the many training data contained in all the training data stored in the data pool, according to instructions stored in the memory 110, and to generate a binary graph matching each of the many training data contained in all the training data with the individual type. Then, according to instructions stored in the memory 110, the processor 120 of the training data selection device 100 can refer to the binary graph and execute a process to select n training data for training the deep learning model such that the number of individual types matching each of the n training data is within a predetermined threshold deviation. The n is the target number of training data for training the deep learning model, and may be an integer representing multiple numbers.

[0039] A method for selecting training data for training a deep learning model will be described below, referring to the training data selection device according to one embodiment of the present invention configured as described above.

[0040] [First Embodiment] Figure 2 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to the first embodiment of the present invention. Referring to Figure 2, the first embodiment of the present invention will be described as follows.

[0041] First, the training data selection device 100 can acquire (S110) at least one individual type (attribute) corresponding to each of the many training data contained in all the training data stored in the data pool.

[0042] In this case, the individual types may include 1_1 individual types to 1_x individual types corresponding to the 1st type possessed by each of the numerous training data, and 2_1 individual types to 2_y individual types corresponding to the 2nd type possessed by each of the numerous training data. Each of x and y may be an integer of 1 or more. However, the present invention is not limited thereto, and may have three or more type types depending on the characteristics of the training data.

[0043] For example, if the learning data is text data, it can be composed of various individual types corresponding to various other types, such as type types related to genres corresponding to individual types like essays, novels, reports, travelogues, and letters; type types related to publication locations corresponding to individual types like blogs, newspapers, and social media posts; type types related to languages ​​corresponding to individual types like Korean, English, and German; and type types related to fields corresponding to individual types like economic terms, social terms, and mathematical terms.

[0044] As another example, if the training data is audio data, it can be composed of various individual types corresponding to various other types, such as type types related to time of day corresponding to individual types like day, night, dawn, and evening; type types related to location corresponding to individual types like city, countryside, mountains, and the sea; and type types related to the subject of the sound corresponding to individual types like car sounds, human sounds, bird songs, and wind sounds.

[0045] As yet another example, if the training data is image data, it can be composed of various individual types corresponding to various individual types, such as type types related to time of day corresponding to individual types such as day, night, evening, and dawn; type types related to weather corresponding to individual types such as sunny, cloudy, rainy, and foggy; and type types related to objects corresponding to individual types such as cars, people, and animals.

[0046] Furthermore, each of the numerous training data sets has at least one unique attribute, which can be determined by a labeler labeling each of the numerous training data sets against a predefined attribute, or by automatically extracting the unique attribute present in each of the numerous training data sets using various techniques such as AI (Artificial Intelligence) and clustering.

[0047] As an example, the learning data sorting device 100 can transmit a large amount of learning data to a labeler terminal, and use a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the large amount of learning data.

[0048] As another example, the training data selection device 100 can automatically generate various individual types corresponding to the various types of images that each of the training images, which constitute all the training data, possesses.

[0049] In other words, referring to Figure 3, the training data selection device 100 can perform a process to generate a first scene vector corresponding to each training image by performing a first embedding operation 10_1 on each training image and then clustering the first scene vectors 20 to generate a first scene cluster, or a process to generate a kth scene vector corresponding to each training image by performing a kth embedding operation 10_k on each training image and then clustering the kth scene vectors 20 to generate a kth scene cluster. The value of k may be an integer of 1 or more. The training data selection device 100 can then generate individual types corresponding to the training images by referring to the first scene cluster or the kth scene cluster.

[0050] In this case, the training data selection device 100 can utilize visual foundation models such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), and DINOv2 for the first to kth embedding operations. Each of the first embedding model for the first embedding operation and each of the kth embedding models for the kth embedding operation may be different embedding models that generate scene vectors corresponding to each of the first to kth types, which are different types of data. Furthermore, the training data selection device 100 can use various clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift to cluster the first to kth scene vectors.

[0051] Furthermore, while the above method involved embedding each training image and performing clustering to generate at least one unique type for each training image, it is also possible to generate unique types using metadata contained in each training image. For example, if k is 1, that is, if one embedding operation generates only unique types for one type corresponding to each training image, then it is also possible to use the metadata of each training image to generate even more unique types for type types different from the one type generated by the embedding operation.

[0052] In other words, it is possible to examine the metadata contained in each training image, and then refer to the shooting time contained in each metadata to generate a unique type corresponding to each training image.

[0053] For example, in order to generate individual categories based on time zone types such as daytime, nighttime, evening, and dawn, the learning data selection device 100 can refer to the shooting time of each learning image, group the learning images according to a predetermined time zone, and set individual categories related to the time zone for each group of grouped learning images.

[0054] In addition, when the learning data selection device 100 further references the capture time of each learning image to generate individual types corresponding to each learning image, it can also use only one specific embedding model, which is one of the first to kth embedding models, to generate individual types corresponding to one type using that specific embedding model, and individual types corresponding to another type using metadata.

[0055] In other words, the training data selection device 100 can perform a specific embedding operation on each training image to generate a specific scene vector corresponding to each training image, and then cluster the specific scene vectors to generate a specific scene cluster. The training data selection device 100 can then refer to the metadata contained in each training image to confirm the shooting time of each training image. Subsequently, the training data selection device 100 can also refer to the specific scene cluster and shooting time to generate an individual type corresponding to each training image.

[0056] As yet another example, the training data sorting device 100 can generate individual types corresponding to various type types for the entire image, but it can also generate individual types corresponding to objects contained within the image.

[0057] In other words, referring to Figure 4, the training data selection device 100 performs object detection 1 on each of the training images, which are all training data, to detect at least one object from each training image, generates a cropped image by cropping 2 the region corresponding to the bounding box of each detected object in each training image, performs an embedding operation 10 on each cropped image to generate an object vector corresponding to each cropped image, clusters the object vectors 20 to generate an object cluster, and generates individual types corresponding to the training images by referring to the object cluster. At this time, in order to perform object detection on each of the training images, the training data selection device 100 can use open-world object detection models such as Grounding DINO, OWL-ViT (short for Vision Transformer for Open-World Localization), and YOLO (You Only Look Once)-World.

[0058] In this case, the training data selection device 100 can also generate individual types corresponding to the training images by further referring to the scene clusters generated based on the above description, in addition to the object clusters.

[0059] In other words, the training data selection device 100 can perform a process to generate a first scene cluster by performing a first embedding operation on each training image to generate a first scene vector corresponding to each training image, and then clustering the first scene vectors. Alternatively, it can perform a process to generate a kth scene cluster by performing a kth embedding operation on each training image to generate a kth scene vector corresponding to each training image, and then clustering the kth scene vectors. The training data selection device 100 can then perform object detection on each training image to detect at least one object from each training image, generate a cropped image by cropping the region corresponding to the bounding box of each detected object in each training image, perform an embedding operation on each cropped image to generate an object vector corresponding to each cropped image, and then cluster the object vectors to generate an object cluster. Subsequently, the training data selection device 100 can generate individual types corresponding to the training images by referring to the first scene clusters to the kth scene clusters and the object clusters.

[0060] On the other hand, while the above method generated a cropped image after performing object detection on each of the training images, if each training image is labeled with its own ground truth, it is also possible to generate a cropped image by referencing the ground truth of each training image.

[0061] In other words, the training data selection device 100 can refer to the ground truth information contained in each training image to generate a cropped image by cropping the region corresponding to the bounding box of each object from each training image, perform an embedding operation on each cropped image to generate an object vector corresponding to each cropped image, cluster the object vectors to generate an object cluster, and refer to the object cluster to generate individual types corresponding to the training images.

[0062] In this case, the training data selection device 100 can also generate individual types corresponding to the training images by further referring to the scene clusters generated based on the above description, in addition to the object clusters.

[0063] In other words, the training data selection device 100 can perform a process to generate a first scene cluster by performing a first embedding operation on each training image to generate a first scene vector corresponding to each training image, and then clustering the first scene vectors. Alternatively, it can perform a process to generate a kth scene cluster by performing a kth embedding operation on each training image to generate a kth scene vector corresponding to each training image, and then clustering the kth scene vectors. The training data selection device 100 can then refer to the ground truth information contained in each training image to generate cropped images by cropping the region corresponding to the bounding box of each object from each training image, perform an embedding operation on each cropped image to generate an object vector corresponding to each cropped image, and then cluster the object vectors to generate an object cluster. Subsequently, the training data selection device 100 can refer to the first scene clusters, the kth scene clusters, and the object clusters to generate individual types corresponding to the training images.

[0064] Next, referring again to Figure 2, the learning data selection device 100 can generate a binary graph (S120) that matches each of the numerous learning data included in the total learning data with an individual type.

[0065] As an example, referring to Figure 5, the bisection graph matches the relationship between the training data group and the individual type group. It can be understood that training data 1 has individual type 1 and individual type 6, training data 2 has individual type 1, individual type 5, and individual type 6, training data 3 has individual type 2 and individual type 7, training data 4 has individual type 1, training data 5 has individual type 3, individual type 6, and individual type 8, training data 6 has individual type 4, and training data 7 has individual type 2, individual type 5, individual type 7, and individual type 8. On the other hand, Figure 5 exemplifies some of the training data and some of the individual types in the bisection graph.

[0066] Next, referring to Figure 2 again, the training data selection device 100, by referring to a binary graph, can select n training data for training the deep learning model (S130) such that the number of individual types matching the n training data is within a predetermined threshold deviation when selecting n training data from all training data to match individual types. The n is the target number of training data for training the deep learning model, and may be an integer representing multiple numbers.

[0067] In this case, if the individual types include 1st_1 individual types to 1st_x individual types corresponding to the first type each of the numerous training data have, and 2nd_1 individual types to 2nd_y individual types corresponding to the second type each of the numerous training data have, the training data selection device 100 can select n training data so that the number of 1st_1 individual types to 1st_x individual types corresponding to the first type and the number of 2nd_1 individual types to 2nd_y individual types corresponding to the second type that match the n training data are within a threshold deviation. In addition, it can select n training data so that the number of 1st_1 individual types or the number of 1st_x individual types are within a first threshold deviation, and the number of 2nd_1 individual types or the number of 2nd_y individual types are within a second threshold deviation. In other words, the training data selection device 100 can select training data so that the number of training data for each individual type is as equal as possible for all individual types of each type.

[0068] On the other hand, the training data selection device 100 can use at least some of the heuristic algorithms, optimization algorithms, etc., to select training data so that the number of data for each individual type is as equal as possible.

[0069] First, the process by which the training data selection device 100 selects training data using a heuristic algorithm is described below.

[0070] The training data selection device 100 can refer to a binary graph to check the number of corresponding individual types that match each of all training data, and select the specific training data with the largest number of corresponding individual types.

[0071] The learning data selection device 100 can then check the number of remaining individual types that match each of the total learning data, excluding the corresponding individual types, and perform the process of selecting other specific learning data with the largest number of remaining individual types.

[0072] In this way, by repeating the process of selecting the training data with the largest number of remaining individual types from the individual types that have been removed from the selected individual types, until all individual types have been selected, the training data selection device 100 can execute a cycle of selecting a portion of the training data that matches all individual types.

[0073] Subsequently, the training data selection device 100 can repeat the process of executing the cycle from the remaining training data, after excluding a portion of the training data selected to match all individual types from the total training data, until n training data are selected.

[0074] As an example, referring to Figures 6a to 6d, in Figure 6a, since training data 7 matches the four individual types with the most data, the training data selection device 100 can select training data 7 from training data 1 to training data 7.

[0075] Then, in the bisection graph of Figure 6a, if we remove individual types 2, 5, 7, and 8, which match the training data 7, individual types 1, 3, 4, and 6 remain, as shown in Figure 6b.

[0076] As shown in Figure 6b, since training data 1, training data 2, and training data 5 are matched to the two individual types with the highest number of occurrences, the training data selection device 100 can select training data 1, which is one of training data 1, training data 2, and training data 5. In this case, if there are multiple training data sets with the highest number of occurrences of the corresponding individual type, such as training data 1, training data 2, and training data 5, the training data selection device 100 can select one of the multiple training data sets, training data 1, training data 2, and training data 5, according to the first criterion, and select training data 1, which is the specific training data set. The first criterion can be set in various ways, such as selecting one training data set from among multiple training data sets using a random selection algorithm, or selecting one training data set based on the sequential ranking of multiple training data sets.

[0077] Then, in the bisection graph of Figure 6b, if we remove individual types 1 and 6, which match training data 1, individual types 3 and 4 remain, as shown in Figure 6c.

[0078] As a result, in Figure 6c, since training data 5 and training data 6 are matched to the single individual type with the largest number of occurrences, the training data selection device 100 can select one of the training data 5 from training data 5 and training data 6. In this case, if there are multiple remaining training data, such as training data 5 and training data 6, with the largest number of occurrences of the remaining individual type, the training data selection device 100 can select one of the multiple remaining training data, training data 5 and training data 6, according to the second criterion, and select the specific training data 5. The second criterion can be set in various ways, such as selecting one of the training data from multiple training data using a random selection algorithm, or selecting one of the training data based on the sequential ranking of multiple training data. The second criterion can be set in the same way as the first criterion described above, or it can be set differently.

[0079] Then, in the bisection graph of Figure 6c, if we remove individual type 3, which matches training data 5, individual type 4 remains as shown in Figure 6d.

[0080] As shown in Figure 6d, since training data 6 matches the single individual type with the most data, the training data selection device 100 selects training data 6, thereby enabling it to execute one cycle of selecting a portion of the training data that matches all of the individual types. Subsequently, the training data selection device 100 repeats the above cycle from the remaining training data, excluding the portion of training data selected from all training data—training data 1, training data 5, and training data 7—to select the desired number of n training data.

[0081] On the other hand, while the above described selecting some training data that matches all individual types, it is also possible to select training data by further referring to the type to which each individual type belongs. In other words, it is possible to select some training data that matches all individual types belonging to all type categories. For example, when selecting some training data from daytime and nighttime type categories and car and person object type categories, it is also possible to select some training data that matches equally when matching daytime individual types and nighttime individual types, matches equally when matching car individual types and person individual types within daytime individual types, and matches equally when matching both car individual types and person individual types within nighttime individual types.

[0082] Next, the process by which the training data selection device 100 selects training data using an optimization algorithm will be described as follows.

[0083] The training data selection device 100 refers to a binary graph and, through an optimization algorithm, selects a specific subset of training data consisting of a predetermined number of training data that includes all individual types, which has the fewest number of training data. It then calculates the remaining training data by removing a specific number of training data included in the specific subset, and repeats the process of selecting at least one other specific subset of the remaining training data consisting of a predetermined number of training data that includes all individual types, so that the number of selected training data is n or more. In this case, ensuring that the number of selected training data is n or more does not mean unconditionally ensuring that the number of selected training data is n or more, but rather that the total number of training data ultimately selected does not fall below n, depending on the number of specific subsets selected last. In other words, if the number of training data points selected so far is less than n, and the current total number of training data points, which is the sum of the numbers of the currently selected specific subsets, becomes n or greater, the process of selecting training data can be terminated, and the current total number of training data points can be generated as a training dataset for training the deep learning model.

[0084] As an example, the training data selection device 100 uses linear programming as its optimization algorithm to calculate the product of a PxQ binary matrix corresponding to P individual types and Q training data in a binary graph with a Q-dimensional vector representing the selection goodness-of-fit variable for each of the Q training data in each of the P individual types. The device then selects a specific subset from among the selection goodness-of-fit variables of the Q-dimensional vector that satisfy the minimum value of the sum of the selection goodness-of-fit variables, and the specific training data corresponding to a specific selection goodness-of-fit variable whose sum of the selection goodness-of-fit variables is 1 or more, and the selection goodness-of-fit variables in the Q-dimensional vector are between 0 and 1. The device then calculates the remaining training data by removing the specific training data included in the specific subset from the Q training data. The P-dimensional vector can represent the sum of the goodness-of-fit variables for each of the P individual types. The training data selection device 100 can then repeat the process of selecting at least one other specific subset from the remaining training data using linear programming, so that the number of selected training data is n or more.

[0085] In this case, linear programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, 0 ≤ x ≤ 1

[0086] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether or not the training data i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection goodness-of-fit variable for the training data i. Ax is the matrix product of A and x, and Ax[j] can represent the sum of the goodness-of-fit values ​​of the training data belonging to individual type j.

[0087] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the linear programming method.

[0088] The dual linear programming method, which is a modification of the linear programming method described above, can be expressed as follows, but the present invention is not limited thereto, and the dual linear programming method can be applied in various ways by modifying the constraints in the linear programming method described above.

[0089] Purpose: min 1 T x Condition: Bx ≧ b, x ≥ 0

[0090] At this time, B=[A;-I], (P+Q)xQ binary matrix, B[j,i]=A[j,i], B[P+i,i]= -1 for all i ∈ [1,2,···,Q], B[P+i,j]= 0 for all j≠i b=[1 P ;-1 P ], (P+Q)-dimensional vector, b[j]= 1 for all j ∈ [1,2,···,P], b[P+i]= -1 for all i ∈ [1,2,···,Q] It can be expressed as follows.

[0091] As another example, the training data selection device 100 uses integer programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training data in a binary graph and a Q-dimensional vector representing each of the selection variables for each of the Q training data in each of the P individual types. From the selection variables of the Q-dimensional vector that satisfy a selection quantity of 1 or more and a selection variable in the Q-dimensional vector being 0 or 1, the device can select a specific subset containing specific training data corresponding to a specific selection variable whose sum of selection variables is the minimum value, and calculate the remaining training data by removing the specific training data included in the specific subset from the Q training data. The P-dimensional vector can represent the selection quantity of training data belonging to each of the P individual types. The training data selection device 100 can then repeat the process of selecting at least one other specific subset from the remaining training data using integer programming until the number of selected training data is n or more.

[0092] In this case, integer programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, x[i] ∈ {0,1} for all i

[0093] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether or not the training data i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of the training data i. Ax is the matrix product of A and x, and Ax[j] can represent the number of training data selected that belong to individual type j.

[0094] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the integer programming method.

[0095] According to the first embodiment of the present invention, it becomes possible to select training data for each type so that they are as evenly distributed as possible for all individual types. Furthermore, if ground truth is not labeled on all training data stored in the data pool, ground truth only needs to be labeled on the training data that has been selected as evenly as possible for each type, thereby reducing the cost of generating training datasets.

[0096] [Second Example] Figure 7 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to a second embodiment of the present invention. Referring to Figure 7, the second embodiment of the present invention will be described as follows. In the following description, detailed explanations will be omitted for parts that can be easily understood from the description of the first embodiment.

[0097] First, the training data selection device 100 can acquire at least one individual type corresponding to each of the many training images stored in the data pool (S210).

[0098] In this case, the individual types may include 1_1 individual types to 1_x individual types corresponding to the first type type possessed by each of the many training images, and 2_1 individual types to 2_y individual types corresponding to the second type type possessed by each of the many training images. Each of x and y may be an integer of 1 or more. However, the present invention is not limited thereto, and may have three or more type types depending on the characteristics of the training images.

[0099] For example, it can be composed of various individual types corresponding to various other types, such as type types related to time of day corresponding to individual types such as day, night, evening, and dawn; type types related to weather corresponding to individual types such as sunny, cloudy, rainy, and foggy; and type types related to location corresponding to individual types such as city, countryside, and highway.

[0100] Furthermore, each of the numerous training images has at least one unique attribute, which can be determined by a labeler labeling each of the numerous training images against a predefined attribute, or by automatically extracting the unique attribute present in each of the numerous training images using various techniques such as AI (Artificial Intelligence) and clustering.

[0101] As an example, the training data sorting device 100 can transmit a large number of training images to a labeler terminal, and the labeler corresponding to the labeler terminal can generate at least one individual type corresponding to each of the large number of training images.

[0102] As another example, the training data selection device 100 can automatically generate various individual types corresponding to the various type types of the overall image that each of the many training images possesses.

[0103] In other words, referring again to Figure 3, the training data selection device 100 can perform a process to generate a first scene vector corresponding to each training image by performing a first embedding operation 10_1 on each training image and clustering the first scene vectors 20 to generate a first scene cluster, or a process to generate a kth scene vector corresponding to each training image by performing a kth embedding operation 10_k on each training image and clustering the kth scene vectors 20 to generate a kth scene cluster. The value of k may be an integer of 1 or more. The training data selection device 100 can then generate individual types corresponding to the training images by referring to the first scene cluster or the kth scene cluster.

[0104] In this case, the training data selection device 100 can utilize visual foundation models such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), and DINOv2 for the first to kth embedding operations. Each of the first embedding model for the first embedding operation and each of the kth embedding models for the kth embedding operation may be different embedding models that generate scene vectors corresponding to each of the first to kth types, which are different types of data. Furthermore, the training data selection device 100 can use various clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift to cluster the first to kth scene vectors.

[0105] Furthermore, while the above method involved embedding each training image and performing clustering to generate at least one unique type for each training image, it is also possible to generate unique types using metadata contained in each training image. For example, if k is 1, that is, if one embedding operation generates only unique types for one type corresponding to each training image, then it is also possible to use the metadata of each training image to generate even more unique types for type types different from the one type generated by the embedding operation.

[0106] In other words, it is possible to examine the metadata contained in each training image, and then refer to the shooting time contained in each metadata to generate a unique type corresponding to each training image.

[0107] For example, in order to generate individual categories based on time zone types such as daytime, nighttime, evening, and dawn, the learning data selection device 100 can refer to the shooting time of each learning image, group the learning images according to a predetermined time zone, and set individual categories related to the time zone for each group of grouped learning images.

[0108] In addition, when the learning data selection device 100 further references the capture time of each learning image to generate individual types corresponding to each learning image, it can also use only one specific embedding model, which is one of the first to kth embedding models, to generate individual types corresponding to one type using that specific embedding model, and individual types corresponding to another type using metadata.

[0109] In other words, the training data selection device 100 can perform a specific embedding operation on each training image to generate a specific scene vector corresponding to each training image, and then cluster the specific scene vectors to generate a specific scene cluster. The training data selection device 100 can then refer to the metadata contained in each training image to confirm the shooting time of each training image. Subsequently, the training data selection device 100 can also refer to the specific scene cluster and shooting time to generate an individual type corresponding to each training image.

[0110] Next, referring again to Figure 7, the learning data selection device 100 can generate a binary graph (S220) that matches each of the numerous learning images with an individual type.

[0111] Next, the training data selection device 100 can refer to a binary graph and execute a cycle (S230) to select a portion of the training images from among the many training images that match all individual types.

[0112] In other words, the training data selection device 100 can refer to a binary graph to check the number of corresponding individual types that match each of the many training images, and select a specific training image with the largest number of corresponding individual types.

[0113] The learning data selection device 100 can then check the number of remaining individual types that match each of the many learning images, excluding the corresponding individual types, and select the other specific learning image with the largest number of remaining individual types.

[0114] In this way, by repeating the process of selecting the training image with the largest number of remaining individual types from the individual types that have been removed from the selected individual types, until all individual types have been selected, the training data selection device 100 can execute a cycle of selecting a portion of training images that match all individual types.

[0115] In this case, if there are multiple training images with the largest number of corresponding individual types, the training data selection device 100 can select one of the multiple training images according to the first criterion to select a specific training image. If there are multiple remaining training images with the largest number of the remaining individual types, the device can select one of the multiple remaining training images according to the second criterion to select another specific training image.

[0116] Subsequently, the training data selection device 100 can repeat the process of executing the cycle from the remaining training images, excluding a portion of the selected training images that match all individual types from the large number of training images, until n training images have been selected (S240).

[0117] In this case, if the individual types include 1_1 individual types to 1_x individual types corresponding to the first type each of the many training images possesses, and 2_1 individual types to 2_y individual types corresponding to the second type each of the many training images possesses, the training data selection device 100 can ensure that the number of 1_1 individual types to 1_x individual types corresponding to the first type and the number of 2_1 individual types to 2_y individual types corresponding to the second type that match the n training images are within a threshold deviation. In addition, it can ensure that the number of 1_1 individual types to the number of 1_x individual types are within a first threshold deviation, and the number of 2_1 individual types to the number of 2_y individual types are within a second threshold deviation, by selecting n training images. In other words, the training data selection device 100 can select training images such that the number of training images for each individual type is as equal as possible for all individual types of each type. For example, when selecting some training images from daytime and nighttime type categories and car and person object type categories, it is also possible to select some training images that match equally when matching to individual daytime and nighttime type categories, match equally when matching to individual car and person type categories within the daytime type category, and match equally when matching to both individual car and person type categories within the nighttime type category.

[0118] [Third Embodiment] Figure 8 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to the third embodiment of the present invention. Referring to Figure 8, the third embodiment of the present invention will be described as follows. In the following description, detailed explanations will be omitted for parts that can be easily understood from the description of the first embodiment.

[0119] First, the training data selection device 100 can acquire at least one individual type corresponding to each of the many training images stored in the data pool (S310).

[0120] In this case, the individual types may include 1_1 individual types to 1_x individual types corresponding to the first type type possessed by each of the many training images, and 2_1 individual types to 2_y individual types corresponding to the second type type possessed by each of the many training images. Each of x and y may be an integer of 1 or more. However, the present invention is not limited thereto, and may have three or more type types depending on the characteristics of the training images.

[0121] For example, it can be composed of various individual types corresponding to various other types, such as type types related to time of day corresponding to individual types such as day, night, evening, and dawn; type types related to weather corresponding to individual types such as sunny, cloudy, rainy, and foggy; and type types related to location corresponding to individual types such as city, countryside, and highway.

[0122] Furthermore, each of the numerous training images has at least one unique attribute, which can be determined by a labeler labeling each of the numerous training images against a predefined attribute, or by automatically extracting the unique attribute present in each of the numerous training images using various techniques such as AI (Artificial Intelligence) and clustering.

[0123] As an example, the training data sorting device 100 can transmit a large number of training images to a labeler terminal, and the labeler corresponding to the labeler terminal can generate at least one individual type corresponding to each of the large number of training images.

[0124] As another example, the training data selection device 100 can automatically generate various individual types corresponding to the various type types of the overall image that each of the many training images possesses.

[0125] In other words, referring again to Figure 3, the training data selection device 100 can perform a process to generate a first scene vector corresponding to each training image by performing a first embedding operation 10_1 on each training image and clustering the first scene vectors 20 to generate a first scene cluster, or a process to generate a kth scene vector corresponding to each training image by performing a kth embedding operation 10_k on each training image and clustering the kth scene vectors 20 to generate a kth scene cluster. The value of k may be an integer of 1 or more. The training data selection device 100 can then generate individual types corresponding to the training images by referring to the first scene cluster or the kth scene cluster.

[0126] In this case, the training data selection device 100 can utilize visual foundation models such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), and DINOv2 for the first to kth embedding operations. Each of the first embedding model for the first embedding operation and each of the kth embedding models for the kth embedding operation may be different embedding models that generate scene vectors corresponding to each of the first to kth types, which are different types of data. Furthermore, the training data selection device 100 can use various clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift to cluster the first to kth scene vectors.

[0127] Furthermore, while the above method involved embedding each training image and performing clustering to generate at least one unique type for each training image, it is also possible to generate unique types using metadata contained in each training image. For example, if k is 1, that is, if one embedding operation generates only unique types for one type corresponding to each training image, then it is also possible to use the metadata of each training image to generate even more unique types for type types different from the one type generated by the embedding operation.

[0128] In other words, it is possible to examine the metadata contained in each training image, and then refer to the shooting time contained in each metadata to generate a unique type corresponding to each training image.

[0129] For example, in order to generate individual categories based on time zone types such as daytime, nighttime, evening, and dawn, the learning data selection device 100 can refer to the shooting time of each learning image, group the learning images according to a predetermined time zone, and set individual categories related to the time zone for each group of grouped learning images.

[0130] In addition, when the learning data selection device 100 further references the capture time of each learning image to generate individual types corresponding to each learning image, it can also use only one specific embedding model, which is one of the first to kth embedding models, to generate individual types corresponding to one type using that specific embedding model, and individual types corresponding to another type using metadata.

[0131] In other words, the training data selection device 100 can perform a specific embedding operation on each training image to generate a specific scene vector corresponding to each training image, and then cluster the specific scene vectors to generate a specific scene cluster. The training data selection device 100 can then refer to the metadata contained in each training image to confirm the shooting time of each training image. Subsequently, the training data selection device 100 can also refer to the specific scene cluster and shooting time to generate an individual type corresponding to each training image.

[0132] Next, referring again to Figure 8, the learning data selection device 100 can generate a binary graph (S320) that matches each of the numerous learning images with an individual type.

[0133] Next, the training data selection device 100 can refer to a binary graph and, through an optimization algorithm, select a specific subset from a predetermined number of training images that include all individual types, which has the fewest number of training images, and calculate the remaining training images after removing a specific number of training images included in the specific subset (S330).

[0134] As an example, the training data selection device 100 uses linear programming as an optimization algorithm to calculate the product of a PxQ binary matrix corresponding to P individual types and Q training images in a binary graph with a Q-dimensional vector representing the selection goodness-of-fit variable for each of the Q training images in each of the P individual types. The device then selects a specific subset from among the selection goodness-of-fit variables of the Q-dimensional vector that satisfy the condition that the sum of the selection goodness-of-fit variables in the Q-dimensional vector is 1 or greater, and that the selection goodness-of-fit variables in the Q-dimensional vector are between 0 and 1. The device then selects a specific subset that includes a specific training image corresponding to a specific selection goodness-of-fit variable whose sum of the selection goodness-of-fit variables is the minimum value, and calculates the remaining training images by excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector can represent the sum of the goodness-of-fit variables for each of the Q training images belonging to each of the P individual types.

[0135] In this case, linear programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, 0 ≤ x ≤ 1

[0136] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether the training image i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection goodness-of-fit variable for the training image i. Ax is the matrix product of A and x, and Ax[j] can represent the sum of the fitness scores of the training images belonging to individual type j.

[0137] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the linear programming method.

[0138] The dual linear programming method transformed from the above linear programming method is expressed as follows. However, the present invention is not limited thereto, and the dual linear programming method can be applied by various methods of transforming the constraints in the linear programming method.

[0139] Objective: min 1 T x Constraints: Bx ≧ b, x ≧ 0

[0140] At this time, B = [A; -I], (P + Q)xQ binary matrix, B[j,i] = A[j,i], B[P + i,i] = -1 for all i ∈ [1,2,···,Q], B[P + i,j] = 0 for all j≠i b = [1 P ;-1 P , (P + Q)-dimensional vector, b[j] = 1 for all j ∈ [1,2,···,P], b[P + i] = -1 for all i ∈ [1,2,···,Q] can be expressed as follows.

[0141] As another example, the training data selection device 100 uses integer programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in a binary graph and a Q-dimensional vector representing each of the selection variables for each of the Q training images in each of the P individual types. From among the selection variables in the Q-dimensional vector that satisfy a selection quantity of 1 or more and a selection variable in the Q-dimensional vector is 0 or 1, the device can select a specific subset that includes a specific training image corresponding to a specific selection variable whose sum of selection variables is the minimum value, and calculate the remaining training images by excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector can represent the selection quantity of training images belonging to each of the P individual types.

[0142] In this case, integer programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, x[i] ∈ {0,1} for all i

[0143] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether the training image i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of the training image i. Ax is the matrix product of A and x, and Ax[j] can represent the number of selected training images belonging to individual type j.

[0144] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the integer programming method.

[0145] Subsequently, the training data selection device 100 can repeat the process of selecting at least one other specific subset consisting of a predetermined number of training images containing all individual types from the remaining training images through an optimization algorithm until n training images for training the deep learning model have been selected (S340).

[0146] In this case, if the individual types include 1_1 individual types to 1_x individual types corresponding to the first type each of the many training images possesses, and 2_1 individual types to 2_y individual types corresponding to the second type each of the many training images possesses, the training data selection device 100 can ensure that the number of 1_1 individual types to 1_x individual types corresponding to the first type and the number of 2_1 individual types to 2_y individual types corresponding to the second type that match the n training images are within a threshold deviation. In addition, it can ensure that the number of 1_1 individual types to the number of 1_x individual types are within a first threshold deviation, and the number of 2_1 individual types to the number of 2_y individual types are within a second threshold deviation, by selecting n training images. In other words, the training data selection device 100 can select training images such that the number of training images for each individual type is as equal as possible for all individual types of each type. For example, when selecting some training images from daytime and nighttime type categories and car and person object type categories, it is also possible to select some training images that match equally when matching to individual daytime and nighttime type categories, match equally when matching to individual car and person type categories within the daytime type category, and match equally when matching to both individual car and person type categories within the nighttime type category.

[0147] [Fourth embodiment] Figure 9 is a schematic diagram illustrating a method for selecting training data for training a deep learning model according to the fourth embodiment of the present invention. Referring to Figure 9, the fourth embodiment of the present invention will be described as follows. In the following description, detailed explanations will be omitted for parts that can be easily understood from the description of the first embodiment.

[0148] First, the training data selection device 100 can acquire (S410) at least one individual type (attribute) corresponding to at least one object contained in each of the many training images stored in the data pool.

[0149] In this case, each object has at least one unique attribute, which can be determined by a labeler labeling each of the numerous training images against a predefined attribute, or by automatically extracting the unique attributes present in each of the numerous training images using various techniques such as AI (Artificial Intelligence) and clustering.

[0150] As an example, the training data sorting device 100 can transmit a large number of training images to a labeler terminal, and the labeler corresponding to the labeler terminal can generate at least one individual type corresponding to each object contained in each of the large number of training images.

[0151] As another example, the training data sorting device 100 can automatically generate at least one distinct type corresponding to each object contained in each of the numerous training images.

[0152] In other words, referring again to Figure 4, the training data selection device 100 performs object detection 1 on each training image to detect at least one object from each training image, generates a cropped image by cropping 2 the region corresponding to the bounding box of each detected object in each training image, performs an embedding operation 10 on each cropped image to generate an object vector corresponding to each cropped image, clusters the object vectors 20 to generate an object cluster, and generates individual types corresponding to the training images by referring to the object cluster. At this time, in order to perform object detection on each training image, the training data selection device 100 can use open-world object detection models such as Grounding DINO, OWL-ViT (short for Vision Transformer for Open-World Localization), and YOLO (You Only Look Once)-World.

[0153] In this case, the training data selection device 100 can also, in addition to object clusters, check the metadata contained in each training image, and further refer to the shooting time contained in each metadata to generate individual types corresponding to each training image.

[0154] On the other hand, while the above method generated a cropped image after performing object detection on each of the training images, if each training image is labeled with its own ground truth, it is also possible to generate a cropped image by referencing the ground truth of each training image.

[0155] In other words, the training data selection device 100 can refer to the ground truth information contained in each training image to generate a cropped image by cropping the region corresponding to the bounding box of each object from each training image, perform an embedding operation on each cropped image to generate an object vector corresponding to each cropped image, cluster the object vectors to generate an object cluster, and refer to the object cluster to generate individual types corresponding to the training images.

[0156] Next, referring again to Figure 9, the learning data selection device 100 can generate a binary graph (S420) that matches each of the numerous learning images with an individual type.

[0157] Next, the training data selection device 100 can refer to a binary graph and, through an optimization algorithm, select a specific subset from a predetermined number of training images that include all individual types, which has the fewest number of training images, and calculate the remaining training images after removing a specific number of training images included in the specific subset (S430).

[0158] As an example, the training data selection device 100 uses linear programming as an optimization algorithm to calculate the product of a PxQ binary matrix corresponding to P individual types and Q training images in a binary graph with a Q-dimensional vector representing the selection goodness-of-fit variable for each of the Q training images in each of the P individual types. The device then selects a specific subset from among the selection goodness-of-fit variables of the Q-dimensional vector that satisfy the condition that the sum of the selection goodness-of-fit variables in the Q-dimensional vector is 1 or greater, and that the selection goodness-of-fit variables in the Q-dimensional vector are between 0 and 1. The device then selects a specific subset that includes a specific training image corresponding to a specific selection goodness-of-fit variable whose sum of the selection goodness-of-fit variables is the minimum value, and calculates the remaining training images by excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector can represent the sum of the goodness-of-fit variables for each of the Q training images belonging to each of the P individual types.

[0159] In this case, linear programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, 0 ≤ x ≤ 1

[0160] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether the training image i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection goodness-of-fit variable for the training image i. Ax is the matrix product of A and x, and Ax[j] can represent the sum of the fitness scores of the training images belonging to individual type j.

[0161] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the linear programming method.

[0162] The dual linear programming method, which is a modification of the linear programming method described above, can be expressed as follows, but the present invention is not limited thereto, and the dual linear programming method can be applied in various ways by modifying the constraints in the linear programming method described above.

[0163] Purpose: min 1 T x Condition: Bx ≧ b, x ≥ 0

[0164] At this time, B=[A;-I], (P+Q)xQ binary matrix, B[j,i]=A[j,i], B[P+i,i]= -1 for all i ∈ [1,2,···,Q], B[P+i,j]= 0 for all j≠i b=[1 P ;-1 P ], (P+Q)-dimensional vector, b[j]= 1 for all j ∈ [1,2,···,P], b[P+i]= -1 for all i ∈ [1,2,···,Q] It can be expressed as follows.

[0165] As another example, the training data selection device 100 uses integer programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in a binary graph and a Q-dimensional vector representing the selection variable for each of the Q training images in each of the P individual types. From the selection variables of the Q-dimensional vector that satisfy a selection quantity of 1 or more and a selection variable of 0 or 1, the device can select a specific subset that includes a specific training image corresponding to a specific selection variable whose sum of selection variables is the minimum value, and calculate the remaining training images by excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector can represent the selection quantity of training images belonging to each of the P individual types.

[0166] In this case, integer programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, x[i] ∈ {0,1} for all i

[0167] In the above, A is a PxQ binary matrix, and A[j,i] can represent whether the training image i matches an individual type j. x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of the training image i. Ax is the matrix product of A and x, and Ax[j] can represent the number of selected training images belonging to individual type j.

[0168] Furthermore, the training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of the constraint merging, separation, and sign changing methods in the integer programming method.

[0169] Subsequently, the training data selection device 100 can repeat the process of selecting at least one other specific subset consisting of a predetermined number of training images containing all individual types from the remaining training images through an optimization algorithm until n training images for training the deep learning model have been selected (S440).

[0170] The embodiments of the present invention described above are implemented in the form of program instructions that can be executed through various computer components and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or they may be known and available to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0171] Although the present invention has been described above with specific details such as concrete components, and with limited embodiments and drawings, these are provided only to aid in a more overall understanding of the invention, and the invention is not limited to the above embodiments. A person with ordinary skill in the art to which the invention pertains can make various modifications and variations from this description.

[0172] Therefore, the concept of the present invention shall not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims described below shall also fall within the scope of the concept of the present invention. [Explanation of symbols]

[0173] 100: Training data sorting device 110: Memory 120: Processor

Claims

1. In a method for selecting training data for training a deep learning model, (a) A learning data selection device generates at least one attribute corresponding to each of the numerous learning images stored in the data pool, and generates a binary graph matching each of the numerous learning images with the attribute; and (b) The training data selection device (i) refers to the binary graph to determine the number of corresponding individual types that match each of the individual types for each of the training images, and selects a specific training image with the largest number of corresponding individual types; (ii) refers to the remaining individual types, excluding the corresponding individual types, to determine the number of the remaining corresponding individual types that match each of the training images, and selects another specific training image with the largest number of the remaining corresponding individual types, repeating this process to select a portion of training images that match all of the individual types; and repeating the process of performing the cycle from the remaining training images, excluding the portion of training images selected from the large number of training images, until n training images for training the deep learning model (where n is the target number of training images for training the deep learning model and is an integer representing a number of multiple images) are selected; A method that includes this.

2. In step (b) above, The individual types include a first_1 individual type to a first_x individual type (where x is an integer of 1 or more) corresponding to a first type type that each of the learning images has, and a second_1 individual type to a second_y individual type (where y is an integer of 1 or more) corresponding to a second type type that each of the learning images has, The method according to claim 1, wherein the learning data selection device selects the n learning images such that the number of 1_1 individual types to 1_x individual types corresponding to the first type and the number of 2_1 individual types to 2_y individual types corresponding to the second type that match the n learning images are within the threshold deviation, the number of 1_1 individual types to 1_x individual types is within the first threshold deviation, and the number of 2_1 individual types to 2_y individual types is within the second threshold deviation.

3. In step (b) above, The method according to claim 1, wherein if there are multiple learning images with the largest number of the corresponding individual types, the learning data selection device selects one of the multiple learning images according to a first criterion to select the specific learning image, and if there are multiple remaining learning images with the largest number of the remaining individual types, it selects one of the multiple remaining learning images according to a second criterion to select the other specific learning image.

4. In step (a) above, The method according to claim 1, wherein the learning data selection device transmits the learning images to a labeler terminal, and a labeler corresponding to the labeler terminal generates at least one individual type corresponding to each of the learning images.

5. In step (a) above, The method according to claim 1, wherein the training data selection device performs a process of generating a first scene vector corresponding to each of the training images by performing a first embedding operation on each of the training images and clustering the first scene vectors to generate a first scene cluster, or performs a process of generating a kth scene vector corresponding to each of the training images by performing a kth embedding operation (where k is an integer of 1 or more) on each of the training images and clustering the kth scene vectors to generate a kth scene cluster, and generates the individual type corresponding to the training images by referring to the first scene cluster to the kth scene cluster.

6. In step (a) above, The method according to claim 1, wherein the learning data selection device checks the metadata contained in each of the learning images, and further refers to the shooting time contained in each of the metadata to generate the individual type corresponding to each of the learning images.

7. In step (a) above, The method according to claim 1, wherein the training data selection device (i) performs a specific embedding operation on each of the training images to generate a specific scene vector corresponding to each of the training images, clusters the specific scene vectors to generate a specific scene cluster, (ii) refers to the respective metadata contained in each of the training images to confirm the shooting time of each of the training images, and (iii) refers to the specific scene cluster and the shooting time to generate the individual type corresponding to each of the training images.

8. In a training data selection device for selecting training data for training a deep learning model, Memory containing instructions for selecting training data for training a deep learning model; and A processor that performs an operation to select training data for training the deep learning model in accordance with the instructions stored in the memory; Includes, The processor performs the following processes: (I) generating at least one attribute corresponding to each of the numerous training images stored in the data pool, and generating a binary graph matching each of the numerous training images with the attribute; and (II) (i) referring to the binary graph, confirming the number of corresponding attributes that match each of the attribute types for each of the training images, selecting a specific training image with the largest number of corresponding attribute types, and (ii) matching each of the remaining attribute types, excluding the corresponding attribute types, for each of the training images. A learning data selection device that checks the number of each of the remaining corresponding individual types, repeats the process of selecting the other specific learning image with the largest number of the remaining corresponding individual types, executes a cycle to select some learning images that match all of the individual types, and repeats the process of executing the above cycle from the remaining learning images excluding the selected some learning images from the large number of learning images until n learning images for training the deep learning model (where n is the target number of learning images for training the deep learning model and is an integer representing multiple numbers) are selected.

9. The aforementioned processor, In the process described in (II) above, the individual type includes a first_1 individual type to a first_x individual type (where x is an integer of 1 or more) corresponding to the first type type that each of the learning images has, and a second_1 individual type to a second_y individual type (where y is an integer of 1 or more) corresponding to the second type type that each of the learning images has, The learning data selection device according to claim 8, which selects the n learning images such that the number of individual types from the first type to the first x type that match the first type and the number of individual types from the second type to the second y type that match the n learning images are within the threshold deviation, the number of individual types from the first type to the first x type is within the first threshold deviation, and the number of individual types from the second type to the second y type is within the second threshold deviation.

10. The aforementioned processor, In the process of (II) described above, if there are multiple learning images that have the largest number of the corresponding individual types, one of the multiple learning images is selected according to the first criterion to select the specific learning image, and if there are multiple remaining learning images that have the largest number of the remaining individual types, one of the multiple remaining learning images is selected according to the second criterion to select the other specific learning image, as described in claim 8.

11. The aforementioned processor, The learning data sorting apparatus according to claim 8, wherein in the process of (I) above, the learning images are transmitted to a labeler terminal, and a labeler corresponding to the labeler terminal generates at least one individual type corresponding to each of the learning images.

12. The aforementioned processor, The learning data sorting device according to claim 8, wherein in the process of (I) above, a process is performed in which a first embedding operation is performed on each of the learning images to generate a first scene vector corresponding to each of the learning images, and the first scene vectors are clustered to generate a first scene cluster, or a process is performed in which a kth embedding operation (where k is an integer of 1 or more) is performed on each of the learning images to generate a kth scene vector corresponding to each of the learning images, and the kth scene vectors are clustered to generate a kth scene cluster, and the individual types corresponding to the learning images are generated by referring to the first scene cluster or the kth scene cluster.

13. The aforementioned processor, The learning data sorting apparatus according to claim 8, wherein in the process of (I) above, the metadata contained in each of the learning images is checked, and the shooting time contained in each of the metadata is further referenced to generate the individual type corresponding to each of the learning images.

14. The aforementioned processor, The learning data sorting device according to claim 8, wherein in the process of (I) above, (i) a specific embedding operation is performed on each of the learning images to generate a specific scene vector corresponding to each of the learning images, the specific scene vectors are clustered to generate a specific scene cluster, (ii) the shooting time of each of the learning images is confirmed by referring to the respective metadata contained in each of the learning images, and (iii) the individual type corresponding to each of the learning images is generated by referring to the specific scene cluster and the shooting time.

Images