METHOD FOR SELECTING TRAINING DATA TO TRAIN A DEEP LEARNING MODEL AND TRAINING DATA SELECTING DEVICE USING THE SAME

Abstract

A method for selecting training data for training a deep learning model is provided. [Solution] The method includes the steps of: a training data selection device generating at least one individual type corresponding to at least one object included in each of a large number of training images stored in a data pool, and generating a bipartite graph matching each of the large number of training images with the individual type; and a training data selection device referencing the bipartite graph and using an optimization algorithm to select a specific subset with the fewest number of training images from among subsets consisting of a predetermined number of training images that include all of the individual types, calculating the remaining training images excluding a specific number of training images included in the specific subset, and repeating this process of selecting at least one other specific subset consisting of a predetermined number of training images that include all of the individual types from the remaining training images using an optimization algorithm until n training images are selected.

Description

[Technical Field]

[0001] The present invention relates to a method for uniformly selecting training data for training a deep learning model from all training data stored in a data pool without data bias or variation, and a training data selection device using the same. [Background technology]

[0002] Generally, deep learning models recognize complex patterns in images, text, sound, and other data to generate accurate insights and predictions, and are applied in a variety of fields, including computer vision, speech recognition, autonomous vehicles, robotics, natural language processing, and medical image analysis.

[0003] In order for such deep learning models to perform their intended tasks accurately, they must be trained using large amounts of training data.

[0004] Conventional methods for selecting training data for training deep learning models from a collected data pool include random sampling, which selects a target number of training data from all the training data stored in the data pool, and vector quantization, which clusters and groups vectors representing all the training data generated by embedding extraction, and selects representative values ​​for each group of grouped vectors.

[0005] For example, Patent Document 1 discloses a method and apparatus for generating learning data required to train an animated character based on deep learning, Patent Document 2 discloses a similarity-based clustering apparatus and method using deep learning techniques, Patent Document 3 discloses an apparatus and method for training a deep learning classification model, and Patent Document 4 discloses a system and method for training a machine learning model using active learning.

[0006] However, when the training data is selected using the conventional method, there is a problem in that bias and variation occurs according to type.

[0007] As an example, if a data pool contains 1 million training images, of which 70% are training images related to sunny weather, 20% are training images related to cloudy weather, 5% are training images related to foggy weather, and 5% are training images related to snowy and / or rainy weather, and 10,000 training images are randomly sampled, only about 500 training images will be selected probabilistically from the total of 50,000 training images related to snowy and / or rainy weather, resulting in bias and variation in the training images selected according to weather-related type.

[0008] Furthermore, when selecting training images using vector quantization, it is possible to somewhat mitigate bias and variation in the types of training images selected by embedding extraction and clustering, but it is not possible to fundamentally prevent problems related to bias and variation in data.

[0009] Therefore, the present applicant proposes a method for uniformly selecting learning data for training a deep learning model from all learning data stored in a data pool by type without bias or variation. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] U.S. Patent No. 1,110,6942 [Patent Document 2] Korean Patent Publication No. 10-2023-0068941 [Patent Document 3] Patent No. 7225614 [Patent Document 4] U.S. Patent No. 11,663,409 Summary of the Invention [Problem to be solved by the invention]

[0011] An object of the present invention is to solve all of the above-mentioned problems.

[0012] Another object of the present invention is to enable the training data for training a deep learning model to be selected evenly by type from a data pool without being biased toward a particular type.

[0013] A further object of the present invention is to enable uniform selection of training data suitable for various scenarios according to the type of training data from a data pool. [Means for solving the problem]

[0014] According to one embodiment of the present invention, there is provided a method for selecting training data for training a deep learning model, the method including: (a) a training data selection device generating at least one individual attribute corresponding to at least one object included in each of a number of training images stored in a data pool, and generating a bipartite graph matching each of the plurality of training images with the individual attribute; and (b) the training data selection device (i) referring to the bipartite graph, selecting a specific subset having the fewest number of training images from subsets consisting of a predetermined number of training images that include all of the individual attributes through an optimization algorithm, and calculating remaining training images excluding a specific number of training images included in the specific subset, and (ii) repeating the process of selecting at least one other specific subset consisting of a predetermined number of training images that include all of the individual attributes through the optimization algorithm from the remaining training images until n training images for training a deep learning model (n is a target number of training images for training the deep learning model and is an integer representing a plural number) are selected.

[0015] In one example, in the step (b), the training data selection device performs linear programming. a PxQ binary matrix corresponding to P individual types and Q training images in the bipartite graph, calculated by using linear programming, and a Q-dimensional vector representing each selection fitness variable of the Q training images in each of the P individual types, to generate a P-dimensional vector (the P-dimensional vector represents the sum of the fitnesses of the Q training images belonging to each of the P individual types), the sum of the fitnesses being 1 or more, and the selection fitness variables in the Q-dimensional vector being between 0 and 1, inclusive; selecting the specific subset including a specific training image corresponding to a specific selection fitness variable having a minimum sum of the selection fitness variables from among the selection fitness variables of the Q training images; calculating the remaining training images by excluding the specific training images included in the specific subset from the Q training images; and repeating the process of selecting at least one other specific subset using the linear programming for the remaining training images until the number of selected training images is equal to or greater than n.

[0016] In one example, in step (b), the training data selection device selects the specific subset using a dual linear programming method that applies at least one of merging, separating, and sign-changing constraints in linear programming.

[0017] In one example, in step (b), the training data selection device uses integer programming to calculate a matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in the bipartite graph and a Q-dimensional vector representing each selection variable of the Q training images in each of the P individual types to generate a P-dimensional vector (the P-dimensional vector represents the selection quantity of training images belonging to each of the P individual types), where the selection quantity in the P-dimensional vector is 1 or more and the selection variable in the Q-dimensional vector satisfies 0 or 1. The device then selects the specific subset including specific training images corresponding to a specific selection variable having a minimum sum of the selection variables from among the selection variables of the Q-dimensional vectors, excluding the specific training images included in the specific subset from the Q training images, calculates the remaining training images by excluding the specific training images included in the specific subset, and repeats the process of selecting at least one other specific subset using the integer programming for the remaining training images until the number of selected training images is equal to or greater than n.

[0018] In one example, in step (b), the training data selection device selects the specific subset using dual integer programming that applies at least one of merging, separating, and sign-changing constraints in the integer programming.

[0019] In one example, in step (a), the training data selection device transmits the training images to a labeler terminal, and uses a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the objects included in each of the training images.

[0020] In one example, in step (a), the training data selection device performs object detection on each of the training images to detect at least one object from each of the training images, generates cropped images by cropping areas corresponding to bounding boxes of each of the detected objects in each of the training images, performs an embedding operation on each of the cropped images to generate object vectors corresponding to each of the cropped images, clusters the object vectors to generate object clusters, and generates the individual types corresponding to the training images by referring to the object clusters.

[0021] In one example, in step (a), the training data selection device refers to the ground truth information included in each of the training images, generates cropped images by cropping areas corresponding to bounding boxes of each object from each of the training images, performs an embedding operation on each of the cropped images to generate object vectors corresponding to each of the cropped images, clusters the object vectors to generate object clusters, and generates the individual types corresponding to the training images by referring to the object clusters.

[0022] In one example, in step (a), the training data selection device checks each piece of meta information included in each of the training images, and further refers to each shooting time included in the meta information to generate the individual type corresponding to each of the training images.

[0023] In one example, in step (b), the specific individual types include 1_1 specific individual types through 1_x specific individual types (where x is an integer greater than or equal to 1) corresponding to the object and 2_1 specific individual types through 2_y specific individual types (where y is an integer greater than or equal to 1) corresponding to the shooting time, and the training data selection device selects the n training images so that the number of 1_1 specific individual types through 1_x specific individual types corresponding to the object and the number of 2_1 specific individual types through 2_y specific individual types corresponding to the shooting time that match the n training images are within the threshold deviation, so that the number of 1_1 specific individual types through 1_x specific individual types is within a first threshold deviation, and so that the number of 2_1 specific individual types through 2_y specific individual types is within a second threshold deviation.

[0024] According to one embodiment of the present invention, a training data selection device for selecting training data for training a deep learning model includes: a memory storing instructions for selecting training data for training a deep learning model; and a processor that executes an operation for selecting training data for training the deep learning model in accordance with the instructions stored in the memory; wherein the processor performs the following steps: (I) generating at least one individual attribute corresponding to at least one object included in each of a number of training images stored in a data pool, and generating a bipartite graph matching each of the number of training images with the individual attribute; and (I I) A training data selection device is provided that (i) refers to the bipartite graph and selects, through an optimization algorithm, a specific subset having the smallest number of training images from among subsets consisting of a predetermined number of training images that include all of the individual types, and calculates remaining training images excluding a specific number of training images included in the specific subset; and (ii) selects, from the remaining training images, at least one other specific subset consisting of a predetermined number of training images that include all of the individual types through the optimization algorithm, repeating this process until n training images (n is a target number of training images for training the deep learning model and is an integer representing a plural number) for training the deep learning model are selected.

[0025] In one example, the processor performs linear programming in the process (II). a PxQ binary matrix corresponding to P individual types and Q training images in the bipartite graph, calculated by using linear programming, and a Q-dimensional vector representing each selection fitness variable of the Q training images in each of the P individual types, to generate a P-dimensional vector (the P-dimensional vector represents the sum of the fitnesses of the Q training images belonging to each of the P individual types), the sum of the fitnesses being 1 or more, and the selection fitness variables in the Q-dimensional vector being between 0 and 1, inclusive; selecting the specific subset including a specific training image corresponding to a specific selection fitness variable having a minimum sum of the selection fitness variables from among the selection fitness variables of the Q training images; calculating the remaining training images by excluding the specific training images included in the specific subset from the Q training images; and repeating the process of selecting at least one other specific subset for the remaining training images using the linear programming until the number of selected training images is equal to or greater than n.

[0026] In one example, in the process (II), the processor selects the specific subset using a dual linear programming method that applies at least one of merging, separating, and changing the sign of constraints in linear programming.

[0027] In one example, in the process (II), the processor uses integer programming to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in the bipartite graph and a Q-dimensional vector representing each selection variable of the Q training images in each of the P individual types to generate a P-dimensional vector (the P-dimensional vector represents the selection quantity of training images belonging to each of the P individual types), where the selection quantity in the P-dimensional vector is 1 or more and the selection variable in the Q-dimensional vector satisfies 0 or 1. From the selection variables of the Q-dimensional vector, the selection quantity is 1 or more, and the specific subset includes specific training images corresponding to a specific selection variable having a minimum sum of the selection variables; the processor calculates the remaining training images by excluding the specific training images included in the specific subset from the Q training images; and repeats the process of selecting at least one other specific subset using the linear programming for the remaining training images until the number of selected training images is equal to or greater than n.

[0028] In one example, in the (II) process, the processor selects the specific subset using dual integer programming that applies at least one of merging, separating, and sign-changing constraints in the integer programming.

[0029] In one example, in the process (I), the processor transmits the training images to a labeler terminal, and uses a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the objects included in each of the training images.

[0030] In one example, in the process (I), the processor performs object detection on each of the training images to detect at least one object from each of the training images, generates cropped images by cropping areas corresponding to the bounding boxes of each of the detected objects in each of the training images, performs an embedding operation on each of the cropped images to generate object vectors corresponding to each of the cropped images, clusters the object vectors to generate object clusters, and generates the individual types corresponding to the training images by referring to the object clusters.

[0031] In one example, in the process (I), the processor refers to the ground truth information included in each of the training images, generates cropped images by cropping areas corresponding to the bounding boxes of each object from each of the training images, performs an embedding operation on each of the cropped images to generate object vectors corresponding to each of the cropped images, clusters the object vectors to generate object clusters, and generates the individual types corresponding to the training images by referring to the object clusters.

[0032] In one example, in the process (I), the processor checks the meta information included in each of the training images, and further refers to the respective shooting times included in the meta information to generate the individual types corresponding to each of the training images.

[0033] In one example, in the (II) process, the processor selects the n training images including 1_1 specific individual types through 1_x specific individual types (where x is an integer greater than or equal to 1) corresponding to the object and 2_1 specific individual types through 2_y specific individual types (where y is an integer greater than or equal to 1) corresponding to the shooting time, and so that the number of 1_1 specific individual types through 1_x specific individual types corresponding to the object and the number of 2_1 specific individual types through 2_y specific individual types corresponding to the shooting time that match the n training images are within the threshold deviation, so that the number of 1_1 specific individual types through 1_x specific individual types is within a first threshold deviation, and so that the number of 2_1 specific individual types through 2_y specific individual types is within a second threshold deviation. [Effects of the Invention]

[0034] The present invention makes it possible to select training data for training a deep learning model from a data pool evenly by type, without bias toward a particular type.

[0035] The present invention makes it possible to uniformly select training data suitable for various scenarios according to the type of training data from a data pool. [Brief explanation of the drawings]

[0036] The following drawings attached for use in explaining embodiments of the present invention are only a part of the embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter referred to as "ordinary engineer") can derive other drawings based on these drawings without performing any inventive work.

[0037] [Figure 1] FIG. 1 is a diagram schematically illustrating a training data selection device for selecting training data for training a deep learning model according to an embodiment of the present invention. [Figure 2]FIG. 2 is a diagram illustrating a method for selecting training data for training a deep learning model according to a first embodiment of the present invention. [Figure 3] FIG. 3 is a diagram illustrating an example of generating an individual type of learning data in the first embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating another example of generating an individual type of learning data in the first embodiment of the present invention. [Figure 5] FIG. 5 is a diagram illustrating an example of a bipartite graph obtained by matching each of the training data with an individual type in the first embodiment of the present invention. [Figure 6a] FIG. 6a is a diagram illustrating a process of selecting training data by referring to a bipartite graph in the first embodiment of the present invention. [Figure 6b] FIG. 6b is a diagram illustrating a process of selecting training data by referring to a bipartite graph in the first embodiment of the present invention. [Figure 6c] FIG. 6c is a diagram illustrating a process of selecting training data by referring to a bipartite graph in the first embodiment of the present invention. [Figure 6d] FIG. 6d is a diagram illustrating a process of selecting training data by referring to a bipartite graph in the first embodiment of the present invention. [Figure 7] FIG. 7 is a diagram schematically illustrating a method for selecting training data for training a deep learning model according to a second embodiment of the present invention. [Figure 8] FIG. 8 is a diagram schematically illustrating a method for selecting training data for training a deep learning model according to a third embodiment of the present invention. [Figure 9] FIG. 9 is a diagram schematically illustrating a method for selecting training data for training a deep learning model according to a fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0038] The following detailed description of the present invention refers to the accompanying drawings, which show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable one of ordinary skill in the art to practice the invention. It should be understood that various embodiments of the present invention, although different from one another, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein, in connection with one embodiment, can be implemented in other embodiments without departing from the spirit and scope of the invention. It should also be understood that the location or arrangement of individual components in each disclosed embodiment may be modified without departing from the spirit and scope of the invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention is limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly set forth. In the drawings, like reference numerals refer to the same or similar functionality throughout the various aspects.

[0039] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following, a number of preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice the present invention.

[0040] FIG. 1 is a schematic diagram of a training data selection device 100 for selecting training data for training a deep learning model according to one embodiment of the present invention. The training data selection device 100 includes a memory 110 storing instructions for selecting training data for training a deep learning model, and a processor 120 that performs operations for selecting training data for training the deep learning model in accordance with the instructions stored in the memory 110.

[0041] Specifically, the training data selection device 100 may typically utilize, but is not limited to, a combination of computing devices (e.g., devices that may include a computer processor, memory, storage, input and output devices, and other conventional computing device components; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause a computing device to function in a particular manner) to achieve the desired system performance.

[0042] The processor 120 of the training data selection device 100 may include hardware components such as an MPU (Micro Processing Unit) or a CPU (Central Processing Unit), a cache memory, a data bus, etc. The network controller 100 may also include software components such as an operating system and an application that executes a specific purpose.

[0043] However, this does not exclude the case where the training data selection device 100 includes an integrated processor in which a medium, a processor, and a memory for implementing the present invention are integrated.

[0044] Meanwhile, the processor 120 of the training data selection device 100 may execute a process of generating at least one individual type (attribute) corresponding to each of a plurality of pieces of training data included in all the training data stored in the data pool, and generating a bipartite graph matching each of the plurality of pieces of training data included in all the training data with the individual type, according to instructions stored in the memory 110. The processor 120 of the training data selection device 100 may execute a process of selecting n pieces of training data for training a deep learning model such that the number of each individual type matching the n pieces of training data is within a predetermined threshold deviation, when selecting n pieces of training data matching the individual types from all the training data, by referring to the bipartite graph, according to instructions stored in the memory 110. The n is a target number of training data for training the deep learning model, and may be an integer representing a plural number.

[0045] With reference to the thus configured training data selection device according to an embodiment of the present invention, a method for selecting training data for training a deep learning model will be described below.

[0046] [First Example] FIG. 2 is a diagram illustrating a method for selecting training data for training a deep learning model according to a first embodiment of the present invention. The first embodiment of the present invention will be described below with reference to FIG. 2.

[0047] First, the training data selection device 100 can acquire at least one individual attribute corresponding to each of a plurality of training data included in all the training data stored in the data pool (S110).

[0048] In this case, the individual types may include 1_1 individual types through 1_x individual types corresponding to the first type possessed by each of the plurality of learning data, and 2_1 individual types through 2_y individual types corresponding to the second type possessed by each of the plurality of learning data. Each of x and y may be an integer greater than or equal to 1. However, the present invention is not limited thereto, and three or more individual types may be provided depending on the characteristics of the learning data.

[0049] For example, if the learning data is text data, it can be configured into various individual types corresponding to various type types, such as type types related to genres corresponding to individual types such as essays, novels, reports, travelogues, letters, etc., type types related to publication locations corresponding to individual types such as blogs, newspapers, and SNS posts, type types related to languages ​​corresponding to individual types such as Korean, English, and German, and type types related to fields corresponding to individual types such as economic terms, social terms, and mathematical terms.

[0050] As another example, if the training data is audio data, it can be configured into various individual types corresponding to various type types, such as type types related to time periods corresponding to individual types such as day, night, dawn, and dusk, type types related to places corresponding to individual types such as city, countryside, mountain, and sea, and type types related to sound subjects corresponding to individual types such as car sounds, human sounds, bird sounds, and wind sounds.

[0051] As another example, if the learning data is image data, it can be configured into various individual types corresponding to various type types, such as type types related to time periods corresponding to individual types such as day, night, dusk, and dawn, type types related to weather corresponding to individual types such as sunny, cloudy, rainy, and foggy, and type types related to objects corresponding to individual types such as automobiles, people, and animals.

[0052] In addition, at least one individual type (attribute) corresponding to each of the plurality of training data can be extracted automatically by labeling each of the plurality of training data to a predefined type by a labeler, or by utilizing various techniques such as AI (Artificial Intelligence) and clustering.

[0053] For example, the training data selection device 100 may transmit a plurality of training data to a labeler terminal, and generate at least one individual type corresponding to each of the plurality of training data using a labeler corresponding to the labeler terminal.

[0054] As another example, the training data selection device 100 can automatically generate various individual patterns corresponding to various pattern types for the entire image contained in each of the training images, which are all the training data.

[0055] 3, the training data selection device 100 may perform a process of performing a first embedding operation 10_1 on each training image to generate a first scene vector corresponding to each training image and clustering 20 the first scene vector to generate a first scene cluster, or a process of performing a k-th embedding operation 10_k on each training image to generate a k-th scene vector corresponding to each training image and clustering 20 the k-th scene vector to generate a k-th scene cluster. The k may be an integer greater than or equal to 1. The training data selection device 100 may generate individual types corresponding to the training images by referring to the first scene cluster to the k-th scene cluster.

[0056] In this case, the training data selection device 100 may use a visual foundation model such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), or DINOv2 for the first through k-th embedding operations, and the first through k-th embedding models for the first through k-th embedding operations may be different embedding models that generate scene vectors corresponding to the first through k-th types, which are different types. The training data selection device 100 may use various clustering algorithms, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift, to cluster the first through k-th scene vectors.

[0057] In addition, although the above description describes performing an embedding operation on each of the training images and clustering to generate at least one individual type for each of the training images, individual types can also be generated using meta information included in each of the training images. For example, when k is 1, that is, when only an individual type for one type corresponding to each of the training images is generated by one embedding operation, it is also possible to generate additional individual types for type types different from the one type generated by the embedding operation by using meta information of each of the training images.

[0058] That is, it is possible to check the meta information included in each of the training images and further refer to the shooting time included in each of the meta information to generate an individual type corresponding to each of the training images.

[0059] For example, to generate individual types according to time period types such as day, night, evening, and dawn, the training data selection device 100 may refer to the shooting time of each training image, group the training images according to a predetermined time period, and set an individual type associated with the time period for each group of grouped training images.

[0060] In addition, when generating individual types corresponding to each of the training images by further referring to the shooting time of each of the training images, the training data selection device 100 can also use only a specific embedding model, which is any one of the first embedding model to the kth embedding model, to generate an individual type corresponding to one of the types using the specific embedding model and an individual type corresponding to another type using meta information.

[0061] That is, the training data selection device 100 can perform a specific embedding operation on each training image to generate a specific scene vector corresponding to each training image, and then cluster the specific scene vector to generate a specific scene cluster.The training data selection device 100 can then check the shooting time of each training image by referring to the respective meta information included in each training image.The training data selection device 100 can then generate an individual type corresponding to each training image by referring to the specific scene cluster and the shooting time.

[0062] As yet another example, the training data selection device 100 may generate individual types corresponding to objects contained in an image, rather than generating various individual types corresponding to various types for the entire image.

[0063] 4, the training data selection device 100 performs object detection 1 on each of the training images, which are all training data, to detect at least one object from each of the training images, generates cropped images by cropping 2 regions corresponding to bounding boxes of each of the detected objects in each of the training images, performs embedding operation 10 on each of the cropped images to generate object vectors corresponding to each of the cropped images, clusters 20 the object vectors to generate object clusters, and generates individual types corresponding to the training images by referring to the object clusters. In this case, to perform object detection on each of the training images, the training data selection device 100 may use an open-world object detection model such as Grounding DINO, OWL-ViT (short for Vision Transformer for Open-World Localization), or YOLO (You Only Look Once)-World.

[0064] At this time, the training data selection device 100 may generate individual types corresponding to the training images by further referring to the scene clusters generated based on the above description in addition to the object clusters.

[0065] That is, the training data selection device 100 may perform a process of performing a first embedding operation on each training image to generate a first scene vector corresponding to each training image and clustering the first scene vector to generate a first scene cluster, or a process of performing a kth embedding operation on each training image to generate a kth scene vector corresponding to each training image and clustering the kth scene vector to generate a kth scene cluster. The training data selection device 100 may then perform object detection on each training image to detect at least one object from each training image, generate cropped images by cropping regions corresponding to the bounding boxes of each detected object in each training image, perform an embedding operation on each cropped image to generate an object vector corresponding to each cropped image, and cluster the object vectors to generate object clusters. The training data selection device 100 may then generate individual types corresponding to the training images by referring to the first through kth scene clusters and the object clusters.

[0066] On the other hand, in the above, object detection is performed on each of the training images, and then a cropped image is generated. However, if each of the training images is labeled with its own ground truth, a cropped image can also be generated by referring to the ground truth of each of the training images.

[0067] That is, the training data selection device 100 may refer to the respective ground truth information included in each of the training images, generate cropped images by cropping areas corresponding to the bounding boxes of each object from each of the training images, perform an embedding operation on each of the cropped images to generate object vectors corresponding to each of the cropped images, cluster the object vectors to generate object clusters, and generate individual types corresponding to the training images by referring to the object clusters.

[0068] At this time, the training data selection device 100 may generate individual types corresponding to the training images by further referring to the scene clusters generated based on the above description in addition to the object clusters.

[0069] That is, the training data selection device 100 may perform a process of performing a first embedding operation on each training image to generate a first scene vector corresponding to each training image and clustering the first scene vector to generate a first scene cluster, or a process of performing a kth embedding operation on each training image to generate a kth scene vector corresponding to each training image and clustering the kth scene vector to generate a kth scene cluster. The training data selection device 100 may also reference the ground truth information included in each training image to generate cropped images by cropping regions corresponding to the bounding boxes of each object from each training image, perform an embedding operation on each cropped image to generate object vectors corresponding to each cropped image, and cluster the object vectors to generate object clusters. The training data selection device 100 may then generate individual types corresponding to the training images by referring to the first through kth scene clusters and the object clusters.

[0070] Next, referring back to FIG. 2, the training data selection device 100 can generate a bipartite graph that matches each of a large number of training data included in all the training data with an individual type (S120).

[0071] 5, for example, the bipartite graph matches the relationship between the training data groups and the individual type groups, and it can be understood that training data 1 has individual type 1 and individual type 6, training data 2 has individual type 1, individual type 5, and individual type 6, training data 3 has individual type 2 and individual type 7, training data 4 has individual type 1, training data 5 has individual type 3, individual type 6, and individual type 8, training data 6 has individual type 4, and training data 7 has individual type 2, individual type 5, individual type 7, and individual type 8. Meanwhile, FIG. 5 exemplarily illustrates some training data and some individual types in the bipartite graph.

[0072] 2 again, the training data selection device 100 may select n pieces of training data for training a deep learning model such that the number of each individual type matching the n pieces of training data is within a predetermined threshold deviation when selecting n pieces of training data matching individual types from all training data by referring to the bipartite graph (S130). The n is a target number of training data for training a deep learning model and may be an integer representing a multiple number.

[0073] In this case, if the individual types include 1_1 individual types through 1_x individual types corresponding to the first type possessed by each of the multiple training data and 2_1 individual types through 2_y individual types corresponding to the second type possessed by each of the multiple training data, the training data selection device 100 can ensure that the number of 1_1 individual types through 1_x individual types corresponding to the first type and the number of 2_1 individual types through 2_y individual types corresponding to the second type, which match the n training data, are within a threshold deviation, and can also select the n training data so that the number of 1_1 individual types through 1_x individual types is within a first threshold deviation and the number of 2_1 individual types through 2_y individual types is within a second threshold deviation. In other words, the training data selection device 100 can select training data so that the number of training data for each individual type is as equal as possible for all individual types by classification type.

[0074] Meanwhile, the training data selection device 100 can use at least some of a heuristic algorithm, an optimization algorithm, etc. to select training data for each individual type so that the number of training data is as equal as possible.

[0075] First, the process in which the training data selection device 100 selects training data using a heuristic algorithm will be described below.

[0076] The training data selection device 100 can refer to the bipartite graph to check the number of corresponding individual types that match each of all the training data among the individual types, and select specific training data that has the largest number of corresponding individual types.

[0077] Then, the training data selection device 100 may check the number of remaining corresponding individual types that match with each of all the training data among the remaining individual types excluding the corresponding individual types, and select another specific training data having the largest number of remaining corresponding individual types.

[0078] In this manner, the process of selecting the training data with the largest number of remaining individual types from the remaining individual types excluding the selected individual types is repeated until all the individual types are selected, so that the training data selection device 100 can perform a cycle of selecting a portion of the training data that matches all the individual types.

[0079] Thereafter, the training data selection device 100 may repeat the process of performing the cycle from the remaining training data, excluding a portion of the training data that matches all of the individual types from all the training data, until n pieces of training data are selected.

[0080] For example, referring to Figures 6a to 6d, in Figure 6a, since training data 7 matches the four most common individual types, the training data selection device 100 can select training data 7 from training data 1 to training data 7.

[0081] Then, in the bipartite graph of Figure 6a, if we remove Individual Types 2, 5, 7, and 8 that match training data 7, we are left with Individual Types 1, 3, 4, and 6, as shown in Figure 6b.

[0082] 6b, since training data 1, training data 2, and training data 5 match the two most numerous individual types, the training data selection device 100 can select training data 1, which is one of training data 1, training data 2, and training data 5. In this case, if there are multiple training data with the most numerous corresponding individual types, such as training data 1, training data 2, and training data 5, the training data selection device 100 can select one of the multiple training data, training data 1, training data 2, and training data 5, based on a first criterion to select specific training data, training data 1. The first criterion can be set in various ways, such as selecting one of the multiple training data using a random selection algorithm or selecting one of the multiple training data based on the sequential ranking of the multiple training data.

[0083] Then, in the bipartite graph of Figure 6b, if we remove individual types 1 and 6 that match training data 1, individual types 3 and 4 remain, as shown in Figure 6c.

[0084] 6c, since training data 5 and training data 6 match the most numerous individual type, the training data selection device 100 can select one of training data 5 and training data 6, i.e., training data 5. In this case, if there are a plurality of remaining training data, such as training data 5 and training data 6, each having the most numerous remaining corresponding individual types, the training data selection device 100 can select one of the remaining training data, i.e., training data 5 and training data 6, according to a second criterion to select specific training data, i.e., training data 5. The second criterion can be set in various ways, such as selecting one of the plurality of training data using a random selection algorithm or selecting one of the plurality of training data based on the sequential ranking of the plurality of training data. The second criterion can be set in the same way as the first criterion described above, or differently.

[0085] Then, in the bipartite graph of Figure 6c, if we remove individual type 3 that matches training data 5, individual type 4 remains as shown in Figure 6d.

[0086] 6d, since training data 6 matches the most common individual type, the training data selection device 100 selects training data 6, thereby completing one cycle of selecting a portion of the training data that matches all of the individual types. Thereafter, the training data selection device 100 repeats the above cycle from the remaining training data, excluding training data 1, training data 5, and training data 7, which are portions of the training data selected from all the training data, to select the desired number of training data, n.

[0087] While the above description has been given of selecting some training data that match all of the individual types, it is also possible to select training data by further referring to the type to which each individual type belongs. That is, it is possible to select some training data that match all of the individual types that belong to all type types. For example, when selecting some training data from the type of daytime and nighttime periods and the type of automobile and person objects, it is also possible to select some training data that matches equally when matching the daytime individual types and the nighttime individual types, that matches equally when matching the automobile individual types and the person individual types in the daytime individual types, and that matches equally when matching both the automobile individual types and the person individual types in the nighttime individual types.

[0088] Next, the process in which the training data selection device 100 selects training data using an optimization algorithm will be described below.

[0089] The training data selection device 100 may select a specific subset having the smallest number of training data from among subsets each consisting of a predetermined number of training data including all individual types through an optimization algorithm with reference to the bipartite graph, calculate remaining training data excluding a specific number of training data included in the specific subset, and select at least one other specific subset each consisting of a predetermined number of training data including all individual types through an optimization algorithm from the remaining training data, thereby repeating the process so that the number of selected training data is n or more. In this case, ensuring that the number of selected training data is a desired number, n or more, does not mean that the number of selected training data is unconditionally n or more, but may mean that the total number of training data finally selected is not less than n, depending on the number of specific subsets finally selected. In other words, when the number of training data selected so far is less than n, and the total number of current training data, which is the sum of the number of specific subsets currently selected, is equal to or greater than n, the process of selecting training data ends, and the current total number of training data can be generated as a training dataset for training a deep learning model.

[0090] As an example, the training data selection device 100 may use linear programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training data in a bipartite graph and a Q-dimensional vector representing the selected fitness variables of the Q training data for each of the P individual types, thereby generating a P-dimensional vector in which the sum of the fitness variables is 1 or greater, and the selected fitness variables in the Q-dimensional vector are between 0 and 1, inclusive. The P-dimensional vector may then be used to select a specific subset including specific training data corresponding to a specific selected fitness variable with a minimum sum of the selected fitness variables, and calculate the remaining training data after excluding the specific training data included in the specific subset from the Q training data. The P-dimensional vector may represent the sum of the fitness variables of the Q training data belonging to each of the P individual types. Then, the training data selection device 100 can repeat the process of selecting at least one other specific subset from the remaining training data using linear programming, until the number of selected training data becomes n or more.

[0091] In this case, the linear programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, 0≦x≦1

[0092] In the above, A is a PxQ binary matrix, and A[j,i] represents whether training data i matches individual type j. x is a Q-dimensional vector, and x[i] represents the selected fitness variable for training data i. Ax is the matrix product of A and x, and Ax[j] can represent the sum of the fitness scores of the training data belonging to individual type j.

[0093] Furthermore, the training data selection device 100 can select a specific subset using a dual linear programming method that applies at least one of merging, separating, and changing the sign of constraints in the linear programming method.

[0094] The dual linear programming, which is modified from the linear programming, is expressed as follows, but the present invention is not limited thereto, and the dual linear programming can be applied by various methods of modifying the constraints in the linear programming.

[0095] Purpose: min 1 T x Condition: Bx ≧ b, x ≧ 0

[0096] Then B=[A;-I], (P+Q)xQ binary matrix, B[j,i]=A[j,i], B[P+i,i]= -1 for all i ∈ [1,2,···,Q], B[P+i,j]= 0 for all j≠i b=[1 P ;-1 P ], (P+Q)-dimensional vector, b[j]= 1 for all j ∈ [1,2,···,P], b[P+i]= -1 for all i ∈ [1,2,···,Q] It can be expressed as follows.

[0097] As another example, the training data selection device 100 may use integer programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training data in a bipartite graph and a Q-dimensional vector representing each selected variable of the Q training data for each of the P individual types. The resulting P-dimensional vector has a selection quantity of 1 or more, and among the selected variables in the Q-dimensional vector whose selection variables satisfy 0 or 1, the device may select a specific subset including specific training data corresponding to a specific selected variable having a minimum sum of the selected variables. The P-dimensional vector may represent the selected quantity of training data belonging to each of the P individual types. The device 100 may then repeat the process of selecting at least one other specific subset using integer programming on the remaining training data until the number of selected training data is n or more.

[0098] In this case, the integer programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, x[i] ∈ {0,1} for all i

[0099] In the above, A is a PxQ binary matrix, and A[j,i] represents whether training data i matches individual type j. x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of training data i. Ax is the matrix product of A and x, and Ax[j] can represent the selection quantity of training data belonging to individual type j.

[0100] The training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of merging, separating, and changing the sign of constraints in the integer programming method.

[0101] According to the first embodiment of the present invention, it is possible to select training data for each type as evenly as possible for all individual types. Furthermore, if the ground truth is not labeled for all training data stored in the data pool, it is only necessary to label the ground truth for training data that has been selected as evenly as possible for each type, thereby reducing the cost of generating a training dataset.

[0102] [Second Example] Fig. 7 is a diagram schematically illustrating a method for selecting training data for training a deep learning model according to a second embodiment of the present invention, and the second embodiment of the present invention will be described below with reference to Fig. 7. In the following description, detailed description of parts that can be easily understood from the description of the first embodiment will be omitted.

[0103] First, the training data selection device 100 can acquire at least one individual type corresponding to each of a large number of training images stored in a data pool (S210).

[0104] In this case, the individual types may include 1_1 individual types through 1_x individual types corresponding to the first type possessed by each of the plurality of training images, and 2_1 individual types through 2_y individual types corresponding to the second type possessed by each of the plurality of training images. Each of x and y may be an integer greater than or equal to 1. However, the present invention is not limited thereto, and three or more individual types may be provided according to the characteristics of the training images.

[0105] For example, the information may be configured with various individual types corresponding to various type types, such as type types related to time periods corresponding to individual types such as day, night, evening, and dawn, type types related to weather corresponding to individual types such as sunny, cloudy, rainy, and foggy, and type types related to locations corresponding to individual types such as city, countryside, and highway.

[0106] In addition, at least one individual type (attribute) corresponding to each of the multiple training images can be extracted automatically by labeling each of the multiple training images to a predefined type by a labeler, or by utilizing various techniques such as AI (Artificial Intelligence) and clustering.

[0107] As an example, the training data selection device 100 may transmit a number of training images to a labeler terminal, and use a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the number of training images.

[0108] As another example, the training data selecting device 100 can automatically generate various individual categories corresponding to various category types for the entire image that each of the multiple training images has.

[0109] 3, the training data selection device 100 may perform a process of performing a first embedding operation 10_1 on each training image to generate a first scene vector corresponding to each training image and clustering 20 the first scene vector to generate a first scene cluster, or a process of performing a k-th embedding operation 10_k on each training image to generate a k-th scene vector corresponding to each training image and clustering 20 the k-th scene vector to generate a k-th scene cluster. The k may be an integer greater than or equal to 1. The training data selection device 100 may generate individual types corresponding to the training images by referring to the first to k-th scene clusters.

[0110] In this case, the training data selection device 100 may use a visual foundation model such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), or DINOv2 for the first through k-th embedding operations, and the first through k-th embedding models for the first through k-th embedding operations may be different embedding models that generate scene vectors corresponding to the first through k-th types, which are different types. The training data selection device 100 may use various clustering algorithms, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift, to cluster the first through k-th scene vectors.

[0111] In addition, although the above description describes performing an embedding operation on each of the training images and clustering to generate at least one individual type for each of the training images, individual types can also be generated using meta information included in each of the training images. For example, when k is 1, that is, when only an individual type for one type corresponding to each of the training images is generated by one embedding operation, it is also possible to generate additional individual types for type types different from the one type generated by the embedding operation by using meta information of each of the training images.

[0112] That is, it is possible to check the meta information included in each of the training images and further refer to the shooting time included in each of the meta information to generate an individual type corresponding to each of the training images.

[0113] For example, to generate individual types according to time period types such as day, night, evening, and dawn, the training data selection device 100 may refer to the shooting time of each training image, group the training images according to a predetermined time period, and set an individual type associated with the time period for each group of grouped training images.

[0114] In addition, when generating individual types corresponding to each of the training images by further referring to the shooting time of each of the training images, the training data selection device 100 can also use only a specific embedding model, which is any one of the first embedding model to the kth embedding model, to generate an individual type corresponding to one of the types using the specific embedding model and an individual type corresponding to another type using meta information.

[0115] That is, the training data selection device 100 can perform a specific embedding operation on each training image to generate a specific scene vector corresponding to each training image, and then cluster the specific scene vector to generate a specific scene cluster.The training data selection device 100 can then check the shooting time of each training image by referring to the respective meta information included in each training image.The training data selection device 100 can then generate an individual type corresponding to each training image by referring to the specific scene cluster and the shooting time.

[0116] Next, referring back to FIG. 7, the training data selecting device 100 can generate a bipartite graph that matches each of the multiple training images with an individual type (S220).

[0117] Next, the training data selection device 100 may execute a cycle of selecting some training images that match all individual types from among a large number of training images by referring to the bipartite graph (S230).

[0118] In other words, the training data selection device 100 can refer to the bipartite graph to check the number of corresponding individual types that match each of the multiple training images among the individual types, and select a specific training image that has the largest number of corresponding individual types.

[0119] Then, the training data selection device 100 may check the number of remaining corresponding individual types that match with each of the multiple training images among the remaining individual types excluding the corresponding individual types, and select another specific training image having the largest number of remaining corresponding individual types.

[0120] In this manner, the process of selecting the training image with the largest number of remaining individual types from the remaining individual types excluding the selected individual types is repeated until all individual types are selected, so that the training data selection device 100 can perform a cycle of selecting some training images that match all individual types.

[0121] In this case, when there are a plurality of training images with the largest number of corresponding individual types, the training data selection device 100 may select one of the plurality of training images according to a first criterion to select a specific training image, and when there are a plurality of remaining training images with the largest number of remaining corresponding individual types, the training data selection device 100 may select one of the plurality of remaining training images according to a second criterion to select another specific training image.

[0122] Thereafter, the training data selection device 100 may repeat the process of performing the cycle from the remaining training images, excluding a portion of the training images selected that match all of the individual types, until n training images are selected (S240).

[0123] In this case, if the individual types include 1_1 individual types through 1_x individual types corresponding to the first type possessed by each of the multiple training images and 2_1 individual types through 2_y individual types corresponding to the second type possessed by each of the multiple training images, the training data selection device 100 can ensure that the number of 1_1 individual types through 1_x individual types corresponding to the first type and the number of 2_1 individual types through 2_y individual types corresponding to the second type, which match the n training images, are within a threshold deviation, and can also select the n training images such that the number of 1_1 individual types through 1_x individual types is within a first threshold deviation and the number of 2_1 individual types through 2_y individual types is within a second threshold deviation. In other words, the training data selection device 100 can select training images for all individual types by pattern type so that the number of training images for each individual type is as equal as possible. For example, when selecting some training images from the daytime and nighttime time zone type and the car and person object type, it is possible to select some training images that match evenly when matching with the daytime individual type and the nighttime individual type, that match evenly when matching with the car individual type and the person individual type in the daytime individual type, and that match evenly when matching with both the car individual type and the person individual type in the nighttime individual type.

[0124] [Third Example] Fig. 8 is a diagram schematically illustrating a method for selecting training data for training a deep learning model according to a third embodiment of the present invention, and the third embodiment of the present invention will be described below with reference to Fig. 8. In the following description, detailed description of parts that can be easily understood from the description of the first embodiment will be omitted.

[0125] First, the training data selection device 100 can acquire at least one individual type corresponding to each of a large number of training images stored in a data pool (S310).

[0126] In this case, the individual types may include 1_1 individual types through 1_x individual types corresponding to the first type possessed by each of the plurality of training images, and 2_1 individual types through 2_y individual types corresponding to the second type possessed by each of the plurality of training images. Each of x and y may be an integer greater than or equal to 1. However, the present invention is not limited thereto, and three or more individual types may be provided according to the characteristics of the training images.

[0127] For example, the information may be configured with various individual types corresponding to various type types, such as type types related to time periods corresponding to individual types such as day, night, evening, and dawn, type types related to weather corresponding to individual types such as sunny, cloudy, rainy, and foggy, and type types related to locations corresponding to individual types such as city, countryside, and highway.

[0128] In addition, at least one individual type (attribute) corresponding to each of the multiple training images can be extracted automatically by labeling each of the multiple training images to a predefined type by a labeler, or by utilizing various techniques such as AI (Artificial Intelligence) and clustering.

[0129] As an example, the training data selection device 100 may transmit a number of training images to a labeler terminal, and use a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each of the number of training images.

[0130] As another example, the training data selecting device 100 can automatically generate various individual categories corresponding to various category types for the entire image that each of the multiple training images has.

[0131] 3, the training data selection device 100 may perform a process of performing a first embedding operation 10_1 on each training image to generate a first scene vector corresponding to each training image and clustering 20 the first scene vector to generate a first scene cluster, or a process of performing a k-th embedding operation 10_k on each training image to generate a k-th scene vector corresponding to each training image and clustering 20 the k-th scene vector to generate a k-th scene cluster. The k may be an integer greater than or equal to 1. The training data selection device 100 may generate individual types corresponding to the training images by referring to the first to k-th scene clusters.

[0132] In this case, the training data selection device 100 may use a visual foundation model such as CLIP (Contrastive Language-Image pre-training), DINO (self-DIstillation with NO labels), or DINOv2 for the first through k-th embedding operations, and the first through k-th embedding models for the first through k-th embedding operations may be different embedding models that generate scene vectors corresponding to the first through k-th types, which are different types. The training data selection device 100 may use various clustering algorithms, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-Means, K-Medians, and Mean-Shift, to cluster the first through k-th scene vectors.

[0133] In addition, although the above description describes performing an embedding operation on each of the training images and clustering to generate at least one individual type for each of the training images, individual types can also be generated using meta information included in each of the training images. For example, when k is 1, that is, when only an individual type for one type corresponding to each of the training images is generated by one embedding operation, it is also possible to generate additional individual types for type types different from the one type generated by the embedding operation by using meta information of each of the training images.

[0134] That is, it is possible to check the meta information included in each of the training images and further refer to the shooting time included in each of the meta information to generate an individual type corresponding to each of the training images.

[0135] For example, to generate individual types according to time period types such as day, night, evening, and dawn, the training data selection device 100 may refer to the shooting time of each training image, group the training images according to a predetermined time period, and set an individual type associated with the time period for each group of grouped training images.

[0136] In addition, when generating individual types corresponding to each of the training images by further referring to the shooting time of each of the training images, the training data selection device 100 can also use only a specific embedding model, which is any one of the first embedding model to the kth embedding model, to generate an individual type corresponding to one of the types using the specific embedding model and an individual type corresponding to another type using meta information.

[0137] That is, the training data selection device 100 can perform a specific embedding operation on each training image to generate a specific scene vector corresponding to each training image, and then cluster the specific scene vector to generate a specific scene cluster.The training data selection device 100 can then check the shooting time of each training image by referring to the respective meta information included in each training image.The training data selection device 100 can then generate an individual type corresponding to each training image by referring to the specific scene cluster and the shooting time.

[0138] Next, referring back to FIG. 8, the training data selecting device 100 can generate a bipartite graph that matches each of the multiple training images with the individual type (S320).

[0139] Next, the training data selection device 100 may refer to the bipartite graph and select a specific subset having the smallest number of training images from among subsets consisting of a predetermined number of training images including all individual types through an optimization algorithm, and calculate the remaining training images excluding a specific number of training images included in the specific subset (S330).

[0140] As an example, the training data selection device 100 may use linear programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in a bipartite graph and a Q-dimensional vector representing the selection fitness variables of the Q training images in each of the P individual types, thereby generating a P-dimensional vector in which the sum of the fitness variables is greater than or equal to 1, and among the selection fitness variables of the Q-dimensional vectors whose selection fitness variables are between 0 and 1, select a specific subset including a specific training image corresponding to a specific selection fitness variable having a minimum sum of the selection fitness variables, and calculate the remaining training images excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector may represent the sum of the fitness variables of the Q training images belonging to each of the P individual types.

[0141] In this case, the linear programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, 0≦x≦1

[0142] In the above, A is a PxQ binary matrix, and A[j,i] represents whether training image i matches individual type j. x is a Q-dimensional vector, and x[i] can represent the selection fitness variable of training image i, Ax is the matrix product of A and x, and Ax[j] can represent the sum of the fitness scores of training images belonging to individual type j.

[0143] Furthermore, the training data selection device 100 can select a specific subset using a dual linear programming method that applies at least one of merging, separating, and changing the sign of constraints in the linear programming method.

[0144] The dual linear programming, which is modified from the linear programming, is expressed as follows, but the present invention is not limited thereto, and the dual linear programming can be applied by various methods of modifying the constraints in the linear programming.

[0145] Purpose: min 1 T x Condition: Bx ≧ b, x ≧ 0

[0146] Then B=[A;-I], (P+Q)xQ binary matrix, B[j,i]=A[j,i], B[P+i,i]= -1 for all i ∈ [1,2,···,Q], B[P+i,j]= 0 for all j≠i b=[1 P ;-1 P ], (P+Q)-dimensional vector, b[j]= 1 for all j ∈ [1,2,···,P], b[P+i]= -1 for all i ∈ [1,2,···,Q] It can be expressed as follows.

[0147] As another example, the training data selection device 100 may use integer programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in a bipartite graph and a Q-dimensional vector representing each selection variable of the Q training images in each of the P individual types, thereby generating a P-dimensional vector in which the selection quantity is 1 or more, and from among the selection variables in the Q-dimensional vector whose selection variables satisfy 0 or 1, select a specific subset including specific training images corresponding to a specific selection variable having a minimum sum of the selection variables, and calculate the remaining training images excluding the specific training images included in the specific subset from the Q training images. The P-dimensional vector may represent the selection quantity of training images belonging to each of the P individual types.

[0148] In this case, the integer programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, x[i] ∈ {0,1} for all i

[0149] In the above, A is a PxQ binary matrix, and A[j,i] represents whether training image i matches individual type j. x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of training image i. Ax is the matrix product of A and x, and Ax[j] can represent the number of selected training images belonging to individual type j.

[0150] The training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of merging, separating, and changing the sign of constraints in the integer programming method.

[0151] Thereafter, the training data selection device 100 may repeat the process of selecting at least one other specific subset consisting of a predetermined number of training images including all individual types through an optimization algorithm for the remaining training images until n training images for training the deep learning model are selected (S340).

[0152] In this case, if the individual types include 1_1 individual types through 1_x individual types corresponding to the first type possessed by each of the multiple training images and 2_1 individual types through 2_y individual types corresponding to the second type possessed by each of the multiple training images, the training data selection device 100 can ensure that the number of 1_1 individual types through 1_x individual types corresponding to the first type and the number of 2_1 individual types through 2_y individual types corresponding to the second type, which match the n training images, are within a threshold deviation, and can also select the n training images such that the number of 1_1 individual types through 1_x individual types is within a first threshold deviation and the number of 2_1 individual types through 2_y individual types is within a second threshold deviation. In other words, the training data selection device 100 can select training images for all individual types by pattern type so that the number of training images for each individual type is as equal as possible. For example, when selecting some training images from the daytime and nighttime time zone type and the car and person object type, it is possible to select some training images that match evenly when matching with the daytime individual type and the nighttime individual type, that match evenly when matching with the car individual type and the person individual type in the daytime individual type, and that match evenly when matching with both the car individual type and the person individual type in the nighttime individual type.

[0153] [Fourth Example] Fig. 9 is a diagram schematically illustrating a method for selecting training data for training a deep learning model according to a fourth embodiment of the present invention, and the fourth embodiment of the present invention will be described below with reference to Fig. 9. In the following description, detailed description of parts that can be easily understood from the description of the first embodiment will be omitted.

[0154] First, the training data selection device 100 can acquire at least one individual attribute corresponding to at least one object included in each of a large number of training images stored in a data pool (S410).

[0155] In this case, at least one individual type (attribute) corresponding to each object can be extracted by labeling each of the multiple training images to a predefined type by a labeler, or by automatically extracting the individual type present in each of the multiple training images using various techniques such as AI (Artificial Intelligence) and clustering.

[0156] As an example, the training data selection device 100 may transmit a number of training images to a labeler terminal, and use a labeler corresponding to the labeler terminal to generate at least one individual type corresponding to each object included in each of the number of training images.

[0157] As another example, the training data selection device 100 can automatically generate at least one individual type corresponding to each object included in each of the multiple training images.

[0158] 4, the training data selection device 100 may perform object detection 1 on each of the training images to detect at least one object from each of the training images, generate cropped images by cropping 2 regions corresponding to bounding boxes of each of the detected objects in each of the training images, perform embedding operation 10 on each of the cropped images to generate object vectors corresponding to each of the cropped images, cluster 20 the object vectors to generate object clusters, and generate individual types corresponding to the training images by referring to the object clusters. In this case, to perform object detection on each of the training images, the training data selection device 100 may use an open-world object detection model such as Grounding DINO, OWL-ViT (short for Vision Transformer for Open-World Localization), or YOLO (You Only Look Once)-World.

[0159] In this case, the training data selection device 100 may check the meta information included in each of the training images in addition to the object cluster, and may further refer to the shooting time included in each piece of meta information to generate an individual type corresponding to each of the training images.

[0160] On the other hand, in the above, object detection is performed on each of the training images, and then a cropped image is generated. However, if each of the training images is labeled with its own ground truth, a cropped image can also be generated by referring to the ground truth of each of the training images.

[0161] That is, the training data selection device 100 may refer to the respective ground truth information included in each of the training images, generate cropped images by cropping areas corresponding to the bounding boxes of each object from each of the training images, perform an embedding operation on each of the cropped images to generate object vectors corresponding to each of the cropped images, cluster the object vectors to generate object clusters, and generate individual types corresponding to the training images by referring to the object clusters.

[0162] Next, referring back to FIG. 9, the training data selecting device 100 can generate a bipartite graph that matches each of the multiple training images with the individual type (S420).

[0163] Next, the training data selection device 100 may refer to the bipartite graph and select a specific subset having the smallest number of training images from among subsets consisting of a predetermined number of training images including all individual types through an optimization algorithm, and calculate the remaining training images excluding a specific number of training images included in the specific subset (S430).

[0164] As an example, the training data selection device 100 may use linear programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in a bipartite graph and a Q-dimensional vector representing the selection fitness variables of the Q training images in each of the P individual types, thereby generating a P-dimensional vector in which the sum of the fitness variables is greater than or equal to 1, and among the selection fitness variables of the Q-dimensional vectors whose selection fitness variables are between 0 and 1, select a specific subset including a specific training image corresponding to a specific selection fitness variable having a minimum sum of the selection fitness variables, and calculate the remaining training images excluding the specific training image included in the specific subset from the Q training images. The P-dimensional vector may represent the sum of the fitness variables of the Q training images belonging to each of the P individual types.

[0165] In this case, the linear programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, 0≦x≦1

[0166] In the above, A is a PxQ binary matrix, and A[j,i] represents whether training image i matches individual type j. x is a Q-dimensional vector, and x[i] can represent the selection fitness variable of training image i, Ax is the matrix product of A and x, and Ax[j] can represent the sum of the fitness scores of training images belonging to individual type j.

[0167] Furthermore, the training data selection device 100 can select a specific subset using a dual linear programming method that applies at least one of merging, separating, and changing the sign of constraints in the linear programming method.

[0168] The dual linear programming, which is modified from the linear programming, is expressed as follows, but the present invention is not limited thereto, and the dual linear programming can be applied by various methods of modifying the constraints in the linear programming.

[0169] Purpose: min 1 T x Condition: Bx ≧ b, x ≧ 0

[0170] Then B=[A;-I], (P+Q)xQ binary matrix, B[j,i]=A[j,i], B[P+i,i]= -1 for all i ∈ [1,2,···,Q], B[P+i,j]= 0 for all j≠i b=[1 P ;-1 P ], (P+Q)-dimensional vector, b[j]= 1 for all j ∈ [1,2,···,P], b[P+i]= -1 for all i ∈ [1,2,···,Q] It can be expressed as follows.

[0171] As another example, the training data selection device 100 may use integer programming as an optimization algorithm to calculate the matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in a bipartite graph and a Q-dimensional vector representing each selection variable of the Q training images in each of the P individual types, thereby generating a P-dimensional vector whose selection quantity is 1 or more, and from among the selection variables of the Q-dimensional vector whose selection variables satisfy 0 or 1, select a specific subset including specific training images corresponding to a specific selection variable whose sum of the selection variables has a minimum value, and calculate the remaining training images excluding the specific training images included in the specific subset from the Q training images. The P-dimensional vector may represent the selection quantity of training images belonging to each of the P individual types.

[0172] In this case, the integer programming can be expressed as follows: Purpose: min 1 T x Condition: Ax ≧ 1, x[i] ∈ {0,1} for all i

[0173] In the above, A is a PxQ binary matrix, and A[j,i] represents whether training image i matches individual type j. x is a Q-dimensional vector, and x[i] can represent the selection variable (1 or 0) of training image i. Ax is the matrix product of A and x, and Ax[j] can represent the number of selected training images belonging to individual type j.

[0174] The training data selection device 100 can also select a specific subset using a dual linear programming method that applies at least one of merging, separating, and changing the sign of constraints in the integer programming method.

[0175] Thereafter, the training data selection device 100 may repeat the process of selecting at least one other specific subset consisting of a predetermined number of training images including all individual types through an optimization algorithm for the remaining training images until n training images for training the deep learning model are selected (S440).

[0176] The above-described embodiments of the present invention may be embodied in the form of program instructions that can be executed by various computer components and stored on a computer-readable storage medium. The computer-readable storage medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions stored on the computer-readable storage medium may be specially designed and constructed for the present invention, or may be well-known and available to those skilled in the art of computer software. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processes according to the present invention, or vice versa.

[0177] Although the present invention has been described above using specific details such as concrete components and limited examples and drawings, these are merely provided to facilitate a more comprehensive understanding of the present invention, and the present invention is not limited to the above examples. Those skilled in the art will be able to make various modifications and variations from such descriptions.

[0178] Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and all modifications equivalent to or equivalent to the scope of the claims, as well as the scope of the claims, are within the spirit of the present invention. [Explanation of symbols]

[0179] 100: Learning data selection device 110: Memory 120: Processor

Claims

1. In a method for selecting training data for training a deep learning model, (a) a training data selection device generates at least one individual attribute corresponding to at least one object included in each of a large number of training images stored in a data pool, and generates a bipartite graph matching each of the large number of training images with the individual attribute; and (b) the training data selection device (i) refers to the bipartite graph and selects a specific subset having the smallest number of training images from among subsets consisting of a predetermined number of training images that include all of the individual types through an optimization algorithm, and calculates remaining training images excluding the specific number of training images included in the specific subset; and (ii) repeats the process of selecting at least one other specific subset consisting of a predetermined number of training images that include all of the individual types from the remaining training images through the optimization algorithm until n training images (n is a target number of training images for training the deep learning model and is an integer representing a plural number) for training a deep learning model are selected; A method comprising:

2. In the step (b), the training data selection device uses linear programming to calculate a matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in the bipartite graph and a Q-dimensional vector representing a selection fitness variable for each of the Q training images in each of the P individual types, to generate a P-dimensional vector (the P-dimensional vector represents a sum of fitnesses of the Q training images belonging to each of the P individual types), the sum of fitnesses being 1 or more, and the selection fitness variable in the Q-dimensional vector being between 0 and 1, inclusive; and selects the specific subset including a specific training image corresponding to a specific selection fitness variable having a minimum sum of the selection fitness variables from among the selection fitness variables of the Q-dimensional vector; and calculates the remaining training images by excluding the specific training images included in the specific subset from the Q training images; 2. The method of claim 1 , further comprising repeating the process of selecting the at least one other particular subset using the linear programming method for the remaining training images until the number of selected training images is equal to or greater than n.

3. In the step (b), The method of claim 2, wherein the training data selection device selects the specific subset using a dual linear programming method that applies at least one of merging, separating, and sign-changing constraints in linear programming.

4. In the step (b), The training data selection device uses integer programming to calculate a matrix product of a PxQ binary matrix corresponding to P individual types and Q training images in the bipartite graph and a Q-dimensional vector representing selection variables for each of the Q training images in each of the P individual types to generate a P-dimensional vector (the P-dimensional vector represents the selection quantity of training images belonging to each of the P individual types), where the selection quantity in the P-dimensional vector is 1 or more and the selection variable in the Q-dimensional vector satisfies 0 or 1, and selects the specific subset including specific training images corresponding to a specific selection variable having a minimum sum of the selection variables, and calculates remaining training images by excluding the specific training images included in the specific subset from the Q training images; 2. The method of claim 1 , further comprising repeating the process of selecting the at least one other particular subset using the integer programming method for the remaining training images until the number of selected training images is equal to or greater than n.

5. In the step (b), The learning data selection device is configured to select a learning data The method of claim 4 , wherein the particular subset is selected using a dual integer programming method with at least one of merging, separating, and changing sign of the constraints in g).

6. In the step (a), 2. The method of claim 1, wherein the training data selection device transmits the training images to a labeler terminal and uses a labeler corresponding to the labeler terminal to generate the at least one individual type corresponding to each of the objects included in each of the training images.

7. In the step (a), 2. The method of claim 1 , wherein the training data selection device performs object detection on each of the training images to detect at least one object from each of the training images, generates cropped images by cropping regions corresponding to bounding boxes of the detected objects in each of the training images, performs an embedding operation on each of the cropped images to generate object vectors corresponding to each of the cropped images, clusters the object vectors to generate object clusters, and generates the individual types corresponding to the training images by referring to the object clusters.

8. In the step (a), 2. The method of claim 1, wherein the training data selection device generates cropped images by cropping regions corresponding to bounding boxes of the respective objects from each of the training images with reference to ground truth information included in each of the training images, performs an embedding operation on each of the cropped images to generate object vectors corresponding to each of the cropped images, clusters the object vectors to generate object clusters, and generates the individual types corresponding to the training images with reference to the object clusters.

9. In the step (a), The method of claim 1 , wherein the training data selection device checks meta information included in each of the training images and further refers to the respective shooting times included in the meta information to generate the individual types corresponding to each of the training images.

10. In the step (b), The specific individual types include a 1_1 specific individual type through a 1_x specific individual type (where x is an integer of 1 or more) corresponding to the object, and a 2_1 specific individual type through a 2_y specific individual type (where y is an integer of 1 or more) corresponding to the photographing time, The learning data selection device is configured to make the number of the 1_1 specific individual type through the 1_x specific individual type corresponding to the object, which matches the n learning images, and the number of the 2_1 specific individual type through the 2_y specific individual type corresponding to the shooting time within a threshold deviation, and The method of claim 9, wherein the n training images are selected so that the number of the 1_x specific individual types is within a first threshold deviation, and the number of the 2_1 specific individual types through the 2_y specific individual types is within a second threshold deviation.

11. A learning data selection device that selects learning data for learning a deep learning model, a memory storing instructions for selecting training data for training a deep learning model; and a processor that performs operations in accordance with the instructions stored in the memory to select training data for training the deep learning model; Including, The processor executes the following processes in the training data selection device: (I) generating at least one individual attribute corresponding to at least one object included in each of a number of training images stored in a data pool, and generating a bipartite graph that matches each of the plurality of training images with the individual attribute; and (II) (i) referring to the bipartite graph, selecting a specific subset with the fewest number of training images from subsets consisting of a predetermined number of training images that include all of the individual attributes through an optimization algorithm, and calculating remaining training images excluding a specific number of training images included in the specific subset; and (ii) selecting at least one other specific subset consisting of a predetermined number of training images that include all of the individual attributes through the optimization algorithm from the remaining training images until n training images (n is a target number of training images for training the deep learning model and is an integer representing a plural number) are selected for training the deep learning model.

12. The processor: In the process (II), a PxQ binary matrix corresponding to the P individual types and the Q training images in the bipartite graph is multiplied by a Q-dimensional vector representing the selection fitness variables of the Q training images in each of the P individual types using linear programming to generate a P-dimensional vector (the P-dimensional vector represents the sum of the fitnesses of the Q training images belonging to each of the P individual types), and the sum of the fitnesses is 1 or more, and the selection fitness variables in the Q-dimensional vector are between 0 and 1. From the selection fitness variables of the Q-dimensional vector, the specific subset is selected, which includes a specific training image corresponding to a specific selection fitness variable having a minimum sum of the selection fitness variables, and the remaining training images are calculated by excluding the specific training images included in the specific subset from the Q training images; The training data selection device of claim 11 , wherein the process of selecting the at least one other specific subset using the linear programming method is repeated for the remaining training images so that the number of selected training images becomes equal to or greater than n.

13. The processor:

13. The training data selection device according to claim 12, wherein in the process (II), the specific subset is selected using a dual linear programming method that applies at least one of merging, separating, and sign-changing constraints in linear programming.

14. The processor: In the process (II), integer programming is used to calculate the matrix product of a PxQ binary matrix corresponding to the P individual types and the Q training images in the bipartite graph and a Q-dimensional vector representing each selection variable of the Q training images in each of the P individual types to generate a P-dimensional vector (the P-dimensional vector represents the selection quantity of training images belonging to each of the P individual types), where the selection quantity in the P-dimensional vector is 1 or more and the selection variable in the Q-dimensional vector satisfies 0 or 1; selecting the specific subset including specific training images corresponding to a specific selection variable having a minimum sum of the selection variables; and calculating the remaining training images by excluding the specific training images included in the specific subset from the Q training images; The training data selection device of claim 11 , wherein the process of selecting the at least one other specific subset using the linear programming method is repeated for the remaining training images so that the number of selected training images becomes equal to or greater than n.

15. The processor:

15. The training data selection device according to claim 14, wherein in the process (II), the specific subset is selected using a dual integer programming that applies at least one of merging, separating, and sign-changing constraints in the integer programming.

16. The processor:

12. The training data selection device of claim 11, wherein in the process (I), the training images are transmitted to a labeler terminal, and a labeler corresponding to the labeler terminal is used to generate the at least one individual type corresponding to each of the objects included in each of the training images.

17. The processor:

12. The training data selection device of claim 11, wherein in process (I), object detection is performed on each of the training images to detect at least one object from each of the training images, cropped images are generated by cropping regions corresponding to bounding boxes of each of the detected objects in each of the training images, an embedding operation is performed on each of the cropped images to generate object vectors corresponding to each of the cropped images, the object vectors are clustered to generate object clusters, and the individual types corresponding to the training images are generated by referring to the object clusters.

18. The processor:

12. The training data selection device of claim 11, wherein in the process (I), by referring to the ground truth information included in each of the training images, a cropped image is generated by cropping an area corresponding to a bounding box of each object from each of the training images, an embedding operation is performed on each of the cropped images to generate an object vector corresponding to each of the cropped images, the object vectors are clustered to generate object clusters, and the individual types corresponding to the training images are generated by referring to the object clusters.

19. The processor:

12. The training data selection device of claim 11, wherein in the process (I), the individual types corresponding to the training images are generated by checking meta information included in each of the training images and further referring to the respective shooting times included in the meta information.

20. The processor: In the process (II), a 1_1 specific individual type through a 1_x specific individual type (where x is an integer of 1 or more) corresponding to the object and a 2_1 specific individual type through a 2_y specific individual type (where y is an integer of 1 or more) corresponding to the shooting time are included; 20. The training data selection device of claim 19, wherein the n training images are selected such that a number of the 1_1 specific individual type through the 1_x specific individual type corresponding to the object matching the n training images and a number of the 2_1 specific individual type through the 2_y specific individual type corresponding to the shooting time are within a threshold deviation, a number of the 1_1 specific individual type through the 1_x specific individual type are within a first threshold deviation, and a number of the 2_1 specific individual type through the 2_y specific individual type are within a second threshold deviation.

Images