Data selection device, data selection method, program, and data selection system

Abstract

The present invention provides a data selection device, a data selection method, a program, and a data selection system that can efficiently select data high in learning effect even in an environment having limited calculation resources. The present invention comprises: an acquisition unit that acquires a feature amount vector corresponding to each piece of new data output by inputting a plurality of pieces of new data to a feature amount extraction model which has been caused to learn, in advance, to output a feature amount vector for input of data and has a fixed parameter related to a feature amount space; and a data selection unit that selects, from among the plurality of pieces of new data, unlabeled data for which the feature amount vector satisfies a predetermined criterion.

Description

Data selection device, data selection method, program, and data selection system

[0001] The present disclosure relates to a data selection device, a data selection method, a program, and a data selection system.

[0002] Various services using machine learning have been proposed. For example, a classifier pre-trained by machine learning can classify various data into multiple classes. For example, in a manufacturing site where products are manufactured, a classifier that can output a classification result indicating whether a product is defective or not in response to input image data is sometimes used for product inspection.

[0003] Generally, the more training data (training data) available, the better the performance of a classifier. However, the task of annotating a large amount of training data with correct labels requires a great deal of time and effort.

[0004] To solve this problem, there is a technique called active learning, in which a minimum amount of data that is estimated to have a high learning effect is extracted, and learning is performed by labeling only the extracted data as a correct answer. For example, Patent Documents 1 to 3 disclose various techniques that employ active learning.

[0005] JP 2017-167834 A JP 2020-187408 A JP 2020-101968 A

[0006] In active learning, an initial classifier is trained using a small amount of labeled data and a large amount of unlabeled data as training data. The initial classifier is then used to automatically select unlabeled data that is estimated to have a high learning effect. An oracle (a person or a separately trained machine) that knows the correct answer assigns a correct label to the selected unlabeled data. The classifier is then retrained by adding new data with a correct label.

[0007] Active learning involves repeating this cycle to improve the performance of the classifier.

[0008] However, active learning uses a classifier to select unlabeled data that is expected to have a high learning effect, so it is subject to constraints imposed by the classifier's requirements (such as the processing speed at the time of implementation).In addition, re-learning is performed every time a new correct label is assigned, which requires a large amount of computational resources.

[0009] The present disclosure aims to provide a data selection device, a data selection method, a program, and a data selection system that can efficiently select data with high learning effectiveness even in an environment with limited computational resources.

[0010] A data selection device according to one aspect of the present disclosure includes: an acquisition unit that inputs a plurality of new data into a feature extraction model that has been trained in advance to output a feature vector in response to input data, and has fixed parameters related to a feature space, and acquires feature vectors corresponding to each of the new data that is output; and a data selection unit that selects, from the plurality of new data, unlabeled data whose feature vector satisfies a predetermined criterion.

[0011] A data selection method according to one aspect of the present disclosure is a data selection method executed by a computer, in which a feature extraction model is trained to output a feature vector in response to input data, and parameters related to a feature space are fixed. A plurality of new data are input and feature vectors corresponding to each of the new data output are obtained from the feature extraction model, and unlabeled data whose feature vectors satisfy predetermined criteria are selected from the plurality of new data.

[0012] A program according to one aspect of the present disclosure causes a computer to execute the following steps: inputting a plurality of new data into a feature extraction model that has been trained to output a feature vector in response to input data, and having parameters related to a feature space fixed; acquiring, from the feature extraction model, feature vectors corresponding to each of the new data that has been output; and selecting, from the plurality of new data, unlabeled data whose feature vectors satisfy predetermined criteria.

[0013] A data selection system according to one aspect of the present disclosure is a data selection method executed by a computer, which trains a feature extraction model to output a feature vector in response to input data, the trained feature extraction model having fixed parameters related to feature space, and the trained feature extraction model is input with a plurality of new data, and feature vectors corresponding to each of the new data output are obtained from the feature extraction model, and unlabeled data whose feature vectors satisfy predetermined criteria is selected from the plurality of new data.

[0014] The present disclosure makes it possible to efficiently select data with high learning effectiveness even in an environment with limited computational resources.

[0015] FIG. 1 is a block diagram illustrating an example of a functional configuration of a diagram data selection device showing an overview of a data selection system including a data selection device according to an embodiment of the present disclosure; FIG. 2 is a flowchart illustrating an example of an operation of a diagram data selection device showing an example of a hardware configuration of the data selection device and illustrating a data selection method by a diagram data selection unit;

[0016] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0017] [Overview] Fig. 1 is a diagram showing an overview of a data selection system 100 including a data selection device 10 according to an embodiment of the present disclosure. In Fig. 1, each device constituting the data selection system 100 is indicated by a square. The devices of the data selection system 100 are connected to each other so that they can communicate with each other. In Fig. 1, data, data sets, or trained models exchanged between the devices are indicated by ellipses.

[0018] In the example shown in FIG. 1, the data selection device 10 is a device that outputs a learning dataset to a product inspection learning device 30 that learns a trained model for product inspection.

[0019] A trained model for product inspection is a trained model for inspecting defects in manufactured products at a product manufacturing site, such as a factory. The trained model for product inspection is trained to output a classification result of whether the product is good or bad (defective) in response to input image data of the product.

[0020] The product inspection learning device 30 is, for example, a computer, and uses the learning dataset acquired from the data selection device 10 to construct and learn a trained model for product inspection. The learning dataset is, for example, a set of image data of multiple products and labels indicating whether the products depicted in each image data are good or defective. The product inspection learning device 30 uses this training dataset to perform supervised learning on the trained model for product inspection.

[0021] Prior to data selection by the data selection device 10, a large amount of product image data is input to the feature extraction device 20. The feature extraction device 20 has a pre-trained feature extraction model. The feature extraction model is a model that has been trained to output a feature vector in response to input image data. The training process for the feature extraction model may be performed by the feature extraction device 20 or by another learning device not shown in FIG. 1 .

[0022] The feature extraction model is a trained model trained using, for example, deep metric learning, which is a deep learning method for learning the distance (similarity) between data.

[0023] Deep metric learning uses a deep neural network to convert input data into a low-dimensional vector representation. This vector representation represents the position of each piece of data in a metric space. Using the converted vector representation, a space (metric space) into which the data is embedded is learned, so that data with similar characteristics are closer and data with different characteristics are farther apart.

[0024] The feature extraction model of the feature extraction device 20 has a feature that parameters related to features are not updated. In other words, once the training process for the feature extraction model is completed, it is not retrained. This allows the data selection system 100 to save computational resources for retraining the feature extraction model.

[0025] The large amount of product image data input to the feature extraction device 20 includes labeled data, to which a label indicating whether the product depicted in the image data is good or defective is assigned, and unlabeled data, to which the label has not yet been assigned. For example, it is expected that most of the image data input to the feature extraction device 20 is unlabeled data, and the amount of labeled data is small.

[0026] The data selection device 10 selects some unlabeled image data that is estimated to have a high learning effect in the product inspection learning device 30, based on the feature vectors acquired from the feature extraction device 20. Then, it accepts new input for labeling the selected image data, and outputs the selected image data and the corresponding labels as a learning dataset.

[0027] The above describes an overview of the data selection system 100 according to the present disclosure using FIG. 1 , but this is merely one aspect of the present disclosure and various modifications are possible. For example, in the example shown in FIG. 1 , the feature extraction device 20 outputs a feature vector based on input image data. However, in a modification of the present disclosure, the data selection device may have, for example, a trained feature extraction model. In this modification, the data selection system may not have a feature extraction device. Also, in this modification, the data selection device may accept input of a large amount of product image data and extract feature vectors using the feature extraction model itself.

[0028] [Data Selection Device 10] The following describes in detail the configuration and operation of the data selection device 10 according to the embodiment of the present disclosure. Fig. 2 is a block diagram illustrating an example of the functional configuration of the data selection device 10.

[0029] As shown in FIG. 2, the data selection device 10 includes an acquisition unit 11 , a storage unit 12 , a data selection unit 13 , an output unit 14 , an input reception unit 15 , and a usefulness calculation unit 16 .

[0030] The acquisition unit 11 acquires feature vectors of multiple pieces of image data from the feature extraction device 20 shown in Fig. 1. The feature vectors acquired by the acquisition unit 11 include feature vectors of labeled data and feature vectors of unlabeled data.

[0031] The storage unit 12 stores the feature vectors acquired from the feature extraction device 20 in association with the ID (identification number) of the image data to which each feature vector corresponds. Here, the ID of the image data may be assigned a flag indicating whether each image data is labeled data or unlabeled data. Furthermore, when a new label is assigned by the input accepting unit 15, the storage unit 12 stores the newly assigned label in association with the corresponding feature vector and the ID of the image data.

[0032] The data selection unit 13 acquires feature vectors of unlabeled data from the storage unit 12. The data selection unit 13 selects unlabeled data whose feature vectors satisfy predetermined criteria. The data selection unit 13 outputs the IDs of the selected unlabeled data to the output unit 14. Details of how the data selection unit 13 selects unlabeled data whose feature vectors satisfy predetermined criteria will be described later.

[0033] Based on the ID of the unlabeled data selected by the data selection unit 13, the output unit 14 outputs the unlabeled data corresponding to the ID to the oracle. The oracle is an entity, such as a person, that knows the correct label of the unlabeled data. More specifically, the oracle is, for example, a user of the data selection system 100. In this case, the output unit 14 is, for example, a display device. The output unit 14 displays the unlabeled data on the display device based on the ID, allowing the user to input a correct label for the unlabeled data via the input acceptance unit 15. The output unit 14 may acquire the unlabeled data corresponding to the ID from, for example, the feature extraction device 20.

[0034] The oracle is not limited to a person such as a user, but may be a machine (computer) including a trained model that has been trained in advance so that it can output a correct label for input of image data. In this case, the output unit 14 transmits the unlabeled data selected by the data selection unit 13 to the machine based on the ID. This allows the machine to input the correct label for the unlabeled data via the input acceptance unit 15.

[0035] The input accepting unit 15 accepts input of a correct label for the unlabeled data selected by the data selecting unit 13. When the oracle is a person, the input accepting unit 15 is an input device such as a keyboard, a mouse, or a touchpad. When the oracle is a machine, the input accepting unit 15 is a communication device for communicating with the machine.

[0036] The usefulness calculation unit 16 calculates a usefulness indicating the degree to which image data corresponding to each feature vector stored in the storage unit 12 is useful for learning in the product inspection learning device 30. Furthermore, when a label is assigned to the unlabeled data selected by the data selection unit 13 via the input acceptance unit 15, the usefulness calculation unit 16 updates the usefulness based on the assigned label. Details of the method of calculating the usefulness by the usefulness calculation unit 16 will be described later.

[0037] With this functional configuration, the data selection unit 13 can efficiently select unlabeled data that is highly effective for learning in the product inspection learning device 30.

[0038] 3 is a diagram showing an example of the hardware configuration of the data selection device 10. The data selection device 10 is a computer having a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.

[0039] The programs or instructions that realize the various functions and processes described below in the data selection device 10 may be stored on a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or flash memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium into the storage device 102 or memory device 103 via the drive device 101. However, the programs or instructions do not necessarily have to be installed from the storage medium, and may be downloaded from an external device via a network or the like.

[0040] The storage device 102 is realized by a hard disk drive or the like, and stores installed programs or instructions as well as files, data, etc. used to execute the programs or instructions.

[0041] The memory device 103 is realized by a random access memory, a static memory, or the like, and when a program or instruction is activated, reads and stores the program, instruction, data, or the like from the storage device 102. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory tangible storage medium.

[0042] The processor 104 may be realized by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuitry, etc., which may be composed of one or more processor cores, and performs various functions and processes of the data selection device 10 described above in accordance with programs, instructions, data such as parameters required to execute the programs or instructions, etc. stored in the memory device 103.

[0043] The user interface (UI) device 105 may be composed of input devices such as a keyboard, mouse, camera, and microphone, output devices such as a display, speaker, headset, and printer, and input / output devices such as a touch panel, and realizes an interface between a user and the data selection device 10. The UI device 105 corresponds to, for example, at least one of the output unit 14 and the input receiving unit 15 shown in FIG. 3. For example, a user operates a GUI (Graphical User Interface) displayed on a display or touch panel using a keyboard, mouse, etc. to operate the data selection device 10.

[0044] The communication device 106 is realized by various communication circuits that execute wired and / or wireless communication processing with external devices, the Internet, a LAN (Local Area Network), a cellular network, or other communication networks.

[0045] The above-described hardware configuration is merely an example, and the data selection device 10 according to the present disclosure may be realized by any other appropriate hardware configuration.

[0046] [Data Selection Method by Data Selector 13] The data selection method by the data selector 13 will be described below with specific examples.

[0047] Fig. 4 is a diagram for explaining the data selection method by the data selection unit 13. Fig. 4 is a diagram showing an example of a feature space expressing feature vectors of multiple image data. In Fig. 4, the positions of labeled data in the feature space are indicated by ○, and the positions of unlabeled data are indicated by ×.

[0048] As described above, the data selection unit 13 selects unlabeled data whose feature vectors satisfy a predetermined criterion. In the example shown in FIG. 4 , the data selection unit 13 calculates the similarity between each of the plurality of unlabeled data to be selected and each of the labeled data to which labels have already been assigned, and determines whether the unlabeled data satisfies the predetermined criterion based on the similarity. The similarity is calculated by comparing the distance and direction between the feature vectors of the respective image data. A known technique may be appropriately adopted as a method for calculating the similarity.

[0049] 4, three labeled data are classified into class A, class B, and class C. The data selection unit 13 calculates the distance between each of the two unlabeled data D1 and D2 and each class in the feature space. In the feature space, data that are close to each other have a high similarity, and data that are far from each other have a low similarity.

[0050] The unlabeled data D1 is data that has the highest similarity to class A among classes A, B, and C. On the other hand, the unlabeled data D2 is data that has low similarity to any of classes A, B, and C.

[0051] The data selection unit 13 selects a predetermined number of pieces of data from the plurality of unlabeled data based on the relationship (similarity) between the existing labeled data and the unlabeled data calculated based on the feature vector. Note that the present disclosure does not limit the degree of similarity of the unlabeled data that the data selection unit 13 selects as data that satisfies the predetermined criteria.

[0052] There are various ways of thinking about what level of similarity the data selection unit 13 should select unlabeled data with, in other words, what data the data selection unit 13 should determine as being highly effective for learning in the product inspection learning device 30.

[0053] For example, unlabeled data that has low similarity to any class is considered to be data that is less distinctive than other data. According to this first concept, unlabeled data that has high similarity to any class can be said to be more effective for learning in the product inspection learning device 30. In the example shown in Figure 4, when the first concept is adopted, the data selector 13 selects unlabeled data D1.

[0054] On the other hand, unlabeled data with low similarity to any class can also be considered to be data for which it is difficult for the product inspection trained model trained by the product inspection learning device 30 to determine the correct answer. According to this second concept, it can be said that unlabeled data with low similarity to any class is more likely to benefit from the oracle assigning a correct label than unlabeled data with high similarity to one of the classes. In the example shown in Figure 4, when the second concept is adopted, the data selector 13 selects unlabeled data D2.

[0055] In the present disclosure, the data selection unit 13 may select data based on what concept and with what degree of similarity, and this may be set arbitrarily by the user of the data selection system 100, for example, at the time of system design.

[0056] As described above, the data selection unit 13 selects unlabeled data whose feature vectors satisfy a predetermined criterion based on the relationship (e.g., similarity) between labeled data and unlabeled data in a feature space based on the feature vectors. This makes it possible to suitably select, from a large amount of unlabeled data, data that is considered to be highly effective in some way for training a trained model for product inspection that is to be trained by the product inspection learning device 30.

[0057] The data selector 13 outputs the similarity used to select the unlabeled data to the memory 12. The memory 12 stores the similarity in association with the selected image data. The similarity stored in the memory 12 is used by the usefulness calculator 16.

[0058] [Method of Calculating Usefulness by Usefulness Calculation Unit] Next, the method of calculating the usefulness by the usefulness calculation unit 16 will be described in detail.

[0059] The usefulness calculation unit 16 assigns an initial value of usefulness to each of all image data corresponding to all feature vectors stored in the storage unit 12. The initial value of usefulness is, for example, the same value for all image data. The usefulness assigned by the usefulness calculation unit 16 is stored in association with each image data in, for example, the storage unit 12.

[0060] When the data selection unit 13 selects image data and the input receiving unit 15 receives input of a correct label for the selected data, the usefulness calculation unit 16 updates the usefulness of the image data based on the relationship between the input correct label and the features and similarity of the selected image data.

[0061] The method for updating the usefulness will be described with a specific example using unlabeled data in the feature space illustrated in Fig. 4. As described above, the unlabeled data D1 shown in Fig. 4 is data that has a high similarity to class A. Assume that this image data D1 is selected by the data selection unit 13, and the input receiving unit 15 receives an input of a correct label for the selected data, resulting in the data being labeled as class B.

[0062] In this case, the image data D1 is actually class B despite having a high similarity to class A, and therefore can be said to be data from which it is difficult to derive a correct answer for the trained model for product inspection trained by the product inspection learning device 30. In other words, the image data D1 can be said to be image data that is highly effective for learning for the trained model for product inspection trained by the product inspection learning device 30.

[0063] On the other hand, for example, suppose that the input receiving unit 15 receives an input of a correct label for unlabeled data D1 shown in Fig. 4 and the data is labeled as class A. In this case, the data is labeled according to the similarity calculated when the data selecting unit 13 selected the data. Therefore, the usefulness calculating unit 16 does not change the usefulness of such image data D1.

[0064] In this way, when the input receiving unit 15 receives an input of a correct label for data selected by the data selecting unit 13, for example, if the class classification result at the time of data selection by the data selecting unit 13 differs from the correct label, the usefulness calculation unit 16 improves the usefulness of the image data. By repeatedly updating the usefulness in this way, an appropriate usefulness is set for each of the image data corresponding to all of the feature vectors stored in the storage unit 12.

[0065] As described above, the data selection unit 13 selects data using usefulness, and therefore, as the usefulness calculation unit 16 repeatedly updates the usefulness, it can be said that the performance of data selection in the data selection device 10 gradually improves.

[0066] [Example of Operation of Data Selection Device 10] The following describes an example of operation of the data selection device 10. FIG.

[0067] In step S1 , the acquisition unit 11 acquires a feature vector from the feature extraction device 20 .

[0068] In step S2, the storage unit 12 stores the feature vector. Here, the storage unit 12 stores the feature vector and the ID of the image data corresponding to the feature vector in association with each other.

[0069] In step S3, the usefulness calculation unit 16 assigns an initial value of usefulness to each image data. As described above, the initial value of usefulness assigned in this step is the same for all image data.

[0070] In step S4, the data selection unit 13 selects data based on the usefulness. The method of data selection by the data selection unit 13 is as described above.

[0071] In step S5, the storage unit 12 stores the similarity calculated for each image data by the data selection unit 13 when selecting data in step S4 in association with the feature vector of each image data.

[0072] In step S6, the output unit 14 outputs the image data selected in step S4.

[0073] In step S7, the input receiving unit 15 receives an input of a correct label for the image data selected in step S4.

[0074] In step S8, the storage unit 12 stores the correct label input in step S7 in association with the corresponding image data.

[0075] In step S9, the usefulness calculation unit 16 updates the usefulness based on the similarity of the image data selected in step S4 and the correct label input received in step S7. The method for updating the usefulness is as described above.

[0076] In step S10, the data selection device 10 determines whether or not there is an addition of a feature vector corresponding to new image data from the feature extraction device 20. If there is no addition (step S10: YES), the data selection device 10 ends the process. If there is an addition (step S10: NO), the data selection device 10 returns the process to step S5 and repeats the processes from step S5 to step S9.

[0077] <Actions and Effects> As described above, the data selection device 10 according to the embodiment of the present disclosure is trained in advance to output a feature vector in response to input data, and includes the acquisition unit 11 that inputs a plurality of new data into a feature extraction model in which parameters related to a feature space are fixed, and acquires feature vectors corresponding to each of the new data that is output, and the data selection unit 13 that selects, from the plurality of new data, unlabeled data whose feature vector satisfies a predetermined criterion.

[0078] As described above, according to the data selection device 10 according to the embodiment of the present disclosure, when data selected by the data selector 13 is labeled by an oracle, the feature extraction model is not retrained using the labeled data. Furthermore, the feature extraction model extracts a feature vector only once.

[0079] Therefore, the data selection device 10 according to the embodiment of the present disclosure does not need to perform operations requiring high computational resources, such as re-learning a feature extraction model and extracting features using the feature extraction model. This allows for optimal data selection from large amounts of data, even in environments where high computational resources are unavailable. Furthermore, because parameter tuning is not required for re-learning the feature extraction model, even those who are not experts in machine learning (deep learning) can select data using the data selection device 10.

[0080] Furthermore, according to the data selection device 10 according to the embodiment of the present disclosure, the data selection device 10 itself does not need to have a feature extraction model. Therefore, the data selection device 10 can operate without a computer having the high processing speed required to properly operate the feature extraction model as a classifier. This allows the manufacturing cost of the data selection device 10 to be reduced.

[0081] Moreover, the data selection device 10 according to the embodiment of the present disclosure further includes a storage unit 12 that stores feature vectors. This means that the data selection device 10 itself does not need to have a feature extraction model.

[0082] Furthermore, in the data selection device 10 according to the embodiment of the present disclosure, the data selector 13 selects new data from among a plurality of new data based on the relationship between labeled data and unlabeled data in the feature space, thereby enabling highly effective data selection.

[0083] The data selection device 10 according to the embodiment of the present disclosure further includes a usefulness calculation unit 16 that calculates a usefulness indicating the degree to which data is useful as training data to be used for training a predetermined machine learning model. The data selection unit 13 then performs selection based on the usefulness.

[0084] This allows the data selection device 10 to efficiently select useful data as training data used in learning a predetermined machine learning model.

[0085] Furthermore, the data selection device 10 according to the embodiment of the present disclosure further includes an input receiving unit 15 that receives input of a correct label for the unlabeled data selected by the data selecting unit 13, and the usefulness calculation unit 16 updates the usefulness based on the relationship between the feature vector corresponding to the unlabeled data for which the input of the correct label has been received and the correct label. As a result, after the usefulness is updated, data selection is performed based on the usefulness updated by the input correct label, thereby improving the accuracy of data selection.

[0086] The data selection device 10 according to the embodiment of the present disclosure is particularly useful in the following usage modes, for example.

[0087] For example, at a product manufacturing site such as a factory, defects in manufactured products may be inspected using a pre-trained trained model for product inspection. The trained model for product inspection is trained to output a classification result of whether the product is good or bad (defective) in response to input image data of the product. In order to efficiently train such a trained model for product inspection, it is desirable to select image data that is expected to have a high learning effect from a large amount of image data and assign correct labels only to the selected image data.

[0088] Due to security and other issues, product image data used in product inspection is often prohibited from being taken outside the factory. For this reason, it is desirable that a data selection device for selecting a training dataset be installed at the manufacturing site, such as inside the factory. Due to issues such as cost and installation space, it is undesirable to install a large computer with high computational resources at the manufacturing site. For this reason, it is desirable that the functions of the data selection device be realized by a small, lightweight computer with limited computational resources, such as a laptop computer.

[0089] Active learning is known as a method for selecting image data that is estimated to have a high learning effect, as disclosed in Patent Documents 1 to 3. However, active learning requires re-learning of the feature extraction model, and the functions of a data selection device that uses active learning are difficult to achieve without a computer with high computational resources.

[0090] As described above, the data selection device 10 according to the embodiment of the present disclosure does not require re-learning of a feature extraction model or feature extraction processing using the feature extraction model. Therefore, it is expected that the functions of the data selection device 10 according to the embodiment of the present disclosure can be sufficiently realized by a small, lightweight computer with limited computing resources, such as a laptop computer.

[0091] Although examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of the present disclosure as set forth in the claims.

[0092] 100 Data selection system 10 Data selection device 11 Acquisition unit 12 Storage unit 13 Data selection unit 14 Output unit 15 Input reception unit 16 Usefulness calculation unit 20 Feature extraction device 30 Learning device for product inspection

Claims

1. A data selection device comprising: an acquisition unit that inputs a plurality of new data into a feature extraction model that has been trained in advance to output a feature vector in response to input data and has fixed parameters related to a feature space, and acquires feature vectors corresponding to each of the new data output; and a data selection unit that selects, from the plurality of new data, unlabeled data whose feature vector satisfies a predetermined criterion.

2. The data selection device according to claim 1, further comprising a storage unit for storing the feature vector.

3. The data selection device according to claim 1, wherein the data selection unit performs the selection based on a relationship in the feature space between labeled data and unlabeled data from among the plurality of new data.

4. The data selection device according to claim 3, further comprising a usefulness calculation unit that calculates a usefulness indicating the degree to which the data is useful as training data used in training a predetermined machine learning model.

5. The data selection device according to claim 4, wherein the data selection unit makes the selection based on the usefulness.

6. The data selection device of claim 4, further comprising an input receiving unit that receives an input of a correct label for the unlabeled data selected by the data selection unit, wherein the usefulness calculation unit updates the usefulness based on a relationship between the feature vector corresponding to the unlabeled data for which the input of the correct label has been received and the correct label.

7. The data selection device according to any one of claims 1 to 6, wherein the feature extraction model is a trained model trained by deep metric learning using a plurality of labeled data.

8. A data selection method executed by a computer, comprising: training a feature extraction model to output a feature vector in response to data input; the trained feature extraction model has fixed parameters related to feature space; a plurality of new data are input to the trained feature extraction model, and feature vectors corresponding to each of the new data output are obtained from the feature extraction model; and unlabeled data whose feature vector satisfies a predetermined criterion is selected from the plurality of new data.

9. A program that causes a computer to execute the steps of: inputting a plurality of new data into a feature extraction model that has been trained to output a feature vector in response to data input, and in which parameters related to feature space are fixed, and obtaining from the feature extraction model feature vectors corresponding to each of the new data output; and selecting, from the plurality of new data, unlabeled data whose feature vector satisfies a predetermined criterion.

10. A data selection system comprising: a feature extraction device having a pre-trained feature extraction model in which parameters related to a feature space are fixed, and inputting a plurality of data into the feature extraction model and outputting a feature vector; and a data selection device selecting, from the plurality of data, unlabeled data whose feature vector satisfies a predetermined criterion.

Images