Training data selection device, training data selection method, and computer program for training data selection

Abstract

The present invention provides a training data selection device capable of appropriately selecting training data to be annotated. [Solution] The training data selection device includes an extraction unit 12 that extracts feature vectors from each of the following: at least one failure data where a predetermined process performed by the target model fails, and a plurality of unlabeled data that have not been annotated; a classification unit 13 that classifies each feature vector into one of a plurality of classes; an identification unit 14 that identifies a feature vector that is included in the failure class containing the feature vector extracted from at least one of the plurality of failure data, and is closest to the centroid of each feature vector of the failure data included in the failure class; and a selection unit 15 that selects the unlabeled data corresponding to the identified feature vector from the plurality of unlabeled data as training data to be annotated.

Description

Technical Field

[0001] The present invention relates to a teacher data selection device, a teacher data selection method, and a computer program for teacher data selection that are used to select teacher data for learning a machine learning model.

Background Art

[0002] Techniques for preparing a set of teacher data used for learning a machine learning model such as a so-called deep neural network (hereinafter sometimes referred to as DNN) have been proposed (see Patent Document 1). The transfer learning method of the object recognition neural network (hereinafter sometimes referred to as ORNN) disclosed in Patent Document 1 determines a plurality of object recognitions within a set of image frames by a first object recognition algorithm that implements ORNN. And this transfer learning method evaluates the object recognition having a plurality of forward results by a second object recognition algorithm different from the first object recognition algorithm, and forms a training set of annotated images including image frames related to the verified object recognition.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to improve the processing accuracy of a target model that executes a predetermined process and is a learning target, it is required to appropriately select teacher data to which annotations are attached.

[0005] Therefore, an object of the present invention is to provide a teacher data selection device that can appropriately select teacher data to which annotations are attached. [Means for solving the problem]

[0006] According to one embodiment, a training data selection device is provided for selecting training data for training a target model that performs a predetermined process. This training data selection device includes: an extraction unit that extracts feature vectors from each of a plurality of candidate data sets, which include at least one failure data set in which the predetermined process by the target model fails, and a plurality of unlabeled data sets that are not annotated; a classification unit that classifies each of the feature vectors of the plurality of candidate data sets into one of a plurality of classes; an identification unit that identifies a feature vector that is included in the failure class which includes a feature vector extracted from at least one of the plurality of failure data sets, and is closest to the centroid of each feature vector of the failure data set included in the failure class; and a selection unit that selects the unlabeled data set corresponding to the identified feature vector from the plurality of unlabeled data sets as training data to be annotated.

[0007] In one embodiment, the training data selection device further includes a generation unit that generates multiple additional data by performing multiple different data transformation processes on each of two or more unlabeled data from a plurality of unlabeled data. The extraction unit extracts feature vectors from each of the generated additional data. Furthermore, the classification unit classifies the feature vectors extracted from each of the plurality of candidate data and the feature vectors extracted from each of the additional data into one of a plurality of classes. The selection unit then further selects a predetermined number of unlabeled data from the two or more unlabeled data, in order of the frequency with which the feature vector extracted from the corresponding additional data is included in the failure class, as training data to be annotated.

[0008] Another embodiment provides a method for selecting training data to train a target model that performs a predetermined process. This training data selection method includes extracting feature vectors from each of a plurality of candidate data sets, which include at least one failure data set in which the predetermined process by the target model fails, and a plurality of unlabeled data sets that are not annotated; classifying the feature vectors of each of the plurality of candidate data sets into one of a plurality of classes; identifying a feature vector that is included in the failure class which includes the feature vector extracted from at least one of the plurality of failure data sets, and is closest to the centroid of each feature vector of the failure data set included in the failure class; and selecting the unlabeled data set corresponding to the identified feature vector from the plurality of unlabeled data sets as training data to be annotated.

[0009] In yet another embodiment, a computer program for selecting training data is provided for training a target model that performs a predetermined process. This training data selection computer program includes instructions for the computer to perform the following: extract feature vectors from each of a plurality of candidate data sets, which include at least one failure data set in which the predetermined process by the target model fails, and a plurality of unlabeled data sets that are not annotated; classify the feature vectors of each of the plurality of candidate data sets into one of a plurality of classes; identify a feature vector that is included in the failure class which includes the feature vector extracted from at least one of the plurality of failure data sets, and is closest to the centroid of each feature vector of the failure data set included in the failure class; and select the unlabeled data set corresponding to the identified feature vector from the plurality of unlabeled data sets as training data to be annotated. [Effects of the Invention]

[0010] The training data selection device described herein has the effect of being able to appropriately select training data to which annotations will be attached. [Brief explanation of the drawing]

[0011] [Figure 1] This is a hardware configuration diagram of the training data selection device. [Figure 2] This is a functional block diagram of the processor in the training data selection device. [Figure 3] This diagram illustrates the selection process for unlabeled data that will be annotated. [Figure 4] This is a flowchart illustrating the operation of the training data selection process. [Modes for carrying out the invention]

[0012] The following describes the training data selection device, the training data selection method executed on the training data selection device, and the computer program for training data selection, with reference to the diagram. This training data selection device selects data to be annotated from among multiple unlabeled data that have not been annotated, so that it can be used as training data for training a target model that performs a predetermined process. To this end, this training data selection device extracts feature vectors from each of the multiple candidate data, which includes at least one failure data where the predetermined process by the target model fails, and multiple unlabeled data, and classifies each extracted feature vector into one of several classes. Furthermore, this training data selection device identifies the feature vector that is closest to the centroid of each feature vector of the failure data included in the failure class, which is included in the failure class that includes the feature vector extracted from the failure data among the multiple unlabeled data. Then, this training data selection device selects the unlabeled data corresponding to the identified feature vector as training data to be annotated.

[0013] The target model can be a DNN trained using supervised learning. However, the target model is not limited to DNNs; it can be a model based on any of the various machine learning methods to which supervised learning is applied. Furthermore, the specified process can be, for example, image recognition, semantic segmentation, or speech recognition. However, the specified process is not limited to these; it is acceptable as long as the target model can perform the specified process by training it with training data.

[0014] Figure 1 is a hardware configuration diagram of the training data selection device. As shown in Figure 1, the training data selection device 1 includes a communication interface 2, an input device 3, a display device 4, a memory 5, a storage device 6, and a processor 7.

[0015] Communication interface 2 has a communication interface and control circuit for connecting to a communication network in accordance with a predetermined communication standard. Communication interface 2 receives various information or data from other devices (not shown) connected via the communication network and passes it to processor 7. The data received by communication interface 2 includes information representing the configuration of a target model to be learned using training data, and a set of data that can become training data used to learn the target model (hereinafter referred to as candidate data).

[0016] Furthermore, the communication interface 2 may transmit to other devices information representing the configuration of the trained target model, which it has received from the processor 7. Similarly, the communication interface 2 may transmit to other devices a set of candidate data that will be used as training data.

[0017] The input device 3 has, for example, a keyboard and a pointing device such as a mouse. Then, the input device 3 generates an operation signal according to an operation by the user, for example, an operation of attaching an annotation to any data included in the set of candidate data or an operation of instructing the start of execution of the teacher data selection process, and outputs the operation signal to the processor 7.

[0018] The display device 4 has, for example, a liquid crystal display or an organic EL display. Then, the display device 4 displays information representing the processing result of the target model for the data for display received from the processor 7, for example, the data included in the set of candidate data or the set of teacher data.

[0019] Note that the input device 3 and the display device 4 may be an integrated device such as a touch panel display.

[0020] The memory 5 is an example of a storage unit, and is, for example, a readable / writable semiconductor memory and a read-only semiconductor memory. Then, the memory 5 stores, for example, various data used in the teacher data selection process and various data generated during the execution of the teacher data selection process.

[0021] The storage device 6 is, for example, a device that accesses a solid-state drive, a hard disk drive, or an optical storage medium and optical recording medium. The storage device 6 is another example of a storage unit. The storage device 6 stores information representing the configuration of the target model and a set of candidate data. Furthermore, the storage device 6 stores information representing the annotations for each data in an annotated sub-dataset included in the set of candidate data. For example, if the data is an image and the predetermined process is object detection, the information representing the annotations is information indicating the type, location, and range of the object to be detected represented on the data. Also, if the predetermined process is semantic segmentation, the information representing the annotations is information indicating the type of object represented in each pixel. In the following, data that is not annotated and included in the set of candidate data will be referred to as unlabeled data. Furthermore, the storage device 6 stores information representing whether the predetermined process performed by the target model on each data in the annotated sub-dataset was successful or unsuccessful. In the following, data on which the predetermined process performed by the target model failed will be referred to as failed data. For example, suppose each data point in the candidate dataset is an image, and the predetermined process is object detection. In this case, if the object detection result obtained by inputting the data into the target model differs from the type, position, and range of the object represented in that data as indicated in the annotation information, then that data is considered failure data. Alternatively, suppose each data point in the candidate dataset is an image, and the predetermined process is semantic segmentation. In this case, if the number of pixels in the segmentation result obtained by inputting the data into the target model and the annotation information differ in the type of object represented by the pixels exceeds a predetermined number, then that data is considered failure data. Furthermore, the storage device 6 stores a computer program for the training data selection process executed by the processor 7.

[0022] Processor 7 has one or more CPUs and its peripheral circuits. Further, processor 7 may have an arithmetic circuit for numerical operations and an arithmetic circuit for graphic processing. And processor 7 executes teacher data selection processing.

[0023] FIG. 2 is a functional block diagram of processor 7. As shown in FIG. 2, processor 7 has a generation unit 11, an extraction unit 12, a classification unit 13, an identification unit 14, a selection unit 15, and a learning unit 16. Each of these units of processor 7 is a functional module realized by, for example, a computer program executed on processor 7. Alternatively, each of these units may be implemented as a dedicated arithmetic circuit provided in processor 7.

[0024] The generation unit 11 generates a plurality of additional data by executing a plurality of different data conversion processes for each of two or more unlabeled data among the plurality of unlabeled data included in the candidate data set. As will be described in detail later, the additional data is used to obtain information that serves as a reference in the selection of teacher data used for learning the target model.

[0025] When the unlabeled data is an image, the data conversion process can be, for example, edge enhancement processing, smoothing processing, contrast conversion processing, local masking processing, resolution conversion processing, rotation processing, translation processing, inversion processing, enlargement or reduction processing, or a combination of two or more of these processes. When the unlabeled data is an audio signal, the data conversion process can be, for example, noise removal processing, noise addition processing, band-pass filter processing, or a combination of two or more of these processes.

[0026] The generation unit 11 generates two or more, preferably three or more, additional data from one unlabeled data. Note that additional data may be generated for all the unlabeled data included in the candidate data set, or additional data may be generated for some of the unlabeled data included in the candidate data set.

[0027] The generation unit 11 stores each generated additional data in the storage device 6, associating it with the unlabeled data from which the additional data originated.

[0028] The extraction unit 12 extracts feature vectors representing the characteristics of each individual unlabeled data, each additional data, and each failure data included in the candidate dataset. For example, the extraction unit 12 extracts feature vectors by inputting the data from which feature vectors are to be extracted into a feature extractor that has been pre-trained to extract feature vectors. As the feature extractor, a DNN pre-trained by so-called unsupervised learning, such as an Auto-Encoder or Stacked What-Where Auto-Encoders, is used. In this case, the feature extractor has, in order from the input side, an encoder that outputs a feature map with reduced dimensionality than the input data, and a decoder into which the features output from the encoder are input. The feature extractor is pre-trained so that the data input to the encoder and the data output from the decoder are the same. Then, by inputting data into the trained feature extractor, the feature map output by the encoder included in that feature extractor becomes the feature vector. Furthermore, a DNN trained by a self-supervised learning method such as Self-Supervised Learning, or a feature extractor based on other feature extraction methods, may be used as the feature extractor. For example, if the data is an audio signal, the extraction unit 12 may divide the audio signal into frames of a specific length and obtain a feature vector in which the signal values ​​for each of several specific frequencies are the individual elements, obtained by performing an FFT on each frame.

[0029] The extraction unit 12 stores each feature vector in the storage device 6, associating it with the data from which the feature vector was extracted.

[0030] The classification unit 13 classifies the feature vectors extracted from each failure data, each unlabeled data, and each additional data into one of several classes. To do this, the classification unit 13 applies a predetermined clustering method, such as the k-nearest neighbors method, to the set of feature vectors to be classified, thereby classifying each feature vector into one of the classes.

[0031] The classification unit 13 notifies the identification unit 14 of information indicating the individual feature vectors included in each class.

[0032] The identification unit 14 identifies the class containing the feature vector extracted from the failure data from among the multiple classes into which each feature vector has been classified by the classification unit 13. The class containing the feature vector extracted from the failure data will be referred to as the failure class below. The identification unit 14 also calculates the centroid of the feature vector extracted from each individual failure data within each failure class.

[0033] The identification unit 14 uses both false positive (FP) and false negative (FN) failure data to identify the failure class. If the target model is a model that detects a predetermined object, FN failure data is data in which the target model failed to detect the predetermined object, and FP failure data is data in which the target model misdetected a predetermined object that is not represented in the data. The identification unit 14 may also use only one of the FP or FN failure data to identify the failure class. For example, if the ratio of FP failure data among the failure data identified by inputting each training data included in the training dataset to the target model is higher than the ratio of FN failure data by a predetermined threshold, the identification unit 14 may use only the FP failure data to identify the failure class. Conversely, if the ratio of FN failure data is higher than the ratio of FP failure data by a predetermined threshold, the identification unit 14 may use only the FN failure data to identify the failure class. In this way, by using either FP (Fixed Failure) or FN (Failure Negative) failure data to identify the failure class, it becomes possible to prepare a training dataset for training the target model to focus on reducing one of the FP or FN failures.

[0034] If there are multiple classes that contain failure data, the identification unit 14 may designate all of those classes as failure classes, or it may designate only the class with the largest number of failure data as a failure class. Alternatively, the identification unit 14 may designate any class with a number of failure data equal to or greater than a predetermined number as a failure class. The predetermined number can be an integer of 2 or more.

[0035] The identification unit 14 further identifies, for each failure class, the feature vector that is closest to the centroid of each failure data feature vector included in that failure class, from among the feature vectors included in that failure class and extracted from any of the multiple unlabeled data. The identification unit 14 then notifies the selection unit 15 of the failure class, the identified feature vector, and the unlabeled data corresponding to that feature vector.

[0036] The selection unit 15 selects, from among multiple unlabeled data, the unlabeled data corresponding to the feature vector identified by the identification unit 14 as the training data to be annotated. The feature vector identified by the identification unit 14 is classified into the same failure class as the feature vector extracted from any of the failure data, and the distance between these feature vectors is also close. Therefore, the unlabeled data corresponding to the identified feature vector is considered to have features similar to the failure data. Thus, by selecting the training data in this manner, the unlabeled data that is highly likely to cause the target model to fail to process is selected as the training data to be annotated. As a result, the selection unit 15 can appropriately select the training data.

[0037] Furthermore, the selection unit 15 determines the frequency with which each unlabeled data point, extracted from the individual additional data points generated from that unlabeled data point, is included in any of the failure classes. If the number of additional data points generated from each unlabeled data point is the same for each unlabeled data point, the selection unit 15 only needs to count the number of additional data points generated from that unlabeled data point that contain a feature vector extracted into the failure class as the frequency. Alternatively, if the number of generated additional data points differs for each unlabeled data point, the selection unit 15 may calculate the frequency as the ratio of the number of additional data points generated from that unlabeled data point that contain a feature vector in the failure class to the total number of additional data points generated from that unlabeled data point.

[0038] The selection unit 15 further selects a predetermined number of unlabeled data from the individual unlabeled data generated by the additional data, in order of the frequency with which the feature vectors extracted from the additional data are included in the failure class, as training data to be annotated. Unlabeled data with a high frequency with which the feature vectors of the additional data are included in the failure class are considered to have features similar to the failure data. Therefore, by selecting a predetermined number of unlabeled data as training data in order of the frequency with which the feature vectors of the additional data are included in the failure class, unlabeled data that is highly likely to fail in processing for the target model is selected as training data to be annotated. Thus, the selection unit 15 can appropriately select the training data.

[0039] Figure 3 illustrates the overview of the selection process for unlabeled data to be annotated. In Figure 3, white circles represent feature vectors extracted from the unlabeled data itself, black circles represent feature vectors extracted from failure data, and asterisks represent feature vectors extracted from additional data.

[0040] In the example shown in Figure 3, each feature vector extracted from individual unlabeled data, individual failure data, and individual additional data is classified into one of several classes 301 to 305. Of these classes 301 to 305, class 301 contains feature vector 311 extracted from the failure data. Similarly, class 303 contains feature vector 312 extracted from the failure data. Therefore, classes 301 and 303 are failure classes. In failure class 301, feature vector 321 is closest to feature vector 311 extracted from the failure data. Therefore, the unlabeled data 331 corresponding to feature vector 321, i.e., the source from which feature vector 321 was extracted, is selected as training data to be annotated. Similarly, in failure class 303, feature vector 322 is closest to feature vector 312 extracted from the failure data. Therefore, the unlabeled data 332 corresponding to feature vector 322 is also selected as training data to be annotated.

[0041] Furthermore, feature vectors 351-352 extracted from additional data 341-342 generated from unlabeled data 333 are included in failure class 301 or failure class 303. Similarly, of the feature vectors 353-354 extracted from additional data 343-344 generated from unlabeled data 334, feature vector 353 is included in failure class 303. Moreover, none of the feature vectors 355-356 extracted from additional data 345-346 generated from unlabeled data 335 are included in the failure class. Therefore, if the predetermined number of unlabeled data to select based on the frequency with which the feature vectors extracted from the additional data are included in the failure class is 1, the unlabeled data 333 with the highest frequency is further selected as training data to be annotated.

[0042] As described above, each selected unlabeled data point is annotated, and these data points are added to the training dataset as training data. The annotation of each selected unlabeled data point is done manually by the user. Alternatively, the annotation may be done automatically based on the processing results obtained by inputting the selected unlabeled data points into another model that has higher processing accuracy than the target model and is capable of performing the same processing as the target model.

[0043] The learning unit 16 updates the target model by training it using a training dataset that includes newly added training data selected and annotated by the selection unit 15. In this process, the learning unit 16 can train the target model according to a predetermined learning method such as backpropagation.

[0044] The learning unit 16 determines whether a training data set is a failure or not by comparing the processing result of the updated target model for that training data set with the annotation attached to that training data set.

[0045] The processor 7 repeatedly performs the training data selection process using the newly failed training data and the remaining unlabeled data. The processor 7 then terminates the training data selection process when predetermined termination conditions are met. These predetermined termination conditions may include the target model's processing accuracy reaching a pre-set target accuracy, the degree of change in the target model's processing accuracy before and after the execution of the training data selection process at a certain point in time falling within a predetermined range, or the training data selection process reaching a predetermined number of iterations.

[0046] Figure 4 is an operation flowchart of the training data selection process according to this embodiment.

[0047] The generation unit 11 generates additional data based on two or more unlabeled data (step S101). The extraction unit 12 extracts feature vectors from each of the unlabeled data, additional data, and failure data (step S102). The classification unit 13 classifies each feature vector into one of several classes (step S103).

[0048] The identification unit 14 identifies the failure class that contains the feature vector extracted from the failure data among multiple classes (step S104). Furthermore, for each failure class, the identification unit 14 identifies the feature vector that is closest to the centroid of the feature vector extracted from the failure data (step S105).

[0049] The selection unit 15 selects unlabeled data corresponding to the identified feature vectors as training data to be annotated (step S106). The selection unit 15 further selects a predetermined number of unlabeled data from the unlabeled data, in order of frequency with which the feature vector extracted from the corresponding additional data is included in the failure class, as training data to be annotated (step S107).

[0050] The learning unit 16 updates the target model by training it using the training dataset, which includes newly added training data that has been annotated (step S108). Furthermore, based on the updated target model, it identifies the failure data among the training data included in the training dataset (step S109).

[0051] Processor 7 determines whether the termination condition has been met (step S110). If the termination condition has not been met (step S110-No), processor 7 repeats the process from step S101 onwards. Note that in the second and subsequent cycles, since feature vector extraction and feature vector classification have already been performed, the processes from steps S101 to S103 may be omitted. On the other hand, if the termination condition has been met (step S110-No), processor 7 terminates the training data selection process.

[0052] As explained above, this training data selection device identifies the feature vector closest to the centroid of each feature vector extracted from each failure data within each failure class, for each failure class that includes a feature vector extracted from the failure data. The training data selection device then selects the unlabeled data corresponding to the identified feature vector as training data to be annotated. Furthermore, the training data selection device also selects a predetermined number of unlabeled data from the unlabeled data, in order of frequency with which the feature vector extracted from the corresponding additional data is included in the failure class, as training data to be annotated. In this way, the training data selection device can appropriately select training data because it selects unlabeled data with features similar to those of the failure data as training data to be annotated.

[0053] In the modified version, the processing by the generation unit 11 may be omitted. In this case, since no additional data is generated, the extraction of feature vectors from the additional data and the selection of training data based on the feature vectors extracted from the additional data may also be omitted. This reduces the computational load of a single training data selection process. In particular, when unlabeled data representing various situations is collected, it is possible to appropriately select training data to be used for training the target model without generating additional data, so the processing related to additional data may be omitted in this way.

[0054] The training data selection device according to the above embodiment or modification may be implemented, for example, in a vehicle. In this case, sensor signals (e.g., images or audio signals) sequentially generated by sensors mounted on the vehicle (e.g., cameras that photograph the surroundings or interior of the vehicle, or microphones that capture sounds around or inside the vehicle) are collected as unlabeled data. A portion of the collected unlabeled data is then transmitted to a server outside the vehicle via a wireless communication terminal mounted on the vehicle. The server then annotates each individual unlabeled data received. The server may annotate each individual unlabeled data manually by the user or by using another model with higher processing accuracy, as described above. The server then transmits information about the annotations attached to each individual unlabeled data to the training data selection device mounted on the vehicle. Using this annotation information, the training data selection device can identify the failure data as described above and perform the training data selection process using the identified failure data. Furthermore, the training data selection device transmits the individual unlabeled data selected as a result of the training data selection process to the server via the wireless communication terminal. Then, by utilizing the annotation information attached to the unlabeled data received from the server via a wireless communication terminal, the training data selection device can repeatedly execute the training data selection process.

[0055] Furthermore, the computer program that implements the functions of each part of the processor 7 of the training data selection device according to the above embodiment or modification may be provided in the form of a computer-readable portable recording medium. Such a recording medium may be, for example, a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0056] As described above, those skilled in the art can make various modifications within the scope of the present invention to suit the implemented form. [Explanation of Symbols]

[0057] 1. Teacher data selection device, 2. Communication interface, 3. Input device, 4. Display device, 5. Memory, 6. Storage device, 7. Processor, 11. Generation unit, 12. Extraction unit, 13. Classification unit, 14. Identification unit, 15. Selection unit, 16. Learning unit

Claims

1. An extraction unit extracts feature vectors from each of several candidate data sets, which include at least one failure data set in which a predetermined process performed by the target model fails, and several unlabeled data sets that have not been annotated. A classification unit that classifies the feature vectors of each of the multiple candidate data into one of the multiple classes, A specification unit identifies a feature vector that is included in a failure class that includes the feature vector extracted from at least one of the failure data from the plurality of classes, and that is closest to the centroid of the feature vector of each failure data included in the failure class, among the feature vectors extracted from any of the plurality of unlabeled data. A selection unit selects, from among a plurality of the aforementioned unlabeled data, the unlabeled data corresponding to the identified feature vector as the training data to be annotated. A training data selection device having the following features.

2. The system further includes a generation unit that generates multiple additional data by performing multiple different data transformation processes on each of two or more of the multiple unlabeled data, The extraction unit extracts the feature vector from each of the generated additional data, The classification unit classifies the feature vectors extracted from each of the multiple candidate data and the feature vectors extracted from each of the additional data into one of the multiple classes. The training data selection device according to claim 1, wherein the selection unit further selects a predetermined number of unlabeled data from the two or more unlabeled data, in order of the frequency with which the feature vector extracted from the corresponding additional data is included in the failure class, as training data.

3. Feature vectors are extracted from each of several candidate data sets, which include at least one failure data point where a predetermined process performed by the target model fails, and several unlabeled data points that have not been annotated. Each of the feature vectors of the aforementioned multiple candidate data is classified into one of the multiple classes. Among the feature vectors included in the failure class, which includes the feature vector extracted from at least one of the failure data from the plurality of classes, and which are extracted from any of the plurality of unlabeled data, the feature vector closest to the centroid of the feature vector of each failure data included in the failure class is identified. From among the multiple unlabeled data points, the unlabeled data point corresponding to the identified feature vector is selected as the training data to be annotated. A method for selecting training data, including the following.

4. Feature vectors are extracted from each of several candidate data sets, which include at least one failure data point where a predetermined process performed by the target model fails, and several unlabeled data points that have not been annotated. Each of the feature vectors of the aforementioned multiple candidate data is classified into one of the multiple classes. Among the feature vectors included in the failure class, which includes the feature vector extracted from at least one of the failure data from the plurality of classes, and which are extracted from any of the plurality of unlabeled data, the feature vector closest to the centroid of the feature vector of each failure data included in the failure class is identified. From among the multiple unlabeled data points, the unlabeled data point corresponding to the identified feature vector is selected as the training data to be annotated. A computer program for selecting training data to perform the following task on a computer.

Images