Machine learning method and information processing device

Abstract

To provide an accurate classifier while improving generalization performance.SOLUTION: An information processing device disclosed herein is configured to: estimate a degree of improvement of an evaluation metric related to a machine learning model, where the improvement is to be obtained on the condition that the machine learning model is configured to be trained based on a combination of training data in a training data set; select a pair of training data from the training data set based on the estimated degree; generate other training data based on the selected pair of training data; and train the machine learning model based on the generated other training data and the training data set.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a machine learning technique for training a machine learning model using training data. [Background technology]

[0002] In recent years, classifiers (machine learning models) trained using training datasets containing multiple training data by applying machine learning algorithms are being used for anomaly detection, medical image diagnosis, etc. In training this classifier, it is known that data expansion is performed by Mixup, which generates new training data by combining and mixing the training data contained in the training dataset, and the generalization performance of the classifier is further improved by training the classifier using the expanded training dataset. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-181204 A [Patent Document 2] Patent Publication No. 2022-80213 [Patent Document 3] JP 2022-124989 A [Patent Document 4] US Patent Application Publication No. 2021 / 0124993 [Patent Document 5] International Publication No. 2021 / 100818 Summary of the Invention [Problem to be solved by the invention]

[0004] When training a classifier using a training dataset, it is difficult to properly evaluate the performance of a machine learning algorithm based on the accuracy of the classifier alone, so a more complex evaluation index may be used. A more complex evaluation index is an evaluation index with constraints to prevent the prediction of the classifier from being biased toward a specific class. However, when data expansion of a training dataset is performed by Mixup as in the above-mentioned conventional technology, although it is useful for further improving the generalization performance of the classifier, there is a problem that it is difficult to optimize a more complex evaluation index.

[0005] In one aspect, the present invention aims to provide a machine learning method and an information processing device that can obtain a classifier with high accuracy while improving generalization performance. [Means for solving the problem]

[0006] In a first proposal, the machine learning method is characterized in that a computer executes the following processes: estimating a degree of improvement of an evaluation index for a machine learning model when the machine learning model is trained based on a combination of training data included in a training dataset, selecting a set of training data from the training dataset based on the estimated degree, generating other training data based on the selected set of training data, and training the machine learning model based on the other generated training data and the training dataset. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a functional block diagram showing a configuration of an information processing device according to this embodiment. [Diagram 2] FIG. 2 is a flowchart illustrating a processing procedure of the information processing device according to the present embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, examples of the information processing device, the machine learning method, and the machine learning program disclosed in the present application will be described in detail with reference to the drawings. Note that the disclosed technology is not limited to these examples. In addition, the following examples may be appropriately combined within a range that does not contradict each other. EXAMPLES

[0009] Reference 1: Narasimhan, H., Menon, AK: Training over-parameterized models with non-decomposable objectives, NeurIPS 2021

[0010] First, we explain the basic evaluation index for the prediction accuracy of a classifier used in Reference 1 and other documents, as well as evaluation indexes that are more complex than the basic evaluation index (e.g., evaluation indexes with constraints).

[0011] We define a classifier as F:X→[K], where X denotes the space of inputs, and [K]={1, ,K} is the set of labels.

[0012] A confusion matrix C[F] with K rows and K columns is defined as shown in formula (1). In formula (1), D is the distribution of the data. In formula (1), "1" is the indicator function. In the indicator function, if "y=i,F(x)=j" holds, the value of the indicator function is "1", and if not, the value of the indicator function is "0". Furthermore, E in formula (1) corresponds to the calculation of the expected value.

[0013]

number

[0014] For each i, we define the class prior as:

[0015]

number

[0016] First, an example of a basic evaluation index will be described. The accuracy of a classifier, acc[F], is defined by formula (3). For example, the accuracy corresponds to the ratio of the number of correct data out of all data input to the classifier.

[0017]

number

[0018] Recall of the classifier per class i [F] is defined by formula (4). Recall corresponds to the proportion of data that can actually be classified out of the number of data to be classified. For example, recall indicates how many of the multiple data that should be classified into the first class were classified into the first class by the classifier.

[0019]

number

[0020] The precision of the classifier per class i [F] is defined by formula (5). Precision corresponds to the percentage of correct answers among the number of "data to be classified" that are actually classified. For example, precision is the percentage of data that should actually be classified as the first class among multiple data classified as the first class by the classifier.

[0021]

number

[0022] The percentage of cases where the classifier predicts class i is defined as coverage. i [F] is defined by equation (6).

[0023]

number

[0024] Next, we explain an example of a more complicated evaluation metric. Worst-case recall is defined by equation (7). Worst-case recall is a useful evaluation metric for class-imbalanced datasets.

[0025]

number

[0026] Similarly, when a dataset is class-imbalanced, there is a problem that the predictions of the classifier are biased towards a particular class. Therefore, in a classifier, it is important to optimize the coverage constraint. Equation (8) is an example of an evaluation index for optimizing the average recall under the coverage constraint. For example, in equation (8), coverage is set to "0.95 × π i This is an evaluation index that maximizes the sum of the recall rates of classes 1 to K under the condition that "K" is equal to or greater than K.

[0027]

number

[0028] In addition, for classifiers, an evaluation index that optimizes under a constraint on the precision rate can also be considered. This evaluation index that optimizes under a constraint on the precision rate is shown in formula (9). For example, formula (9) is an evaluation index that maximizes the accuracy acc[F] under the condition that the precision rate is equal to or greater than "τ (threshold)".

[0029]

number

[0030] Reference 2: Rangwani et al, Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics, NeurIPS 2022

[0031] Reference 3: Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.

[0032] Reference 4: Linjun Zhang et al, How Does Mixup Help With Robustness and Generalization?, ICLR 2021

[0033] CSST (Reference 2) allows for the training of a classifier that optimizes complex evaluation metrics using unlabeled datasets, but this comes at the expense of reduced classifier accuracy.

[0034] In Mixup (Reference 3), the training data for a classifier are (x1, y1),…,(x N ,y N ) data extension.

[0035] For example, in Mixup, we consider an arbitrary number λ∈[0,1] and two samples (x n ,y n ),(x m ,y m ) we create new samples as shown in equation (10).

[0036]

number

[0037] In Mixup, this sample creation is repeated L times, and the new samples shown in equation (11) are used as training data for the classifier.

[0038]

number

[0039] It is known that the above Mixup improves the generalization performance of the classifier (Reference 4). However, the above Mixup cannot optimize complex evaluation indices related to the classifier.

[0040] The information processing device of this embodiment generates new training data by combining training data from a training dataset including multiple training data, and trains parameters of a classifier (machine learning model) based on the generated new training data and the multiple training data.

[0041] Here, the information processing device according to the present embodiment estimates the degree of improvement of an evaluation index (e.g., a complex evaluation index) for the classifier when the classifier is trained based on the generated training data. Then, the information processing device according to the present embodiment selects a set of training data from the training data set based on the estimated degree, and generates new training data based on the selected set of training data. Therefore, the information processing device according to the present embodiment can obtain a classifier with high accuracy in which the evaluation index (e.g., a complex evaluation index) for the classifier is improved while improving the generalization performance of the classifier.

[0042] An example of an information processing device according to the present embodiment will be described below. In the information processing device according to the present embodiment, the last linear layer of a K-class classifier for deep learning is trained.

[0043] Also, for a training data sample x included in the training data set, a feature vector z is given. The classifier F outputs a label k (k=1,...,K).

[0044] Specifically, the matrix W(W k is a d-dimensional column vector, k = 1, ..., K), and for each sample i, a feature vector z i It is given that:

[0045]

number

[0046] In addition, the probability that the classifier F predicts that the feature vector z corresponding to a sample is label k is given by the following equation (13) using a softmax function.

[0047]

number

[0048] For k=1,...,K, the distribution of class labels is expressed by the following equation (14).

[0049]

number

[0050] The confusion matrix C[F] of a classifier F is defined similarly to equation (1) (1≦i,j≦K).

[0051] Also, (π i Softmax j (A i )) 1≦j≦K is the i-th row of C[F], so that K-dimensional vector A i has been decided.

[0052] In addition, the evaluation index M of the optimization target is expressed as (A1, ..., A K ) (differentiable function). Here, the evaluation index M may be a basic evaluation index related to the prediction accuracy of the classifier F, or a complex evaluation index (e.g., an evaluation index with constraints), and may be one or more that are appropriately set by the user.

[0053]

number

[0054] Also, let a and u be a pair of class labels. That is, a and u are 1≦a, u≦K. z a Let z be the feature vector of a sample (labeled training data) whose label is a. u Let be the feature vector of the unlabeled sample (unlabeled training data) whose pseudo-label is u.

[0055] Here, the pseudo label is a label output by the classifier F for an unlabeled sample and assigned to the unlabeled sample.

[0056] The information processing device according to the present embodiment performs a and z u Mixup is expressed by the following formula (16): Here, λ is an appropriate distribution (for example, uniform distribution).

[0057]

number

[0058] The information processing device according to this embodiment trains parameters (matrix W) for a classifier F based on new training data generated by Mixup and training data (labeled training data) included in a training dataset.

[0059] In the information processing device according to the present embodiment, since the pseudo label (u) for new training data (z) generated by Mixup is unreliable, the parameters (matrix W) of the classifier are trained so as to minimize the cross-entropy loss function l(z, a) with the label of z as a. The cross-entropy loss function l(z, a) is given by the following formula (17).

[0060]

number

[0061] The information processing apparatus according to this embodiment trains parameters to minimize the sum of several samples of the cross-entropy loss function l(z, a).

[0062] Next, it will be described how the information processing device according to this embodiment estimates the degree to which the evaluation index M for the classifier F is improved and selects a pair of class labels (a, u) based on the estimated degree.

[0063] In order to determine how to select (a, u), the information processing device of this embodiment estimates the increment in the evaluation index M of the classifier F when the classifier F is trained using training data generated by Mixup as follows.

[0064] As an example, the information processing device according to the present embodiment uses gradient descent in training parameters for the classifier F. When using gradient descent, the matrix W is updated by training the parameters as shown in the following formula (18).

[0065]

number

[0066] The information processing device according to this embodiment calculates the (k, l) component of ∂l(z, a) / ∂W according to the following formula (19). Here, δ ai is the Kronecker delta.

[0067]

number

[0068] With the update of W, A k A k +ΔA k and is considered to have been updated.

[0069] The information processing device according to this embodiment estimates the change in the evaluation index M due to the update of W according to the following formula (20).

[0070]

number

[0071] The information processing device according to the present embodiment calculates the average of the feature quantities of samples (training data) belonging to class k for k=1, . . . , K as Z k Then, ΔA k ΔWZ k is approximately estimated as

[0072] In this way, the information processing device according to the present embodiment estimates the change ΔM(a, u) in the evaluation index M for each (a, u) (1≦a, u≦K). Then, the information processing device according to the present embodiment selects a pair of class labels (a, u) with the probability of the following formula (21).

[0073]

number

[0074] Next, a configuration example of an information processing device according to the present embodiment will be described. Fig. 1 is a functional block diagram showing the configuration of an information processing device according to the present embodiment. As shown in Fig. 1, the information processing device 1 has a communication unit 10, an input unit 20, a display unit 30, a storage unit 40, and a control unit 50.

[0075] The communication unit 10 executes data communication with an external device etc. via a network. The communication unit 10 may receive a training data set 41, a validation set 42, initial value data 43 etc., which will be described later, from the external device.

[0076] The input unit 20 receives operations from a user. The user uses the input unit 20 to specify the evaluation index M and the like.

[0077] The display unit 30 displays the results of processing by the control unit 50 .

[0078] The storage unit 40 has a training data set 41, a validation set 42, initial value data 43, and classifier data 44. For example, the storage unit 40 is realized by a memory or the like.

[0079] The training dataset 41 includes a plurality of training data used for training the classifier F. Specifically, the training dataset 41 includes a labeled training dataset 41a including a plurality of training data to which a label (a) is assigned, and an unlabeled training dataset 41b including unlabeled training data.

[0080] Here, the labeled training data is composed of a set of input data and a correct label (a). The unlabeled training data includes the input data and does not have a correct label. The pseudo-labels assigned to the unlabeled training data are generated by the control unit 50, which will be described later.

[0081] The validation set 42 includes a plurality of validation data. The validation data is composed of a pair of input data and a correct label. The validation set 42 is used when estimating the confusion matrix C.

[0082] The initial value data 43 includes the number of iterations T of training (learning) of the classifier F, the initial values ​​of the weights of the final layer of the classifier F, the learning rate, the minimum value of λ, and the like.

[0083] The classifier data 44 is data of the classifier F to be trained. For example, the classifier F is a Neural Network (NN).

[0084] The control unit 50 includes an estimation unit 51, a selection unit 52, a training data generation unit 53, and a training execution unit 54. For example, the control unit 50 is realized by a processor.

[0085] The estimation unit 51 is a processing unit that estimates the increment of the evaluation index M of the classifier F, i.e., the degree of improvement of the evaluation index M for the classifier F, when the classifier F is trained using training data generated by Mixup.

[0086] The selection unit 52 selects a training data set from the training data set 41 based on the degree estimated by the estimation unit 51. Specifically, the selection unit 52 selects a training data set from each of the labeled training data set 41a and the unlabeled training data set 41b based on the degree estimated by the estimation unit 51, the training data set having class labels (or pseudo labels) a and u, respectively. An appropriate class is selected from the training data set based on the degree, regardless of whether it is labeled or unlabeled. For example, the training data set is selected uniformly at random from both data sets (the labeled training data set 41a and the unlabeled training data set 41b).

[0087] Here, the selection unit 52 generates a pseudo-label for the unlabeled training data selected from the unlabeled training data set 41b by inputting the value of the training data to the classifier F. The selection unit 52 assigns the generated pseudo-label to the unlabeled training data selected from the unlabeled training data set 41b.

[0088] In the information processing device 1 according to the present embodiment, semi-supervised learning is performed by combining labeled training data and unlabeled training data. However, the learning of the classifier F is not limited to semi-supervised learning. For example, the information processing device 1 may perform only supervised learning. In this case, the selection unit 52 selects a set of training data from the labeled training data set 41a based on the degree of estimation by the estimation unit 51.

[0089] The training data generation unit 53 is a processing unit that generates new training data based on the set of training data selected by the selection unit 52. Specifically, the training data generation unit 53 generates new training data by mixing up the set of training data according to the above-mentioned formula (16).

[0090] The training execution unit 54 is a processing unit that trains parameters (matrix W) for the classifier F based on new training data generated by the training data generation unit 53 and labeled training data included in the labeled training dataset 41a. For example, when training data is input to the classifier F, the training execution unit 54 trains the parameters of the classifier F using a gradient descent method or the like so that the labels output by the classifier F become the labels assigned to the training data.

[0091] Here, a detailed description will be given of the processes of the estimation unit 51, the selection unit 52, the training data generation unit 53, and the training execution unit 54 in the information processing device 1 according to this embodiment. Fig. 2 is a flowchart showing the processing procedure of the information processing device according to this embodiment.

[0092] As shown in FIG. 2, when the process starts, the control unit 50 receives an input of data related to the process (S101).

[0093] Specifically, the control unit 50 derives labeled training data D as shown in equation (22) from the labeled training data set 41a. L Here, D L In x i (x i ∈X) is a vector representing a sample such as an image. i x i The class label of (1≦y i ≦K).

[0094]

number

[0095] In addition, the control unit 50 derives unlabeled training data D as shown in equation (23) from the unlabeled training data set 41b. U Accepts input.

[0096]

number

[0097] The control unit 50 also selects, from the validation set 42, labeled validation data D for calculating the confusion matrix C[F] as shown in equation (24). val Accepts input.

[0098]

number

[0099] Furthermore, the control unit 50 accepts an input specifying an evaluation index M via the input unit 20 or the like. As for the evaluation index M, one or more are specified from among the basic evaluation indexes related to the prediction accuracy of the classifier F described above or complex evaluation indexes (e.g., evaluation indexes with constraints).

[0100] In addition, the control unit 50 calculates the initial weighting values ​​W of the final layer of the classifier F as shown in equation (25) from the initial value data 43. (1) Here, W1,…,W K is a d-dimensional column vector.

[0101]

number

[0102] In addition, the control unit 50 determines from the initial value data 43 the number of times the learning is repeated, T, the learning rate η (η>0), and the minimum value λ of λ used in mixup, λ min Accepts input.

[0103] The output (S111) of the result of the processing (S103 to S110) of the control unit 50 in response to the above input is the classifier data 44 of the classifier F after training.

[0104] Specifically, for a sample x, the trained classifier F returns the value of the following equation (26): (T) is the initial value W (1) is the weight at the T-th iteration updated by the iterative equation (18).

[0105]

number

[0106] Here, the values ​​of the following equation (27) are the weights (parameters) updated in the classifier data 44 during training.

[0107]

number

[0108] After S101, the control unit 50 sets t=1 (S102). Next, the estimation unit 51 estimates the verification data D val Based on the confusion matrix C (t) (S103). Note that the confusion matrix C (t) is defined by the following equation (28) for 1≦i,j≦K.

[0109]

number

[0110] Here, F (t) (x) is defined by the following equation (29).

[0111]

number

[0112] Also, |D val | is D val is the number of samples. 1(y=i,F (t) (x)=j) is the value of the indicator function, y=i and F (t) It represents 1 when (x)=j, and 0 otherwise.

[0113] Next, the estimation unit 51 calculates a partial differential of the evaluation index M with respect to the parameter A (S104). Next, the estimation unit 51 estimates an increment of the evaluation index M using the calculated partial differential (S105).

[0114] Here, a description will be given of the calculation of partial differentials performed by the estimation unit 51. A confusion matrix C with K rows and K columns is expressed by the following equation (30) using a matrix A with K rows and K columns.

[0115]

number

[0116] C i , A i are the i-th row of the confusion matrix C and matrix A, respectively, and s ∈ R K is defined as follows:

[0117]

number

[0118] Also, π i represents P(y=i), and D val It can be calculated as the proportion of samples (x, y) for which y=i.

[0119] In addition, the partial derivative ∂M / ∂A of the evaluation index M (a function of the confusion matrix C) with respect to the parameter A kl can be calculated since the functional form of the evaluation index M is known.

[0120] The estimation unit 51 is C (t) Parameter A corresponding to (t) Using this, calculate the following equation (32). Note that equation (32) is the partial differential ∂M / ∂A kl A=A (t) Represents the value at .

[0121]

number

[0122] Next, the estimation of the increment of the evaluation index M performed by the estimation unit 51 will be described. Let 1≦a, u≦K be class labels. For each 1≦k≦K, the average of feature vectors having label k is given by the following formula (33).

[0123]

number

[0124] In addition, in the following equation (34), l(z,i) is the cross-entropy loss function.

[0125]

number

[0126] The cross-entropy loss function of l(z,i) is given by the following equation (35):

[0127]

number

[0128] Also, ΔW (t) (z, i) is defined as follows:

[0129]

number

[0130] Here, ∂l(z,i) / ∂W(W (t) ) is the gradient ∂l(z,i) / ∂W of W=W (t) Also, for 1≦l≦K, v (t) l (au) is the l-th column vector of the following equation (37), where λ is the interval [λ min Let x, be real numbers sampled according to a uniform distribution on [,1].

[0131]

number

[0132] The estimation unit 51 uses the calculated partial differential to calculate an estimate ΔM(a, u) of the increment of the evaluation index M according to the following equation (38).

[0133]

number

[0134] Following S105, the selection unit 52 performs sampling within 1≦a, u≦K based on the estimated increment ΔM(a, u) of the evaluation index M calculated by the estimation unit 51 (S106).

[0135] Specifically, the selection unit 52 performs sampling with the probability of the above-mentioned formula (21). As an example, the selection unit 52 selects a pair of class labels (a, u) with the above-mentioned probability, and then uniformly randomly selects X1 from the samples (labeled training data having class label a) shown in the following formula (39).

[0136]

number

[0137] In addition, the selection unit 52 selects the pair of class labels (a, u) with the above-mentioned probability, and then selects X2 uniformly at random from the samples (unlabeled training data having pseudo label u) shown in the following equation (40).

[0138]

number

[0139] Next, the training data generation unit 53 performs data extension of the training data by generating new training data based on the pair of training data (X1, X2) selected by the selection unit 52 (S107). Specifically, the training data generation unit 53 performs mixup according to the following formula (41).

[0140]

number

[0141] The new training data (z) obtained by mixup is expressed by the following equation (42).

[0142]

number

[0143] Next, the training execution unit 54 performs learning (training) of the classifier F using the mixed training data (S108). Specifically, the training execution unit 54 performs training on the classifier F so as to minimize the above-mentioned cross-entropy loss function l(z, a). (t) The control unit 50 may perform learning in batches in which data augmentation by mixup is performed multiple times. The training execution unit 54 updates the updated W to W (t+1) Let us assume that.

[0144] Next, the control unit 50 increments t to t=t+1 (S109), and determines whether t>T is satisfied (S110).

[0145] If t>T is not satisfied (S110: No), the control unit 50 returns the process to S103. If t>T is satisfied (S110: Yes), the control unit 50 outputs the classifier data 44 of the classifier F described above (S111) and ends the process.

[0146] Next, the effect of the information processing device 1 according to the present embodiment will be described. The information processing device 1 estimates the degree of improvement of an evaluation index related to a machine learning model when the machine learning model (classifier) ​​is trained based on training data generated by combining training data from a training data set 41 including a plurality of training data. The information processing device 1 selects a set of training data from the training data set 41 based on the estimated degree. The information processing device 1 generates new training data based on the selected set of training data. The information processing device 1 trains parameters of the machine learning model based on the generated new training data and the plurality of training data.

[0147] As a result, in the information processing device 1, it is possible to obtain a highly accurate machine learning model in which the generalization performance of the machine learning model (classifier) ​​is improved and the evaluation index for the machine learning model is improved.

[0148] In addition, the information processing device 1 prepares a pair of training data consisting of first training data selected from a labeled training dataset 41a including multiple labeled training data, and second training data selected from an unlabeled training dataset 41b including multiple unlabeled training data to which a pseudo label has been assigned.

[0149] As a result, in the information processing device 1, a machine learning model can be obtained by utilizing the unlabeled training data through semi-supervised learning based on a combination of labeled training data and unlabeled training data.

[0150] In addition, when training a machine learning model based on new training data, the information processing device 1 trains the parameters of the machine learning model based on the value of the loss function when the label assigned to the first training data that is the basis of the new training data is used as the label of the new training data.

[0151] As a result, in the information processing device 1, a more accurate machine learning model can be obtained using labels that are more reliable than the pseudo labels that are pre-assigned to the training data, rather than using pseudo labels that are less reliable.

[0152] Next, an example of a hardware configuration of a computer that realizes the same functions as the information processing device 1 described in the above embodiment will be described. Fig. 3 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device of this embodiment.

[0153] 3, the computer 200 has a CPU 201 that executes various arithmetic processes, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that transmits and receives data to and from external devices, etc., via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various information, and a hard disk device 207. The devices 201 to 207 are connected to a bus 208.

[0154] The hard disk device 207 has an estimation program 207a, a selection program 207b, a training data generation program 207c, and a training execution program 207d. The CPU 201 reads out each of the programs 207a to 207d and loads them in the RAM 206.

[0155] The estimation program 207a functions as an estimation process 206a. The selection program 207b functions as a selection process 206b. The training data generation program 207c functions as a training data generation process 206c. The training execution program 207d functions as a training execution process 206d.

[0156] The processing of the estimation process 206a corresponds to the processing of the estimation unit 51. The processing of the selection process 206b corresponds to the processing of the selection unit 52. The processing of the training data generation process 206c corresponds to the processing of the training data generation unit 53. The processing of the training execution process 206d corresponds to the processing of the training execution unit .

[0157] It is not necessary that each of the programs 207a to 207d is stored in the hard disk device 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207d.

Claims

1. Estimating a degree of improvement of an evaluation index for a machine learning model when the machine learning model is trained based on a combination of training data included in a training dataset; selecting a set of training data from the training data set based on the estimated degree; generating other training data based on the selected set of training data; training the machine learning model based on the generated other training data and the training dataset; A machine learning method characterized in that processing is executed by a computer.

2. The machine learning method according to claim 1 , wherein the estimating process includes estimating based on an average of features of training data belonging to a specific class.

3. the selecting process includes a process of selecting a first label and a second label from among a plurality of labels assigned to the training data set based on the degree, and selecting, from the training data set, a first training data set assigned with the first label and a second training data set assigned with the second label as the set of training data; The machine learning method according to claim 1 .

4. the first label is a correct label to be assigned to the first training data; The second label is a pseudo label output by inputting the second training data into the machine learning model before training. The machine learning method according to claim 3 .

5. The training process includes, in training the machine learning model based on new training data, training parameters of the machine learning model based on a value of a loss function when a label assigned to the first training data that is the basis of the new training data is used as a label of the new training data. The machine learning method according to claim 3 .

6. Estimating a degree of improvement of an evaluation index for a machine learning model when the machine learning model is trained based on a combination of training data included in a training dataset; selecting a set of training data from the training data set based on the estimated degree; generating other training data based on the selected set of training data; training the machine learning model based on the generated other training data and the training dataset; An information processing device comprising a control unit.

7. The information processing apparatus according to claim 6 , wherein the control unit performs the estimation based on an average of features of training data belonging to a specific class.

8. the control unit selects a first label and a second label from among a plurality of labels assigned to the training data set based on the degree, and selects, from the training data set, first training data assigned with the first label and second training data assigned with the second label as the pair of training data; 7. The information processing apparatus according to claim 6,

9. the first label is a correct label to be assigned to the first training data; The second label is a pseudo label output by inputting the second training data into the machine learning model before training.

9. The information processing apparatus according to claim 8,

10. The control unit, in training the machine learning model based on new training data, trains parameters of the machine learning model based on a value of a loss function when a label assigned to the first training data on which the new training data is based is used as a label of the new training data.

9. The information processing apparatus according to claim 8,

Images