Machine learning program, method, and device

Abstract

Provided is a machine learning device that: calculates the independence between a prediction result, which is obtained when inputting each of a plurality of pieces of unlabeled data into a machine learning model, and the value of a first attribute of each of the plurality of pieces of data; selects a first piece of data from the plurality of pieces of data on the basis of the independence; obtains a label for the first piece of data; and executes training of the machine learning model on the basis of the first piece of data and the label.

Description

Machine learning program, method, and device

[0001] The disclosed technology relates to a machine learning program, a machine learning method, and a machine learning device.

[0002] Techniques related to machine learning models that take fairness into consideration have been proposed. For example, a learning device has been proposed that inputs training data for training a classifier and a causal graph representing the causal relationships between variables included in the training data. Using the input training data and causal graph, this learning device trains a classifier by solving a constrained optimization problem in which the average of the causal effects between specified variables is within a specified range and the variance of the causal effects is equal to or less than a specified value.

[0003] Furthermore, an information processing device has been proposed that generates training data for making fair judgments on each piece of input data by, for example, artificially increasing data on minority attributes. The information processing device holds first training data used to train a machine learning model, acquires information about bias in the training data, and generates second training data using data included in the training data based on the information about bias. The information processing device then trains the machine learning model using the first training data and the second training data.

[0004] For example, a system has been proposed that labels unlabeled data according to the amount of label bias, samples input data according to the discrepancy between the amount of selection bias and the rarity of features, and trains a classifier using the sampled and labeled data and additional unlabeled data.

[0005] In addition, a method that integrates active learning and semi-supervised learning has been proposed to train machine learning models using fairness-conscious active learning. This method selects the most valuable unlabeled data and sends it to an expert system for labeling. The method then establishes connections between the unlabeled data and labeled data, uses the unique information in the unlabeled data to improve the model, and assigns pseudo-labels to those samples.

[0006] International Publication No. WO 2021 / 084609 International Publication No. WO 2022 / 123907 U.S. Patent Application Publication No. 2020 / 0372406

[0007] Quan Ren, Hongbing Zhang, Dailu Zhang, Xiang Zhao, Lizhi Yan, Jianwen Rui, Fanxin Zeng, Xinyi Zhu, "A framework of active learning and semi-supervised learning for lithology identification based on improved naive Bayes," Expert Systems with Applications, Volume 202, 15 September 2022, 117278.

[0008] However, in conventional learning methods that train machine learning models using active learning that takes fairness into consideration (hereinafter referred to as "fair active learning"), the process of selecting data to be labeled also requires training of the machine learning model, which poses a problem of high processing load.

[0009] In one aspect, the disclosed technology aims to reduce the processing load of fair active learning.

[0010] In one aspect, the disclosed technology calculates independence between a prediction result when each of a plurality of unlabeled data items is input to a machine learning model and a value of a first attribute of each of the plurality of data items, and then selects first data items from the plurality of data items based on the independence, obtains a label for the first data items, and trains the machine learning model based on the first data items and the label.

[0011] As one aspect, the disclosed technology has an effect of reducing the processing load of fair active learning.

[0012] FIG. 1 is a diagram for explaining active learning. FIG. 2 is a diagram for explaining conventional fair active learning. FIG. 3 is a diagram for explaining fair active learning in this embodiment. FIG. 4 is a functional block diagram of a machine learning device according to this embodiment. FIG. 5 is a diagram for explaining an example of the degree of accuracy improvement. FIG. 6 is a block diagram showing a schematic configuration of a computer that functions as a machine learning device. FIG. 7 is a flowchart showing an example of machine learning processing. FIG. 8 is a diagram comparing the prediction accuracy, fairness, and execution time of fair active learning of a machine learning model between the present method and a comparative method.

[0013] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0014] Before describing the details of the embodiment, fair active learning of machine learning models and its challenges will be described.

[0015] First, training a machine learning model involves learning the relationship between data labels (outcome variables) and features (explanatory variables) and identifying parameters that approximate that relationship. Furthermore, fairness in machine learning refers to the absence of bias or discrimination in predictions made by a machine learning model based on the innate or acquired characteristics of individuals or groups in decision-making (hereinafter referred to as "protected attributes"). The social implementation of machine learning models often requires improvements in indicators of fairness and inequality based on groups, such as gender differences in loan screening, racial differences in facial recognition, and age differences in disease diagnosis. Therefore, it is important to train machine learning models to produce fair predictions. Furthermore, because there is a trade-off between fairness and predictive accuracy in machine learning models, it is also important to strike a balance between these two.

[0016] Training a machine learning model using supervised learning requires data with labels indicating the correct answer (hereinafter referred to as "labeled data"). Labeled data is data tagged with one or more labels. The labels are usually tagged by humans or other information sources such as oracles. Without a sufficient amount of labeled data, the fairness and predictive accuracy of the machine learning model cannot be sufficiently improved. However, labeled data is more expensive to collect than unlabeled data (hereinafter referred to as "unlabeled data").

[0017] Active learning is an interactive machine learning technique that improves machine learning models by asking questions. Specifically, an information processing device that performs active learning performs the following processes, as shown in Figure 1: (1) data selection, (2) questioning, (3) answering, (4) training, and (5) propagation.

[0018] More specifically, in the process of "(1) data selection," the information processing device calculates a data acquisition function for each piece of data included in the unlabeled dataset, and prioritizes and selects data useful for training the machine learning model based on the data acquisition function. The data acquisition function is an index that uses information such as the current parameters of the machine learning model to represent the ambiguity of the prediction made by the machine learning model for each piece of data, the representativeness of each piece of data relative to the unlabeled dataset, and so on. In the process of "(2) question," the information processing device queries an oracle for the label of the selected data. In the process of "(3) answer," the information processing device obtains a label, which is the answer from the oracle, and assigns the obtained label to the selected data to create labeled data, which is added to the labeled dataset. In the process of "(4) training," the information processing device trains the machine learning model using the labeled dataset. In the process of "(5) transmission," the information processing device transmits information such as the parameters of the machine learning model after training to the process of (1) data selection.

[0019] In active learning, the above steps (1) to (5) are repeatedly executed, and unlabeled data that is useful for training a machine learning model is preferentially labeled, allowing labeled data to be collected effectively.

[0020] 1, circles represent each data, white circles represent unlabeled data, hatched circles represent labeled data, and different hatching indicates different labels. The same applies to the following FIGS. 2 and 3.

[0021] In typical active learning, (1) data selection involves selecting data that helps improve the predictive accuracy of the machine learning model, without considering fairness. As a result, as labeled data is added and the machine learning model progresses, fairness can deteriorate. For example, in a machine learning model for facial expression recognition, data of a specific race may be exclusively selected from the unlabeled data set.

[0022] Therefore, in conventional fair active learning, (1) in the data selection process, data is selected taking into consideration the trade-off between fairness and prediction accuracy. Specifically, as shown in Figure 2, in conventional fair active learning, a portion of an unlabeled data set is designated as an unlabeled validation data set, and the remainder is designated as an unlabeled candidate data set. An information processing device that performs conventional fair active learning executes the processes of (A) provisional question, (B) provisional answer, (C) training, (D) evaluation, and (E) selection shown in Figure 2 to estimate the degree of unfairness improvement of a machine learning model in validation data for each candidate data.

[0023] More specifically, in the process of "(A) temporary question," the information processing device inputs each piece of candidate data into a labeling model that outputs a temporary label for the data. Furthermore, in the process of "(B) temporary answer," the information processing device acquires the temporary label output from the labeling model and assigns the acquired temporary label to each piece of candidate data to create a temporary-labeled candidate data set. Furthermore, in the process of "(C) training," the information processing device trains a machine learning model using the temporary-labeled candidate data set. Furthermore, in the process of "(D) evaluation," the information processing device uses an unlabeled validation data set to evaluate the fairness of each piece of temporary-labeled candidate data, taking into account the difference in unfairness of the machine learning model before and after training. Furthermore, the information processing device evaluates the prediction accuracy of the machine learning model using the unlabeled validation data set. Furthermore, in the process of "(E) selection," the information processing device selects the candidate data with the best value based on an index that takes into account the trade-off between fairness and prediction accuracy.

[0024] In conventional fair active learning, in "(1) data selection," training of a machine learning model is performed, and data that improves prediction accuracy is selected taking into account the decision boundary of the machine learning model. In addition, data that improves fairness is selected through evaluation using validation data. However, training a machine learning model imposes a high processing load, and there is a problem in that data selection cannot be performed efficiently. In particular, when the machine learning model is a complex model such as a deep learning model or a nonlinear model, it is difficult to apply fair active learning within a realistic execution time.

[0025] Therefore, in this embodiment, as shown in Figure 3, the fairness of each candidate data is evaluated based on the independence of the model's prediction results for unlabeled data and the values ​​of the protection attributes, without requiring training of a machine learning model.

[0026] Another approach to solving the above problem is to estimate the degree of unfairness of each candidate data based on the prediction results of a machine learning model for each candidate data, and select data with a low degree of unfairness from the unlabeled candidate data set. However, in this case, the selected data does not take into account the impact on the validation data, and therefore outliers or similar data are likely to be selected, resulting in low data representativeness. Therefore, in this embodiment, the prediction results of the validation data when candidate data are given are used as the prediction results of the model for unlabeled data. The machine learning device according to this embodiment will be described below.

[0027] As shown in Figure 4, a labeled dataset 20 and an unlabeled dataset 22 are input to the machine learning device 10. The amount of labeled data included in the labeled dataset 20 is assumed to be much smaller than the amount of unlabeled data included in the unlabeled dataset 22. The machine learning device 10 selects data from the unlabeled dataset 22 while taking fairness into consideration, and performs training of the machine learning model 24. In other words, the machine learning device 10 performs fair active learning.

[0028] 4 , the machine learning device 10 functionally includes a control unit 11. The control unit 11 further includes a training unit 12, a calculation unit 14, a selection unit 16, and an acquisition unit 18. A machine learning model 24 is stored in a predetermined storage area of ​​the machine learning device 10.

[0029] The training unit 12 uses a plurality of labeled data included in the labeled dataset 20 as training data to train the machine learning model 24. As will be described later, in this embodiment, the acquisition unit 18 adds new labeled data to the labeled dataset 20. When new labeled data is added to the labeled dataset 20, the training unit 12 uses the original labeled data and the added labeled data to train the machine learning model 24.

[0030] The calculation unit 14 calculates the independence between a prediction result (hereinafter referred to as a "predicted label") when each of the multiple unlabeled data included in the unlabeled data set 22 is input to the machine learning model 24 and the value of each protected attribute of the multiple data. The protected attribute is an example of a "first attribute" of the disclosed technology. The calculation unit 14 calculates the mutual information between the predicted label and the value of the protected attribute as the independence. The mutual information is an index that quantitatively represents whether two variables are dependent on each other, and if the two variables are completely independent of each other, the mutual information is 0. In other words, if the mutual information between the predicted label and the value of the protected attribute is 0, it can be said that the machine learning model 24 is completely fair. Therefore, it can be said that the unlabeled data with the smallest mutual information is the most fair data.

[0031] In this embodiment, the calculation unit 14 calculates the degree of unfairness improvement of the machine learning model 24 based on the mutual information between the predicted label and the value of the protected attribute for the validation data, conditioned on each piece of candidate data. Specifically, the calculation unit 14 sets some of the unlabeled data included in the unlabeled data set 22 as validation data, and sets the rest of the unlabeled data as candidate data. The calculation unit 14 calculates the independence between the predicted label and the value of the protected attribute for each piece of validation data, which is conditioned on the independence between the predicted label and the value of the protected attribute for each piece of candidate data.

[0032] More specifically, the calculation unit 14 calculates the predicted label Y of the verification data v. ｖ and the value S of the protection attribute in the verification data v. ｖ The difference between the mutual information I between the candidate data u and the candidate data u before and after the candidate data u is given is defined as the unfairness improvement degree Fｕ and is calculated using the following formula (1).

[0033]

[0034] In formula (1), Y ｕ is the predicted label of candidate data u, S ｕ is the value of the protection attribute in the candidate data u. The first term in Σ on the right side of equation (1) is the mutual information of the verification data v before the candidate data u is given, and the second term is the mutual information of the verification data v after the candidate data u is given. In the case of equation (1), the unfairness improvement degree F ｕ The larger the value of candidate data u, the greater the degree of improvement in unfairness.

[0035] The calculation unit 14 calculates the second term in Σ on the right side of equation (1) as follows: First, the calculation unit 14 converts the second term into equation (2) below: In equation (2), H(X) is the entropy of X.

[0036]

[0037] Next, the calculation unit 14 approximates a probability distribution corresponding to each entropy by Monte Carlo dropout. The calculation unit 14 assumes that the parameters of the machine learning model 24 and the distribution of the predicted labels of the machine learning model 24 are conditionally independent, and calculates Y ｉ The probability of P(Y ｉ ) is calculated. ｉ = {Y ｖ , S ｖ , Y ｕ , S ｕ}.

[0038]

[0039] In equation (3), θ is a parameter of the machine learning model 24, and M is the number of Monte Carlo samplings. The calculation unit 14 calculates the mutual information of equation (2) using the probability distribution of equation (3).

[0040] Furthermore, the calculation unit 14 calculates, for each candidate data, the degree of improvement in the prediction accuracy of the machine learning model 24 due to each candidate data, based on the uncertainty of the prediction result when each candidate data is input to the machine learning model 24. For example, data located near the decision boundary of the machine learning model 24 can be said to be data that is difficult for the machine learning model 24 to judge, and using such data as training data improves the prediction accuracy of the machine learning model 24. For example, in the machine learning model 24, the decision boundary is defined in a feature space as shown in FIG. 5. Note that the example of FIG. 5 shows an example of binary classification of a linear model, and each circle represents the feature value of each data. In this case, data near the decision boundary (e.g., data indicated by a dotted circle in FIG. 5) is unstable as to which label it belongs to, and is useful for improving the accuracy of the machine learning model 24. Therefore, the calculation unit 14 calculates a degree of accuracy improvement that increases as the candidate data approaches the decision boundary of the machine learning model. For example, the calculation unit 14 calculates the predicted label Y of candidate data u as shown in the following equation (4): ｕ The entropy, which indicates the uncertainty of ｕ In equation (4), Y is a set of predicted labels of the machine learning model 24.

[0041]

[0042] Furthermore, the calculation unit 14 calculates the unfairness improvement degree F as shown in the following formula (5), for example. ｕ and accuracy improvement degree A ｕ and the degree of unfairness improvement F ｕ and accuracy improvement degree A ｕ The evaluation value E for each candidate data u is expressed by a coefficient α that represents the trade-off between ｕ Calculate E ｕ = α × F ｕ + (1-α) × A ｕ (5) α is a value between 0 and 1 (for example, 0.6), and is the unfairness improvement degree F ｕ and accuracy improvement degree A ｕ This is a coefficient that determines to what degree the priority is given to either the

[0043] The selection unit 16 selects the evaluation value E for each candidate data u calculated by the calculation unit 14. ｕ The target data to be labeled is selected from the plurality of candidate data u based on the evaluation value E. The target data is an example of the "first data" of the disclosed technology. For example, the selection unit 16 ｕ Alternatively, the candidate data u with the highest evaluation value E ｕ Alternatively, candidate data u may be selected that has an evaluation value E ｕ Alternatively, a predetermined number of top candidate data u may be selected.

[0044] The acquisition unit 18 queries a human or other information source, such as an oracle, for labels of the target data selected by the selection unit 16 and acquires the labels as a response from the oracle. The acquisition unit 18 assigns the acquired labels to the target data to generate labeled data, which is then added to the labeled dataset 20. As a result, as described above, the training unit 12 uses the added labeled data to train the machine learning model 24.

[0045] The machine learning device 10 may be realized by, for example, a computer 40 shown in FIG. 6 . The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes an input / output device 45 such as an input device and a display device, and an R / W (Read / Write) device 46 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 47 that is connected to a network such as the Internet. The CPU 41, GPU 42, memory 43, storage device 44, I / O device 45, R / W device 46, and communication I / F 47 are connected to one another via a bus 48.

[0046] The storage device 44 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage device 44, which serves as a storage medium, stores a machine learning program 50 for causing the computer 40 to function as the machine learning device 10. The machine learning program 50 includes training process control instructions 52, calculation process control instructions 54, selection process control instructions 56, and acquisition process control instructions 58. The storage device 44 also includes an information storage area 60 in which information constituting the machine learning model 24 is stored.

[0047] The CPU 41 reads the machine learning program 50 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions of the machine learning program 50. The CPU 41 operates as the training unit 12 shown in FIG. 4 by executing the training process control instruction 52. The CPU 41 also operates as the calculation unit 14 shown in FIG. 4 by executing the calculation process control instruction 54. The CPU 41 also operates as the selection unit 16 shown in FIG. 4 by executing the selection process control instruction 56. The CPU 41 also operates as the acquisition unit 18 shown in FIG. 4 by executing the acquisition process control instruction 58. The CPU 41 also reads information from the information storage area 60 and loads the machine learning model 24 into the memory 43. As a result, the computer 40 that executed the machine learning program 50 functions as the machine learning device 10. The CPU 41 that executes the program is hardware. A portion of the program may be executed by the GPU 62.

[0048] The functions realized by the machine learning program 50 may be realized, for example, by a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0049] Next, the operation of the machine learning device 10 according to this embodiment will be described. When unbiased active learning is instructed for the machine learning model 24, the machine learning device 10 executes the machine learning process shown in Fig. 7. Note that the machine learning process is an example of a machine learning method of the disclosed technology.

[0050] In step S10, the training unit 12 acquires the labeled dataset 20 and uses the labeled data as training data to train the machine learning model 24. Next, in step S12, the training unit 12 determines whether a termination condition for fair active learning is met. The termination condition may be, for example, when the number of data newly added to the labeled dataset 20 exceeds a predetermined number. If the termination condition is not met, the process proceeds to step S14.

[0051] In step S14, the calculation unit 14 sets some of the unlabeled data included in the unlabeled data set 22 as verification data and the rest as candidate data. Next, in step S16, the calculation unit 14 calculates the predicted label Y of the verification data v, for example, as shown in equation (1): ｖ and the value S of the protection attribute in the verification data v. ｖ The difference between the mutual information I before and after the candidate data u is given is expressed as the unfairness improvement degree F ｕ It is calculated as follows.

[0052] Next, in step S18, the calculation unit 14 calculates the predicted label Y of each candidate data u, for example, as shown in equation (4). ｕ The entropy, which indicates the uncertainty of ｕ Next, in step S20, the calculation unit 14 calculates the unfairness improvement degree F ｕ and accuracy improvement degree A ｕ and the degree of unfairness improvement F ｕ and accuracy improvement degree A ｕ The evaluation value E for each candidate data u is expressed by a coefficient α that represents the trade-off between ｕ Calculate.

[0053] Next, in step S22, the selection unit 16 selects an evaluation value E ｕBased on this, target data to be labeled is selected from the plurality of candidate data u. Next, in step S24, the acquisition unit 18 queries the oracle for the label of the target data and acquires the label as a response from the oracle. Next, in step S26, the acquisition unit 18 assigns the acquired label to the target data to make it labeled data and adds it to the labeled data set 20, and also deletes the candidate data that has become the target data from the unlabeled data set 22, and returns to step S10.

[0054] Returning to step S10, the training unit 12 uses the added labeled data to train the machine learning model 24. Next, if it is determined in step S12 that the termination condition for fair active learning is met, the process proceeds to step S28. In step S28, the training unit 12 outputs a machine learning model trained by fair active learning, and the machine learning process ends.

[0055] As described above, the machine learning device according to this embodiment calculates the independence between the predicted results when each of multiple unlabeled data items is input to a machine learning model and the values ​​of the protection attributes of each of the multiple unlabeled data items. Furthermore, the machine learning device selects target data from the multiple unlabeled data items based on the calculated independence, queries an oracle to obtain a label for the target data, and trains the machine learning model based on the target data and the obtained label. That is, the machine learning device according to this embodiment evaluates the fairness of the unlabeled data based on an information-theoretic approach and selects data to be labeled without retraining the machine learning model. This allows the machine learning device according to this embodiment to reduce the processing load of fair active learning.

[0056] Furthermore, the machine learning device according to this embodiment calculates, for each candidate data, a degree of unfairness improvement using mutual information as the independence between the prediction result and the value of the protection attribute, and calculates a degree of improvement in the accuracy of the machine learning model based on the uncertainty of the candidate data.The machine learning device then selects target data based on an evaluation value that takes into account the trade-off between the degree of unfairness improvement and the degree of accuracy improvement.As a result, the machine learning device according to this embodiment can select as target data data that optimizes the trade-off between the degree of unfairness improvement and the degree of accuracy improvement while reducing the processing load.

[0057] Furthermore, the machine learning device according to this embodiment uses part of the unlabeled data as validation data and the rest as candidate data, and calculates the difference in mutual information between the prediction result of the validation data and the value of the protection attribute before and after providing the candidate data as the degree of unfairness improvement. In this way, by calculating the degree of unfairness improvement for the candidate data while taking into account the impact on the validation data, it becomes easier to select highly representative data from the candidate data.

[0058] FIG. 8 shows a schematic comparison of the prediction accuracy, fairness, and execution time of fair active learning of the machine learning model between the method of this embodiment (hereinafter referred to as "this method") and a comparative method. The comparative method here is a method that requires re-learning of the machine learning model when selecting data to be labeled, like the method described in Non-Patent Document 1. This method has the same trade-off between prediction accuracy and fairness as the comparative method. Furthermore, this method significantly reduces execution time compared to the comparative method. The theoretical computational cost of fair active learning is O((T+N ｖ ) CN ｕ ) and this method is "O(N ｕ N ｖ C ２ M), where T is the computational cost of training the machine learning model, and N ｖ is the number of validation data, N ｕ is the number of candidate data, C is the number of labels, and M is the number of Monte Carlo samplings.

[0059] In the above embodiment, a case has been described in which part of the unlabeled data is used as verification data to evaluate fairness, but the present invention is not limited to this. All data included in the unlabeled data set may be used as candidate data. In this case, for example, an equation obtained by excluding the second term in Σ on the right side of equation (1) may be used as the fairness index. Furthermore, in the above embodiment, a case has been described in which data is selected using an evaluation value that takes into account the trade-off between fairness and prediction accuracy, but data may also be selected taking only fairness into account.

[0060] In the above embodiment, a case has been described in which mutual information is used as an index showing the independence between the prediction result of a machine learning model for unlabeled data and the value of a protected attribute. The mutual information can be any index that mathematically defines the independence between the prediction result and the value of a protected attribute, and examples of such indexes include Kullback-Leibler divergence, Jensen-Shannon divergence, covariance, demographic parity difference, and disparate impact ratio.

[0061] In the above embodiment, the machine learning program is stored (installed) in advance in a storage device, but this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0062] REFERENCE SIGNS LIST 10 Machine learning device 11 Control unit 12 Training unit 14 Calculation unit 16 Selection unit 18 Acquisition unit 20 Labeled data set 22 Unlabeled data set 24 Machine learning model 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 Input / output device 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 50 Machine learning program 52 Training process control command 54 Calculation process control command 56 Selection process control command 58 Acquisition process control command 60 Information storage area

Claims

1. Calculating the independence between a prediction result when each of a plurality of unlabeled data items is input to a machine learning model and a value of a first attribute of each of the plurality of data items; selecting a first piece of data from the plurality of pieces of data based on the independence; Obtaining a label for the first data; training the machine learning model based on the first data and the labels; A machine learning program that causes a computer to execute processing.

2. The machine learning program according to claim 1 , wherein the independence is a mutual information between the prediction result and the value of the first attribute.

3. The process of selecting the first data includes selecting the first data based on a degree of improvement in prediction accuracy of the machine learning model by each of the plurality of data, which is based on uncertainty of a prediction result when each of the plurality of data is input to the machine learning model, and the independence.

3. The machine learning program according to claim 1 or 2.

4. The degree of improvement is higher as the first data is closer to the decision boundary of the machine learning model.

4. The machine learning program according to claim 3.

5. the process of selecting the first data includes selecting data for which an index represented by the degree of improvement, the independence, and a coefficient representing a trade-off between the degree of improvement and the independence is equal to or greater than a predetermined value, or for which the index is in a top predetermined number of positions; 4. The machine learning program according to claim 3.

6. the process of calculating the independence includes: setting a part of the plurality of data as verification data and the remaining part of the plurality of data other than the verification data as candidate data; and calculating the independence between a prediction result when each of the verification data is input to the machine learning model and a value of the first attribute of each of the verification data, the independence being conditioned on the independence between a prediction result when each of the candidate data is input to the machine learning model and a value of the first attribute of each of the candidate data; the process of selecting the first data includes selecting the first data from the candidate data; 3. The machine learning program according to claim 1 or 2.

7. Calculating the independence between a prediction result when each of a plurality of unlabeled data items is input to a machine learning model and a value of a first attribute of each of the plurality of data items; selecting a first piece of data from the plurality of pieces of data based on the independence; Obtaining a label for the first data; training the machine learning model based on the first data and the labels; A machine learning method characterized in that the processing is executed by a computer.

8. Calculating the independence between a prediction result when each of a plurality of unlabeled data items is input to a machine learning model and a value of a first attribute of each of the plurality of data items; selecting a first piece of data from the plurality of pieces of data based on the independence; Obtaining a label for the first data; training the machine learning model based on the first data and the labels; A machine learning device comprising a control unit that executes processing.