Training data evaluation system and program of same
The learning data evaluation system addresses the inadequacy of existing methods by evaluating the similarity of output histograms to refine learning datasets, ensuring they contain necessary information, thus enhancing machine learning model performance.
Patent Information
- Application Number
- PCT/JP2023/019492
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-28
AI Technical Summary
Existing methods for evaluating the superiority or inferiority of learning datasets for machine learning models are inadequate, as they may contain biases and do not ensure that the dataset contains all necessary information, leading to suboptimal model performance.
A learning data evaluation system and program that evaluates the similarity between the output histograms of a machine learning model trained on a learning dataset and a test dataset, using a cycle-by-cycle approach to update the dataset by adjusting weights and deleting or adding data based on deviation and similarity metrics, ensuring the dataset contains sufficient information.
This approach allows for the appropriate selection of learning datasets by evaluating their superiority or inferiority through output histogram similarity, ensuring the dataset contains necessary information, thereby improving model performance by iteratively refining the dataset.
Smart Images

Figure JP2023019492_28112024_PF_FP_ABST
Abstract
Description
Learning data evaluation system and its program
[0001] The present invention relates to a training data evaluation system and a program for evaluating the merits of a training data set used in training a machine learning model.
[0002] In the field of machine learning, the selection and construction of training datasets, which are collections of multiple data sets used to train machine learning models, is an important factor that determines the performance of the model. This is because the constructed training dataset does not necessarily contain all the information to be learned. Furthermore, if there is a bias in features that depend on the properties of the data, such as time or space, or if there is a bias toward a specific type of data even though many types of data are included, the performance of the machine learning model may deteriorate or be overestimated beyond its actual performance. Therefore, various studies have been attempted to select training data.
[0003] For example, Patent Literature 1 discloses an information processing method that can easily prepare a training dataset for ensuring the performance of a machine learning model. Specifically, first, a machine learning model is trained using a training dataset. Next, test data is input to the trained machine learning model, and the performance of the machine learning model, for example, whether the accuracy rate satisfies a predetermined condition, is evaluated based on the output. If the performance of the machine learning model is evaluated as not satisfying the predetermined condition, the training dataset is updated, and learning and evaluation are repeated until the predetermined condition is satisfied. The training dataset is updated by extracting input data corresponding to a label determined to have an insufficient number of input data from the input data stored in the log storage, and adding training data including this input data and the label.
[0004] Furthermore, Patent Literature 2 discloses a training dataset generation system that generates a training dataset for machine learning a neural network model for inferring objects in an image. Specifically, first, a test image included in a test dataset is input to a current trained neural network model, and an inference process is performed on the objects in the test image. Next, based on the inference results, the bias of the training data in the training dataset of the current trained neural network model is verified. Next, the probability distribution of the entities of each component in a specific environment reflected in the training image is changed so as to reduce the bias of the training data identified in this verification. Then, a training dataset corresponding to the probability distribution of the entities of each component after the change is generated, and an updated training dataset is generated based on the training data in this training dataset. This makes it possible to accurately reproduce the probability distribution of the training target, thereby generating a highly accurate trained neural network model.
[0005] JP 2022-99685 A JP 2021-111101 A
[0006] An object of the present invention is to provide a novel evaluation method for evaluating the merits and demerits of a training dataset.
[0007] To solve this problem, a first invention provides a training data evaluation system that has a training data setting unit, a model training unit, a model testing unit, and an evaluation index calculation unit, and that evaluates the superiority or inferiority of a training dataset used to train a machine learning model. The training data setting unit sets a training dataset to be evaluated. The model training unit trains a machine learning model using the training dataset. The model testing unit tests the machine learning model trained by the model training unit using a test dataset. The evaluation index calculation unit calculates the similarity between an output histogram of the machine learning model for inputs from the training dataset and an output histogram of the machine learning model for inputs from the test dataset as an evaluation index indicating the superiority or inferiority of the training dataset.
[0008] In the first aspect of the present invention, the training data setting unit may set an evaluation target on a cycle-by-cycle basis by selecting, for each cycle, a training data set having a different data combination from among pre-prepared data sets. In this case, it is preferable that the model training unit trains a machine learning model for each cycle, the model testing unit tests the machine learning model for each cycle, and the evaluation index calculation unit calculates an evaluation index for each cycle. Furthermore, a training data selection unit may be provided that selects the best training data set by comparing multiple evaluation indexes calculated for each cycle.
[0009] In the first invention, the learning data setting unit may perform an update process of the learning data set to be used in the next cycle based on the deviation for each test data calculated using the output of the machine learning model for a certain test data input and the average output of the machine learning model for the learning data input.
[0010] In the first invention, the learning data setting unit may perform an update process of the learning data set to be used in the next cycle based on the difference between the evaluation index of the current cycle and the evaluation index of the previous cycle, and the difference between the learning data set of the current cycle and the learning data set of the previous cycle.
[0011] In the first aspect of the present invention, the training data setting unit may perform processing to update the training data set to be used in the next cycle by randomly deleting a part of the training data set used in the current cycle.
[0012] A second invention provides a training data evaluation program that causes a computer to execute a process having the following steps to evaluate the merits of a training dataset used in training a machine learning model. In a first step, a training dataset to be evaluated is set. In a second step, a machine learning model is trained using the training dataset. In a third step, the machine learning model trained by the model training unit is tested using a test dataset. In a fourth step, the similarity between the output histogram of the machine learning model for inputs from the training dataset and the output histogram of the machine learning model for inputs from the test dataset is calculated as an evaluation index indicating the merits of the training dataset.
[0013] In the second invention, the first step may set an evaluation target on a cycle-by-cycle basis by selecting, for each cycle, a training dataset having a different data combination from among previously prepared datasets. In this case, it is preferable that the second step trains a machine learning model for each cycle, the third step tests the machine learning model for each cycle, and the fourth step calculates an evaluation index for each cycle. Also, a fifth step may be provided in which the best training dataset is selected by comparing the multiple evaluation indexes calculated for each cycle.
[0014] In a second invention, the first step may update the learning dataset to be used in the next cycle based on the deviation for each test data calculated using the output of the machine learning model for a certain test data input and the average output of the machine learning model for the learning data input.
[0015] In a second invention, the first step may perform an update process of the learning dataset to be used in the next cycle based on the difference between the evaluation index of the previous cycle and the evaluation index of the current cycle, and the difference between the learning dataset of the previous cycle and the learning dataset of the current cycle.
[0016] In a second aspect of the present invention, the first step may include updating the training data set to be used in the next cycle by randomly deleting a portion of the training data set used in the current cycle.
[0017] According to the present invention, the similarity between the output histogram of a machine learning model for a training dataset input and the output histogram of a machine learning model for a test dataset input is evaluated. A high similarity between the two indicates that the training dataset contains sufficient information to be learned. Therefore, by using the similarity of the output histograms as an evaluation index indicating the quality of the training data, it becomes possible to appropriately select a training dataset.
[0018] Block diagram of the learning data evaluation system Flowchart of the learning data evaluation process Diagram showing the initialized learning weight array Diagram showing the initialized learning weight array Diagram of the separating hyperplane Diagram of the output histogram of the machine learning model Flowchart of the learning dataset update process Diagram showing the learning weight array after weight adjustment for the learning forgotten portion Diagram showing the deviation array Diagram showing the learning weight array after weight adjustment for the learning memorized portion Explanatory diagram of the deviation
[0019] FIG. 1 is a block diagram of a training data evaluation system according to this embodiment. This training data evaluation system 1 trains a machine learning model using a training dataset and tests the machine learning model using a test dataset, and outputs the best training dataset as a selection result based on an evaluation index that evaluates the merits and demerits of the training dataset. The best training dataset output as the selection result may be a single one, or multiple training datasets may be output, provided that they satisfy a predetermined criterion. The training data evaluation system 1 is primarily comprised of a training data setting unit 2, a model training unit 3, a model testing unit 4, an evaluation index calculation unit 5, a termination determination unit 6, and a training data selection unit 7. The processing system comprised of the training data setting unit 2, the model training unit 3, the model testing unit 4, the evaluation index calculation unit 5, and the termination determination unit 6 forms a feedback loop, whereby loop processing is repeated for each evaluation target while changing the combination of training datasets to be evaluated. The training data selection unit 7 is provided outside the loop processing system and is processed after the loop processing is completed.
[0020] The learning data evaluation system 1 can be realized as dedicated hardware, but it may also be realized on a general-purpose computer by installing a computer program (learning data evaluation program) that executes the processing described below (including implementation on the cloud).
[0021] FIG. 2 is a flowchart showing the procedure for training data evaluation processing. A dataset, which is a collection of a large number of pieces of data used to create a machine learning model, is prepared in advance and divided into a training dataset and a test dataset. The training dataset is a collection of multiple pieces of data used to train the machine learning model. This training adjusts the internal parameters of the machine learning model. The test dataset is a collection of multiple pieces of data used to test the trained machine learning model. This test makes it possible to evaluate the output performance of the machine learning model. In this embodiment, as an example, a case will be described in which a portion of a prepared dataset is used as the training dataset and the entire dataset is used as the test dataset.
[0022] First, in step S1, the training data setting unit 2 extracts a portion of a pre-prepared dataset (a collection of numerous data, N>2) and sets n (N>n≧2) training datasets to be evaluated in the current cycle. In this embodiment, the N pieces of data constituting the dataset are managed by a training weight array that indicates which data should be used for training. The training weight array has a length equal to the total number of data, and manages weights by linking them to IDs that identify each piece of data. For example, as shown in FIG. 3 as a relatively simple example, if three pieces of data designated by ID1, ID3, and ID4 out of five pieces of data designated by IDs 0 to 4 are to be used as the training dataset, the weights of these pieces are set to "1" (used as training data), and the weights of the other pieces are set to "0" (not used as training data). The initial weight values ("0" / "1") are determined randomly. The proportion of "1"s is determined by manually assigning a fixed value or by specifying the proportion relative to the total number of data. Data groups in the training weight array with a weight of "1" or greater are transferred to the model training unit 3 as the training dataset to be processed in the current cycle.
[0023] In this case, a value greater than "1" may be set as the weight of data in the learning weight array. For example, as shown in Figure 4, the weight of the data of ID4 may be set to "0" and the weight of the data of ID1 may be set to "2" instead (the total weight is "3" as in Figure 3). In this case, the data of ID1 is input twice (oversampling), which means that this data is an important element for learning.
[0024] 5 is an explanatory diagram of a separating hyperplane that separates data of class A and data of class B. In the figure, the weight of the data defining the separating hyperplane is represented by the size of the circle. Adjusting the number of times each piece of data is input is equivalent to the task of adjusting the separating hyperplane in a data-driven manner, for example, in a linear discrimination method, and makes it possible to control the weight of the data when defining the separating hyperplane.
[0025] Next, in step S2, the model training unit 3 trains a machine learning model using the training dataset set in step S1. The trained machine learning model is stored in a storage unit (not shown) included in the training data evaluation system 1.
[0026] Next, in step S3, the model testing unit 4 tests the trained machine learning model obtained in step S2 using the above-mentioned test dataset. The output results obtained by this test, i.e., a collection of output results for each input, are stored in a storage unit provided in the training data evaluation system 1.
[0027] Next, in step S4, the evaluation index calculation unit 5 evaluates the merits or demerits of the training dataset processed in the current cycle based on the output result obtained in step 3, and calculates an evaluation index that quantifies this. Essentially, the matching between the training dataset and the test dataset for a certain label is evaluated. This is based on the knowledge that if learning is perfect, the output characteristics of a machine learning model using the training dataset and the output characteristics of a machine learning model using the test dataset should tend to match. Therefore, the similarity between the output histogram of the machine learning model for input of the training dataset and the output histogram of the machine learning model for input of the test dataset is evaluated.
[0028] 6 is an explanatory diagram of the output histogram of a machine learning model that obtains one output for a certain data input. As shown in FIG. 6(a), if the similarity of the output histograms between the test data set and the training data set is low, in other words, if the output histogram of the training data set deviates from that of the test data set, it means that the selected training data set does not contain sufficient information to be learned and is an inferior training data set. In contrast, as shown in FIG. 6(b), if the similarity of the output histograms between the test data set and the training data set is high, it means that the selected training data set contains sufficient information to be learned and is an excellent training data set.
[0029] Typically, the evaluation index for expressing the similarity of such histograms can be the correlation or intersection between histograms. For example, the correlation of histograms can be evaluated by comparing the output histogram of the training data with h t , the output histogram of the test dataset is h e , the average of each / h t , / h e Then, it is calculated based on the following formula 1. The correlation value ranges from -1 to 1, and the higher this value (closer to 1), the smaller the deviation of the training dataset from the test dataset, meaning that all the features of the test dataset have been learned.
[0030]
[0031] The intersection of the histograms is calculated based on the following formula 2. The sum of the differences between the elements of the histogram is evaluated in a range of 0 to 1. The closer this value is (closer to 1), the smaller the deviation of the training dataset from the test dataset, meaning that all the features of the test dataset have been learned.
[0032]
[0033] If the output of the machine learning model is multidimensional, the similarity of the histograms for each output is evaluated, and the sum or average of these is used as the evaluation index. Other indices that clarify the difference between the training dataset and the test dataset, such as the area under the curve (AUC) or average precision, can also be used as the evaluation index.
[0034] In step S5, the termination determination unit 6 determines whether the current cycle satisfies a predetermined termination condition, and if the termination condition is not satisfied, the process returns to step S1 and starts the processing of the next cycle. The termination condition may be that the number of loops in the cycle has reached a predetermined number, or that an evaluation index equal to or greater than a predetermined threshold value has been obtained in the current cycle.
[0035] In the next cycle, first, in step S1, the learning data setting unit 2 changes the combination of learning data set set in the previous cycle (accompanying a change in weights) and sets a new learning data set to be processed (updating the learning data). Fig. 7 is a flowchart showing the detailed procedure of the learning data set update process.
[0036] In step S11, data equivalent to the learning forgetting rate is deleted from the training data set processed in the previous cycle. The reason for deleting data equivalent to the learning forgetting rate is to avoid the possibility of falling into a local optimum while maintaining the information contained in the training data set. Here, the learning forgetting rate refers to the percentage of data whose weights are to be reduced relative to the total number of data in the training data set, and 100% (i.e., total data replacement) is excluded. Specifically, 1 is subtracted from the weight of data randomly selected from data having a weight of "1" or greater in the training weight array. For example, if the learning forgetting rate is 20%, the weight of data ID4, which is randomly selected from the five data in the training weight array shown in FIG. 3, is changed from "1" to "0," and the training weight array is updated to the one shown in FIG. 8.
[0037] In step S12, data for the learning retention rate is registered based on the deviation for the learning weight array in which the weight adjustment for the learning forgetting portion was performed in step S11. Here, the learning retention rate refers to the ratio of the number of data to be weighted in the learning data set to the total number of data. The deviation is the difference in the model output of the test data set with the representative value of the learning data set as the reference. For example, when the average output in the learning data set is used as the reference value, the deviation d for certain test data a is a is calculated based on the following formula: where v a is the model output for test data a, / v t is the average value of the model output in the training data.
[0038]
[0039] As shown in FIG. 9 , the deviation is calculated individually for each test data set. Then, for example, if the learning memory rate is 40%, one is added to the weights of the top two data with the highest deviations (corresponding to 40%) among the five data in the learning weight array shown in FIG. 8 . As a result, as shown in FIG. 10 , the weight of the data with ID0 is changed from "0" to "1," and the weight of the data with ID1 is changed from "1" to "2." As shown in FIG. 11 , the data with ID0 and ID1 are data with high deviations. By including these in the training data set, information that was missing from the previous training data set is incorporated, resulting in a better training data set.
[0040] The training data set updated through the processes of steps S11 and S12 above is used as the processing target in a new cycle, and the processes of steps S2 to S4 following step S1 are performed. The processes of steps S1 to S4 are repeated in cycles, changing the training data set with a different data combination for each cycle, until the termination condition of step S5 is met. Then, when the termination condition of step S5 is met, the process proceeds to step S6.
[0041] In step S6, the termination determination unit 6 compares the multiple evaluation indexes calculated for each cycle to select at least one training data set as the best, and outputs the result as the selection result.
[0042] As described above, according to this embodiment, the similarity between the output histogram of a machine learning model for a training dataset input and the output histogram of a machine learning model for a test dataset input is evaluated. A high degree of similarity between the two means that the selected training dataset contains sufficient information that should be learned. Therefore, by using the similarity of the output histograms as an evaluation index, it is possible to appropriately evaluate the merits and demerits of training datasets.
[0043] Furthermore, according to this embodiment, the training data selection unit 7 is provided, and by comparing multiple evaluation indices calculated for each cycle and identifying the best one, the training data evaluation system 1 can automatically select and output the best training data set.
[0044] Furthermore, according to this embodiment, when updating the learning dataset (registering data for the learning memory rate), the deviation degree for each test data is taken into consideration, thereby making it possible to efficiently derive a better learning dataset.
[0045] The learning weight update method described in the above embodiment is the simplest one, and the present invention is not limited to this. For example, adaptive weighting as described below may be adopted.
[0046] Specifically, in the update of the learning weights, which adjusts the learning rate depending on the increase or decrease of the evaluation index of the data set obtained by updating the learning weights, the learning weight array in a certain cycle c (c≧2) is expressed as X c , and the learning weight array one cycle before that is X c-1 , the evaluation index of cycle c is q c , and the evaluation index one cycle before that is q c-1 , the reward coefficient is λ, the learning weight array X c+1 is calculated based on the following formula:
[0047]
[0048] In the above weight update formula, the evaluation index q of this cycle c and the evaluation index q of the previous cycle c-1 The difference between this and the learning data set for this cycle (learning weight array X c ) and the training data set from the previous cycle (training weight array is X c-1 ), the training data set for this cycle (training weight array X c ) to adjust the training data set (training weight array to X c+1) is updated. That is, by evaluating the weight adjustment amount for the next cycle c+1 related to the change in the learning weight based on the magnitude (increase or decrease) of the evaluation index between cycles c and c-1, it is possible to efficiently determine the learning weight that provides a better evaluation index. Note that when c=1, the above weight update formula cannot be applied, so a random weight array or a fixed value array must be provided to X.
[0049] In the above weight update formula, it is possible to search for better learning weights by adding an auxiliary term as shown in the following formula. Here, rand is a function that outputs real-valued Gaussian noise in the range 0 to 1, which has the same size as the input matrix, and l o is the learning forgetting rate.
[0050]
[0051] In general, the iterative weight update given by Equation 4 may fall into a local optimum depending on the initial value. Therefore, by providing a term that randomly subtracts the weight as in Equation 5, it is possible to eliminate bias in the weight to some extent and effectively avoid local optimum.
[0052] As a further measure to avoid local optimization, it is effective to provide an item that suddenly gives a large learning weight with a certain probability, with the aim of broadening the search range of the learning weight, in order to obtain a better learning weight array. For example, t , learning memory rate is l m Then, the learning weight array given by the following formula can improve the possibility of arriving at a better global optimum by searching a wider search space: where poisson is a function that outputs values sampled from a Poisson distribution whose expected value is the element of an input matrix that is equal in size to the input matrix, and norm is a normalization function.
[0053]
[0054] Furthermore, the present invention can also be understood as a learning data evaluation program that causes a computer to execute the processes shown in the flowcharts of FIGS.
[0055] 1 Learning data evaluation system 2 Learning data setting unit 3 Model learning unit 4 Model testing unit 5 Evaluation index calculation unit 6 End determination unit 7 Learning data selection unit
Claims
1. A learning data evaluation system for evaluating the superiority or inferiority of a learning dataset used to train a machine learning model, comprising: a learning data setting unit that sets a learning dataset to be evaluated; a model learning unit that trains a machine learning model using the learning dataset; a model testing unit that tests the machine learning model trained by the model learning unit using a test dataset; and an evaluation index calculation unit that calculates a similarity between an output histogram of the machine learning model for an input of the learning dataset and an output histogram of the machine learning model for an input of the test dataset as an evaluation index indicating the superiority or inferiority of the learning dataset.
2. The learning data evaluation system described in claim 1, characterized in that the learning data setting unit sets the evaluation target on a cycle-by-cycle basis by selecting a learning data set having a different combination of data from pre-prepared data sets for each cycle, the model learning unit learns the machine learning model for each cycle, the model testing unit tests the machine learning model for each cycle, and the evaluation index calculation unit calculates the evaluation index for each cycle.
3. The training data evaluation system according to claim 2, further comprising a training data selection unit that selects the best training data set by comparing the multiple evaluation indexes calculated for each cycle.
4. The learning data evaluation system described in claim 2, characterized in that the learning data setting unit performs an update process for the learning data set to be used in the next cycle based on the deviation for each test data calculated using the output of the machine learning model for a certain test data input and the average output of the machine learning model for the learning data input.
5. The learning data evaluation system described in claim 2, characterized in that the learning data setting unit performs an update process for the learning data set to be used in the next cycle based on the difference between the evaluation index of the current cycle and the evaluation index of the previous cycle, and the difference between the learning data set of the current cycle and the learning data set of the previous cycle.
6. A learning data evaluation system as described in claim 4 or 5, characterized in that the learning data setting unit updates the learning data set to be used in the next cycle by randomly deleting a portion of the learning data set used in the current cycle.
7. A learning data evaluation program for evaluating the superiority or inferiority of a learning dataset used in training a machine learning model, the program causing a computer to execute a process including: a first step of setting a learning dataset to be evaluated; a second step of training a machine learning model using the learning dataset; a third step of testing the machine learning model trained by the model training unit using a test dataset; and a fourth step of calculating a similarity between an output histogram of the machine learning model for an input of the learning dataset and an output histogram of the machine learning model for an input of the test dataset as an evaluation index indicating the superiority or inferiority of the learning dataset.
8. The learning data evaluation program described in claim 7, characterized in that the first step sets the evaluation target on a cycle-by-cycle basis by selecting a learning data set having a different combination of data from pre-prepared data sets for each cycle, the second step trains the machine learning model for each cycle, the third step tests the machine learning model for each cycle, and the fourth step calculates the evaluation index for each cycle.
9. The training data evaluation program described in claim 8, further comprising a fifth step of selecting the best training data set by comparing the multiple evaluation indexes calculated for each cycle.
10. The learning data evaluation program described in claim 8, characterized in that the first step performs an update process for the learning data set to be used in the next cycle based on the deviation for each test data calculated using the output of the machine learning model for a certain test data input and the average output of the machine learning model for the learning data input.
11. A training data evaluation program as described in claim 8, characterized in that the first step includes updating the training data set to be used in the next cycle based on the difference between the evaluation index of the current cycle and the evaluation index of the previous cycle, and the difference between the training data set of the current cycle and the training data set of the previous cycle.
12. A learning data evaluation program as described in claim 10 or 11, characterized in that the first step updates the learning data set to be used in the next cycle by randomly deleting a portion of the learning data set used in the current cycle.
Citation Information
Patent Citations
Data set verification device, data set verification method, and data set verification program
JP2019159538A
System and method for evaluating machine learning model behavior over data segments
US20220012613A1
Information processing device, information processing method, and program
WO2020149073A1
