Evaluation program, method, and device

Abstract

To provide an evaluation program, method, and device carrying out comparative evaluation on a plurality of machine learning models through securing statistical reliability and fairness.SOLUTION: An evaluation method comprises: a training section 12 training a plurality of first machine learning models different to each other using training data with a plurality of correct labels; a generation section 14 generating a plurality of data for evaluation which correct labels are unknown and similarity levels with the plurality of training data with the correct labels are equal to or lower than specific value; a prediction section 16 acquiring prediction results by each of the first machine learning models and a second machine learning models for the plurality of evaluation data for the evaluation; and an evaluation section 18 outputting parameters showing ability upon optimization through inputting the predicted results to a probability model including parameters showing ability of each of the first machine learning models and the second machine learning models for the plurality of evaluation data for the evaluation and parameters showing correct labels of the plurality of data for the evaluation and showing probability acquiring the predicted results by each of the first machine learning models and the second machine learning models as evaluation indexes showing ability of each of the second learning models.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The disclosed technology relates to an evaluation program, an evaluation method, and an evaluation device. [Background technology]

[0002] A method for evaluating the reliability of an estimated value for an input value by machine learning has been proposed. In this method, a learning process is performed by machine learning for an untrained machine learning program P, using multiple input values ​​and known output values ​​empirically obtained from the multiple input values ​​as training data TD. In addition, this method generates multiple trained estimation models M1 to Mn that obtain output values ​​from input values, inputs the same input value a to each of the multiple trained estimation models M1 to Mn generated, and obtains output values ​​X1 to Xn from each estimation model. Then, this method calculates the average value Xm and standard deviation ΔXm of the multiple obtained output values, and evaluates that the smaller the standard deviation ΔXm of the output value, the higher the reliability of the output value for the input value.

[0003] In addition, item response theory (IRT) has been introduced to evaluate machine learning models related to natural language processing, etc. IRT is widely used in educational tests as a method to simultaneously evaluate the ability of examinees and the quality of test questions. When applying IRT to evaluate machine learning models, it is possible to evaluate both the capabilities of the machine learning model and the characteristics of the evaluation data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-56139 A [Non-patent literature]

[0005] [Non-Patent Document 1] Pedro Rodriguez, Phu Mon Htut, John Lalor, Joao Sedoc, "Clustering Examples in Multi-Dataset Benchmarks with Item Response Theory," In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 100-112, Dublin, Ireland, Association for Computational Linguistics, May 2022. [Non-Patent Document 2] Joao Sedoc and Lyle Ungar, "Item Response Theory for Efficient Human Evaluation of Chatbots," In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 21-33, Online, Association for Computational Linguistics, November 2020. Summary of the Invention [Problem to be solved by the invention]

[0006] To evaluate a machine learning model, data with a correct answer label is necessary, but this data with a correct answer label is often difficult to obtain. For example, in the field of biology, experiments are necessary to obtain data with a correct answer label, but there are limitations to this, and it is difficult to prepare a large amount of data with a correct answer label. If the amount of data with a correct answer label available when evaluating a machine learning model is small, the evaluation may be statistically insufficient and the reliability of the evaluation may be low.

[0007] In addition, in order to properly evaluate a machine learning model, it is desirable to evaluate it using data different from the training dataset used to train the machine learning model to be evaluated. However, in the case of a machine learning model developed and published externally, it is possible that publicly available data with correct answers was used to train the machine learning model. In addition, when comparing and evaluating the capabilities of multiple machine learning models, such as comparing a machine learning model developed externally with a machine learning model developed by oneself, it is desirable to evaluate the multiple machine learning models using the same benchmark dataset. However, there are cases where each of the multiple machine learning models is trained with a different training dataset, or where the training dataset is unpublished and unknown. In such a situation, it is difficult to prepare an appropriate benchmark dataset and perform a fair comparative evaluation between multiple machine learning models.

[0008] In one aspect, the disclosed technology aims to comparatively evaluate multiple machine learning models while ensuring statistical reliability and fairness. [Means for solving the problem]

[0009] In one aspect, the disclosed technology trains a plurality of first machine learning models with different capabilities using a plurality of training data with correct answer labels, and generates a plurality of evaluation data whose similarity to the plurality of training data with correct answer labels is equal to or less than a predetermined value and whose correct answer labels are unknown. The disclosed technology also obtains prediction results for the plurality of evaluation data by each of the plurality of first machine learning models and one or more second machine learning models to be evaluated. The disclosed technology then inputs the prediction results into a probability model that indicates the probability that each of the plurality of first machine learning models and the one or more second machine learning models will obtain the prediction results, and optimizes the prediction results. The probability model includes a parameter indicating the capability of each of the plurality of first machine learning models and the one or more second machine learning models, and a parameter indicating the correct answer label of the plurality of evaluation data. The disclosed technology outputs the parameter indicating the capability when the probability model is optimized as an evaluation index indicating the capability of each of the one or more second machine learning models. Effect of the Invention

[0010] One aspect of the present invention is that it has the effect of enabling comparative evaluation of multiple machine learning models while ensuring statistical reliability and fairness. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a functional block diagram of the evaluation device. [Diagram 2] FIG. 1 is a diagram for explaining the relationship between a machine learning model, a problem, a prediction result, and a latent variable indicating an estimated value of a correct label. [Diagram 3] FIG. 2 is a block diagram showing a schematic configuration of a computer functioning as an evaluation device. [Figure 4] 13 is a flowchart illustrating an example of an evaluation process. [Diagram 5] FIG. 13 is a diagram illustrating an example of training data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0013] As shown in FIG. 1, a training dataset including a plurality of training data whose correct labels are known (hereinafter also referred to as "correctly labeled data") and one or more machine learning models to be evaluated (hereinafter referred to as "models to be evaluated") are input to the evaluation device 10. The correctly labeled data is an example of "training data with correct labels" of the disclosed technology. In this embodiment, the task targeted by the machine learning model is a classification problem. That is, the output of the machine learning model is, for example, 0 or 1 in the case of binary classification, and is, for example, multi-value such as 1, 2, . . . , L (L is the number of types of correct labels) in the case of multi-value classification.

[0014] As shown in FIG. 1, the evaluation device 10 functionally includes a training unit 12, a generation unit 14, a prediction unit 16, and an evaluation unit 18.

[0015] The training unit 12 trains multiple machine learning models with different capabilities using the training dataset. The multiple machine learning models trained by the training unit 12 are hereinafter referred to as "self-made models." In addition, when there is no distinction between the evaluation target model and the self-made model, they are also simply referred to as "machine learning models."

[0016] Specifically, the training unit 12 may use the machine learning models acquired at each of multiple different stages in the process from the start of training to convergence as the self-made models. In the training process, the capability of the machine learning model increases as the training progresses. Therefore, the training unit 12 generates multiple self-made models with different capabilities by, for example, saving the machine learning model in a snapshot-like manner for each predetermined number of epochs in the training process.

[0017] The method of generating multiple homemade models with different capabilities is not limited to the above example. The training unit 12 may set the machine learning model trained by changing at least one of the initial value and the hyperparameter, that is, by changing the learning route, as the homemade model. The training unit 12 may set the machine learning model trained by using a part of the correct answer labeled data selected from the training data set for each machine learning model as the homemade model. Furthermore, multiple homemade models with different capabilities may be generated by performing training using a combination of these methods.

[0018] As described later, in this embodiment, since an evaluation index that relatively evaluates a plurality of machine learning models is obtained, the point is that the generated self-made models have variations in performance. In addition, by applying the above-mentioned various techniques to provide diversity to the generated plurality of self-made models, it is possible to robustly estimate the performance of each machine learning model, which will be described later.

[0019] The generation unit 14 generates a plurality of questions whose similarity to the data with a correct answer label is equal to or less than a predetermined value and whose correct answer label is unknown. The questions generated by the generation unit 14 are an example of "assessment data" of the disclosed technology. Specifically, the generation unit 14 generates question candidates by random generation, changing at least a part of the data with a correct answer label, deleting at least a part of the data with a correct answer label, adding information to the data with a correct answer label, or the like. For example, when the question is text data in a natural language or an amino acid sequence, the generation unit 14 generates at least a part of the data with a correct answer label, which is a part of a character string, and when the question is composed of attribute values ​​for multiple attributes, which is an attribute value of some of the attributes.

[0020] The generation unit 14 generates questions by selecting, from among the question candidates, question candidates whose similarity to any of the labeled data included in the training data set is equal to or less than a predetermined value. For example, when the question is text data in a natural language or an amino acid sequence, the generation unit 14 may determine the similarity as the ratio of the number of characters that match between the labeled data and the question. The generation unit 14 may also represent the labeled data and the question as vectors, respectively, and determine the similarity as the inverse of the distance between the vectors. By generating questions whose similarity is equal to or less than a predetermined value, it is possible to generate questions that can be considered independent of the labeled data.

[0021] The prediction unit 16 obtains prediction results by the self-made model and the evaluation target model for each of the multiple questions generated by the generation unit 14. This corresponds to an examinee answering a question in IRT. Specifically, the prediction unit 16 inputs each question to the self-made model and the evaluation target model, and obtains the output of the self-made model and the evaluation target model as a prediction result.

[0022] The evaluation unit 18 optimizes a probabilistic model that includes parameters indicating the capabilities of the machine learning models and parameters indicating the correct label of the question, and indicates the probability that each of the machine learning models will obtain the prediction result obtained by the prediction unit 16. Then, the evaluation unit 18 estimates the parameters indicating the capabilities when the probabilistic models are optimized as an evaluation index indicating the capabilities of the model to be evaluated.

[0023] Here, in this embodiment, Item Response Theory (IRT) is applied to the probability model. Item Response Theory is also called Item Response Theory. IRT is a test theory for measuring the characteristics of subjects, such as cognitive ability, physical ability, technique, knowledge, attitude, personality traits, etc., and the difficulty and discrimination ability of each evaluation item, based on responses to a group of evaluation items. The main feature of this theory is that it attempts to stochastically determine parameters such as an individual's ability value and the difficulty of an item from discrete results such as correct or incorrect answers to evaluation items. For example, the probability p of examinee i answering item (sub-question) j correctly is i,j is modeled using the sigmoid function shown in the following equation (1).

[0024]

number

[0025] θ i is a capability parameter, which is a real value that represents the overall ability of each examinee i to answer correctly. i Unlike the percentage of correct answers or the total score, this is an interval scale estimated based on the characteristics of each question. j is a discrimination parameter, a real-valued value that represents the resolution ability of question j to discriminate the examinee's ability. j is a difficulty parameter, which is a real number that indicates the difficulty of an examinee to correctly answer question j. For example, it may be the ability score of an examinee who has a 50% correct answer rate for each question. j is the chance correct answer rate parameter, which is the probability of getting the correct answer by chance even if you select the answer option randomly in the case of a multiple choice format.

[0026] In the above-mentioned general IRT, the ability of the machine learning model can be evaluated by regarding the examinee as a machine learning model. However, in a probabilistic model to which the general IRT is applied, the correct answer to the question is necessary for parameter optimization. As mentioned above, it is not easy to prepare a large amount of data with correct answer labels. Therefore, in this embodiment, the above-mentioned parameters θ i , a j , b j , and c j At the same time, we estimate a parameter indicating the correct label for the question and a parameter indicating whether the prediction result of each machine learning model is correct or not.

[0027] Specifically, the hypothesis is that predictions (answers) by low-performance machine learning models will be close to random answers. On the other hand, if many high-performance machine learning models output a matching answer, the hypothesis is that the correct answer for the problem is likely to be that matching answer. This hypothesis can be incorporated into IRT to set up a formulated probability model.

[0028] For example, let the total number of machine learning models be n and the total number of problems be m. For problem (item) j, let the prediction result of machine learning model i be x i,j (x i,j = 1, 2, , L), and the latent variables that indicate the estimates of the correct labels are z j (z j = 1, 2, , L). The function that indicates whether the prediction result of machine learning model i for question j is correct or incorrect is expressed as Δ i,j In this case, n × m x i,j The probability P of obtaining is given by the following equation (3).

[0029]

number

[0030] p j (θ i) is a probability model using the IRT theory shown in equation (1). The evaluation unit 18 selects a parameter z j , a j , b j , c j , and θ i For example, the evaluation unit 18 maximizes the probability P by using statistical modeling such as maximum likelihood method, Bayesian estimation, Markov chain Monte Carlo method, etc., and estimates the parameter z j , a j , b j , c j , and θ i Estimate.

[0031] Figure 2 shows the machine learning model i, problem j, and predicted result x. i,j , and the latent variable z j The relationship between x i,j is a fixed value that serves as an input to the probability P. j During the parameter optimization process, dynamically takes on the values ​​“1, 2, . . . , L”, and the final value at the time of learning convergence becomes the estimate of the correct label.

[0032] The evaluation unit 18 estimates a capability parameter θ i As described above, by applying IRT theory to the probability model, the ability parameter θ i The estimated value of is not an evaluation index with upper and lower limits based on fixed data, such as the accuracy rate, but an index that shows the relative difference in performance when comparing machine learning models.

[0033] The evaluation device 10 may be realized by, for example, a computer 40 shown in Fig. 3. The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes an input / output device 45 such as an input device and a display device, and an R / W (Read / Write) device 46 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 47 that is connected to a network such as the Internet. The CPU 41, the GPU 42, the memory 43, the storage device 44, the input / output device 45, the R / W device 46, and the communication I / F 47 are connected to one another via a bus 48.

[0034] The storage device 44 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage device 44 as a storage medium stores an evaluation program 50 for causing the computer 40 to function as the evaluation device 10. The evaluation program 50 has a training process control command 52, a generating process control command 54, a predicting process control command 56, and an evaluation process control command 58.

[0035] The CPU 41 reads out the evaluation program 50 from the storage device 44, expands it in the memory 43, and sequentially executes the control instructions of the evaluation program 50. The CPU 41 operates as the training unit 12 shown in FIG. 1 by executing the training process control instruction 52. The CPU 41 also operates as the generation unit 14 shown in FIG. 1 by executing the generation process control instruction 54. The CPU 41 also operates as the prediction unit 16 shown in FIG. 1 by executing the prediction process control instruction 56. The CPU 41 also operates as the evaluation unit 18 shown in FIG. 1 by executing the evaluation process control instruction 58. As a result, the computer 40 that has executed the evaluation program 50 functions as the evaluation device 10. The CPU 41 that executes the program is hardware. A part of the program may be executed by the GPU 42.

[0036] The functions realized by the evaluation program 50 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0037] Next, the operation of the evaluation device 10 according to this embodiment will be described. When a training dataset and a model to be evaluated are input to the evaluation device 10 and an instruction to evaluate the model to be evaluated is given, the evaluation device 10 executes the evaluation process shown in FIG. 4. The evaluation process is an example of an evaluation method of the disclosed technology. Here, an example of a machine learning model that executes an allergen prediction task that predicts whether a protein to be predicted is recognized by the human immune system and causes an allergic reaction will be described. Since it is often unknown which protein will be an allergen or not, this is a field with high demand for prediction.

[0038] In step S10, the training unit 12 acquires the training dataset and the model to be evaluated that have been input to the evaluation device 10. Here, each piece of data with a correct answer label included in the training dataset is text data of an amino acid sequence as shown in FIG. 5 to which a correct answer label has been added. The correct answer label is a label (e.g., expressed as a value of 0 or 1) indicating whether the protein corresponding to the amino acid sequence is an allergenic protein or a non-allergenic protein. The model to be evaluated is, for example, an existing allergen prediction model such as AllerCatPro (https: / / allercatpro.bii.a-star.edu.sg / ), AllerTOPv2 (https: / / www.ddg-pharmfac.net / AllerTOP / ), etc.

[0039] Next, in step S12, the training unit 12 generates a plurality of (for example, 5 to 100) homemade models with different capabilities using an initial machine learning model prepared in advance by the user and the acquired training dataset.

[0040] Next, in step S14, the generating unit 14 generates a plurality of questions with unknown correct answer labels from the training dataset. For example, the generating unit 14 generates amino acid sequences translated from the genome sequence, transcript RNA sequence, etc. of a biological species that may be an allergen (e.g., wheat) as question candidates. The generating unit 14 also generates amino acid sequences acquired from a protein database as question candidates. Furthermore, the generating unit 14 may generate question candidates by replacing some character strings in these sequences with other amino acids. Then, the generating unit 14 generates questions by selecting, from among the question candidates, question candidates whose similarity to any of the correctly labeled data included in the training dataset is equal to or less than a predetermined value.

[0041] The process of step S12 and the process of step S14 may be executed in any order, or may be executed in parallel.

[0042] Next, in step S16, the prediction unit 16 obtains prediction results (values ​​of 0 or 1) by each of the self-made model and the model to be evaluated for each of the generated multiple questions. Next, in step S18, the evaluation unit 18 indicates the probability that the prediction result by the prediction unit 16 is obtained, and inputs the prediction result into a probability model (for example, the probability P shown in formula (3)) including parameters indicating the ability parameters of each machine learning model and the estimated value of the correct label. Then, the evaluation unit 18 estimates the parameters of the probability model so as to maximize the probability. Next, in step S20, the evaluation unit 18 outputs the ability parameters for the model to be evaluated from among the estimated parameters as the evaluation result, and the evaluation process ends.

[0043] As described above, the evaluation device according to this embodiment trains a plurality of self-made models with different capabilities using a training dataset including a plurality of pieces of data with correct labels. The evaluation device also generates a plurality of questions whose similarity to the data with correct labels is equal to or less than a predetermined value and whose correct labels are unknown. The evaluation device also obtains prediction results for each of the plurality of questions by each of the self-made model and the model to be evaluated. The evaluation device then inputs the prediction results into a probability model that includes parameters indicating the capabilities of each machine learning model and parameters indicating the correct labels of the questions and indicates the probability that a prediction result will be obtained by each machine learning model. The evaluation device outputs the capability parameters obtained when the probability model is optimized as an evaluation index indicating the capability of the model to be evaluated.

[0044] In this way, by generating new problems that can be considered independent of the data with correct labels used to train the self-made model, data that has not been used to train the machine learning model can be used to evaluate the machine learning model whose training data is unknown. This ensures the fairness of the evaluation. Furthermore, by estimating the correct answer labels for the problems along with the capabilities of the machine learning model, a large number of problems can be generated since the correct answers to the problems to be generated do not need to be known. This ensures the statistical reliability of the evaluation.

[0045] Furthermore, in embedded representation learning of knowledge graphs, random patterns may be generated and used as negative examples for training. In this case, even if you try to generate positive examples from the generated data, it is almost impossible. Since there is a very high possibility that negative examples such as the <capital> of <America> being <Beijing> will be generated randomly based on the relationship with words, and there is no procedure for estimating or confirming positive or negative, the randomly generated data can only be regarded as negative examples. In contrast, in this embodiment, there is a procedure for estimating or confirming positive or negative, so both positive and negative examples can be generated.

[0046] In addition, using the prediction result of a machine learning model as the correct label of the data used for evaluating the machine learning model may lead to a decrease in the reliability of the evaluation. In this embodiment, by simultaneously using multiple machine learning models with different capabilities, the error rate of estimating the correct label can be made lower than when using a single machine learning model.

[0047] In addition, in semi-supervised learning, the prediction result of the machine learning model is attached to unlabeled data as a correct answer label and used to train the machine learning model. However, in semi-supervised learning, training data is generated from unlabeled data prepared in advance, and the number of data generated is limited. In this embodiment, there is no limit to the number of data generated, and data can be increased almost infinitely. In addition, this embodiment differs from semi-supervised learning in that the correct answer to the problem generated in this embodiment is estimated at the same time as the machine learning model is evaluated.

[0048] For example, in the biology field, a large amount of data on existing DNA sequences and amino acid sequences is registered in databases. However, it is often costly to obtain information of interest (correct answers) from this data through experiments, and in the case of rare phenomena, correct answers are not given. In this way, in fields where a large amount of input data (questions) can be generated from existing data, the application of the technology disclosed herein is particularly effective.

[0049] In the above embodiment, the evaluation program is stored (installed) in advance in the storage device, but this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0050] Regarding the above embodiment, the following supplementary notes are further disclosed.

[0051] (Appendix 1) Using multiple training data with correct labels, multiple first machine learning models with different capabilities are trained, generating a plurality of evaluation data whose similarity to the plurality of correctly labeled training data is equal to or less than a predetermined value and whose correct labels are unknown; obtaining prediction results for the plurality of evaluation data by each of the plurality of first machine learning models and one or more second machine learning models to be evaluated; a parameter indicating a capability of each of the plurality of first machine learning models and the one or more second machine learning models, and a parameter indicating a correct label of the plurality of evaluation data, the parameter indicating the capability when the prediction results are inputted into a probabilistic model that indicates the probability that each of the plurality of first machine learning models and the one or more second machine learning models will obtain the prediction result and optimized, is output as an evaluation index indicating the capability of each of the one or more second machine learning models. An evaluation program for causing a computer to execute a process including the above.

[0052] (Appendix 2) The probabilistic model is a model for simultaneously estimating parameters indicating the ability and parameters indicating features of each of the plurality of evaluation data based on item response theory, and simultaneously estimating parameters indicating the correct label and parameters indicating whether the prediction result is correct or not.

[0053] (Appendix 3) The evaluation program of Appendix 2, wherein the parameters indicating the characteristics include a parameter indicating a decomposition ability of the evaluation data to distinguish between the capabilities of each of the plurality of first machine learning models and the one or more second machine learning models, a parameter indicating the difficulty of predicting a correct answer for the evaluation data, and a parameter indicating the probability that a correct answer for the evaluation data will be predicted by chance.

[0054] (Appendix 4) The first machine learning model is A machine learning model obtained at each of different stages in the process from the start of training to convergence; A machine learning model trained with different initial values ​​and / or hyperparameters; and At least one machine learning model trained using a portion of training data selected from the plurality of training data with correct answers labels for each machine learning model. An evaluation program according to any one of Supplementary Note 1 to Supplementary Note 3.

[0055] (Appendix 5) The evaluation program according to any one of Supplementary Note 1 to Supplementary Note 4, wherein generating the plurality of evaluation data includes selecting data whose similarity to any of the plurality of labeled training data is a predetermined value or less from data generated by at least one of the following methods: random generation, modifying at least a portion of the labeled training data, deleting at least a portion of the labeled training data, and adding information to the labeled training data.

[0056] (Appendix 6) Using multiple training data with correct labels, multiple first machine learning models with different capabilities are trained, generating a plurality of evaluation data whose similarity to the plurality of correctly labeled training data is equal to or less than a predetermined value and whose correct labels are unknown; obtaining prediction results for the plurality of evaluation data by each of the plurality of first machine learning models and one or more second machine learning models to be evaluated; a parameter indicating a capability of each of the plurality of first machine learning models and the one or more second machine learning models, and a parameter indicating a correct label of the plurality of evaluation data, the parameter indicating the capability when the prediction results are inputted into a probabilistic model that indicates the probability that each of the plurality of first machine learning models and the one or more second machine learning models will obtain the prediction result and optimized, is output as an evaluation index indicating the capability of each of the one or more second machine learning models. The evaluation method includes a computer-implemented process.

[0057] (Appendix 7) The evaluation method according to claim 6, wherein the probabilistic model is a model for simultaneously estimating a parameter indicating the ability and a parameter indicating each of the plurality of evaluation data based on item response theory, and simultaneously estimating a parameter indicating the correct label and a parameter indicating whether the prediction result is correct or not.

[0058] (Appendix 8) The evaluation method of claim 7, wherein the parameters indicating the characteristics include a parameter indicating a decomposition ability of the evaluation data to distinguish between the capabilities of each of the plurality of first machine learning models and the one or more second machine learning models, a parameter indicating the difficulty of predicting a correct answer for the evaluation data, and a parameter indicating the probability that a correct answer for the evaluation data will be predicted by chance.

[0059] (Appendix 9) The first machine learning model is A machine learning model obtained at each of different stages in the process from the start of training to convergence; A machine learning model trained with different initial values ​​and / or hyperparameters; and At least one machine learning model trained using a portion of training data selected from the plurality of training data with correct answers labels for each machine learning model. The evaluation method according to any one of Supplementary Notes 6 to 8.

[0060] (Appendix 10) The evaluation method according to any one of Appendices 6 to 9, wherein generating the plurality of evaluation data includes selecting data, the similarity of which to any of the plurality of labeled training data being a predetermined value or less, from data generated by at least one of the following methods: random generation, modifying at least a portion of the labeled training data, deleting at least a portion of the labeled training data, and adding information to the labeled training data.

[0061] (Appendix 11) A training unit that trains a plurality of first machine learning models having different capabilities using training data with a plurality of correct answer labels; a generation unit that generates a plurality of evaluation data whose similarity to the plurality of correctly labeled training data is equal to or less than a predetermined value and whose correct labels are unknown; a prediction unit that obtains prediction results for the plurality of evaluation data by each of the plurality of first machine learning models and one or more second machine learning models to be evaluated; an evaluation unit that includes a parameter indicating a capability of each of the plurality of first machine learning models and the one or more second machine learning models and a parameter indicating a correct label of the plurality of evaluation data, and outputs the parameters indicating the capability when the prediction results are inputted into a probabilistic model that indicates the probability that each of the plurality of first machine learning models and the one or more second machine learning models will obtain the prediction result and optimized as an evaluation index indicating the capability of each of the one or more second machine learning models; An evaluation device comprising:

[0062] (Appendix 12) The evaluation device described in Appendix 11, wherein the probabilistic model is a model for simultaneously estimating a parameter indicating the ability and a parameter indicating a feature of each of the plurality of evaluation data based on item response theory, and simultaneously estimating a parameter indicating the correct label and a parameter indicating whether the prediction result is correct or not.

[0063] (Appendix 13) The parameters indicating the characteristics include a parameter indicating a decomposition ability of the evaluation data to distinguish between the capabilities of each of the plurality of first machine learning models and the one or more second machine learning models, a parameter indicating a difficulty of predicting a correct answer for the evaluation data, and a parameter indicating a probability that a correct answer for the evaluation data will be predicted by chance.

[0064] (Appendix 14) The first machine learning model is A machine learning model obtained at each of different stages in the process from the start of training to convergence; A machine learning model trained with different initial values ​​and / or hyperparameters; and At least one machine learning model trained using a portion of training data selected from the plurality of training data with correct answers labels for each machine learning model. The evaluation device according to any one of Supplementary Note 11 to Supplementary Note 13.

[0065] (Appendix 15) The evaluation device according to any one of Supplementary Note 11 to Supplementary Note 14, wherein the generation unit selects data whose similarity to any of the plurality of labeled training data is a predetermined value or less from data generated by at least one of the following methods: random generation, changing at least a portion of the labeled training data, deleting at least a portion of the labeled training data, and adding information to the labeled training data.

[0066] (Appendix 16) Using multiple training data with correct labels, multiple first machine learning models with different capabilities are trained, generating a plurality of evaluation data whose similarity to the plurality of correctly labeled training data is equal to or less than a predetermined value and whose correct labels are unknown; obtaining prediction results for the plurality of evaluation data by each of the plurality of first machine learning models and one or more second machine learning models to be evaluated; a parameter indicating a capability of each of the plurality of first machine learning models and the one or more second machine learning models, and a parameter indicating a correct label of the plurality of evaluation data, the parameter indicating the capability when the prediction results are inputted into a probabilistic model that indicates the probability that each of the plurality of first machine learning models and the one or more second machine learning models will obtain the prediction result and optimized, is output as an evaluation index indicating the capability of each of the one or more second machine learning models. A non-transitory storage medium storing an evaluation program for causing a computer to execute a process including the steps of: [Explanation of symbols]

[0067] 10 Evaluation equipment 12 Training Department 14 Generation part 16 Prediction Department 18 Evaluation Section 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 I / O Devices 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 50 Evaluation Program 52 Training process control instructions 54 Generation process control instructions 56 Predictive Process Control Instructions 58 Evaluation Process Control Order

Claims

1. Training a plurality of first machine learning models having different capabilities using training data with a plurality of correct answers labels; generating a plurality of evaluation data whose similarity to the plurality of correctly labeled training data is equal to or less than a predetermined value and whose correct labels are unknown; obtaining prediction results for the plurality of evaluation data by each of the plurality of first machine learning models and one or more second machine learning models to be evaluated; a parameter indicating a capability of each of the plurality of first machine learning models and the one or more second machine learning models, and a parameter indicating a correct label of the plurality of evaluation data, the parameter indicating the capability when the prediction results are inputted into a probabilistic model that indicates a probability that each of the plurality of first machine learning models and the one or more second machine learning models will obtain the prediction result and optimized, is output as an evaluation index indicating the capability of each of the one or more second machine learning models. An evaluation program for causing a computer to execute a process including the above.

2. 2. The evaluation program according to claim 1, wherein the probabilistic model is a model for simultaneously estimating a parameter indicating the ability and a parameter indicating the characteristics of each of the plurality of evaluation data based on item response theory, and simultaneously estimating a parameter indicating the correct label and a parameter indicating whether the prediction result is correct or not.

3. The evaluation program of claim 2, wherein the parameters indicating the characteristics include a parameter indicating a decomposition ability of the evaluation data to distinguish between the capabilities of each of the plurality of first machine learning models and the one or more second machine learning models, a parameter indicating the difficulty of predicting a correct answer for the evaluation data, and a parameter indicating the probability that a correct answer for the evaluation data will be predicted by chance.

4. The first machine learning model is A machine learning model obtained at each of different stages in the process from the start of training to convergence; A machine learning model trained with different initial values ​​and / or hyperparameters; and At least one machine learning model trained using a portion of training data selected from the plurality of training data with correct answer labels for each machine learning model. The evaluation program according to any one of claims 1 to 3.

5. The evaluation program according to any one of claims 1 to 3, wherein generating the plurality of evaluation data includes selecting data whose similarity to any of the plurality of labeled training data is a predetermined value or less from data generated by at least one of the following methods: random generation, modifying at least a portion of the labeled training data, deleting at least a portion of the labeled training data, and adding information to the labeled training data.

6. Training a plurality of first machine learning models having different capabilities using training data with a plurality of correct answers labels; generating a plurality of evaluation data whose similarity to the plurality of correctly labeled training data is equal to or less than a predetermined value and whose correct labels are unknown; obtaining prediction results for the plurality of evaluation data by each of the plurality of first machine learning models and one or more second machine learning models to be evaluated; a parameter indicating a capability of each of the plurality of first machine learning models and the one or more second machine learning models, and a parameter indicating a correct label of the plurality of evaluation data, the parameter indicating the capability when the prediction results are inputted into a probabilistic model that indicates a probability that each of the plurality of first machine learning models and the one or more second machine learning models will obtain the prediction result and optimized, is output as an evaluation index indicating the capability of each of the one or more second machine learning models. The evaluation method includes a computer-implemented process.

7. A training unit that trains a plurality of first machine learning models having different capabilities using training data with a plurality of correct answer labels; a generation unit that generates a plurality of evaluation data whose similarity to the plurality of correctly labeled training data is equal to or less than a predetermined value and whose correct labels are unknown; a prediction unit that obtains prediction results by each of the plurality of first machine learning models and one or more second machine learning models to be evaluated for the plurality of evaluation data; an evaluation unit that includes a parameter indicating a capability of each of the plurality of first machine learning models and the one or more second machine learning models, and a parameter indicating a correct label of the plurality of evaluation data, and outputs the parameters indicating the capabilities when the prediction results are inputted into a probabilistic model that indicates the probability that each of the plurality of first machine learning models and the one or more second machine learning models will obtain the prediction result and optimized, as an evaluation index indicating the capability of each of the one or more second machine learning models; An evaluation device comprising:

Images