Model evaluation device, model evaluation method, program

Abstract

A model evaluation device 100 according to the present disclosure comprises a generation unit 121 that generates a plurality of second machine learning models different from a first machine learning model being evaluated, and an evaluation unit 122 that evaluates the first machine learning model on the basis of a prediction label outputted due to the same data being inputted to the first machine learning model and each of the second machine learning models, thereby making it possible for the model evaluation device 100 to assist users in making decisions.

Description

Technical Field

[0001] The present disclosure relates to a model evaluation apparatus, a model evaluation method, and a program.

Background Art

[0002] There exist techniques for evaluating and managing the performance of machine learning models. Patent Document 1 discloses a technique for labeling data to generate data with pseudo ground truth. A user can evaluate the performance of a machine learning model by means of the technique disclosed in Patent Document 1.

Prior Art Literature

Patent Literature

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] However, obtaining data for evaluating the performance of a machine learning model may take time and cost. For example, when labeled data cannot be obtained until a predetermined time has elapsed, or when labeled data is generated through investigation by experts or the like, obtaining the data takes time and cost. Generating data with pseudo ground truth with the technique of Patent Document 1 takes time and cost.

[0005] Therefore, an object of the present disclosure is to provide a model evaluation apparatus that can solve the above-described problem that time and cost are required when evaluating the performance of a machine learning model.

Means for Solving the Problem

[0006] A model evaluation apparatus that is one aspect of the present disclosure includes: a generation unit that generates a plurality of second machine learning models different from the first machine learning model which is a target of performance evaluation; and An evaluation unit evaluates the first machine learning model based on the predicted labels output by inputting the same data into both the first machine learning model and the second machine learning model. Equipped with, This is the structure it takes.

[0007] Furthermore, the model evaluation method, which is one form of this disclosure, Multiple second machine learning models, different from the first machine learning model that is the subject of performance evaluation, are generated. The first machine learning model is evaluated based on the predicted labels output by inputting the same data into both the first and second machine learning models. This is the structure it takes.

[0008] Furthermore, one form of this disclosure is a program, Multiple second machine learning models, different from the first machine learning model that is the subject of performance evaluation, are generated. The first machine learning model is evaluated based on the predicted labels output by inputting the same data into both the first and second machine learning models. To have the computer perform the process. This is the structure it takes. [Effects of the Invention]

[0009] This disclosure, configured as described above, can reduce the time and cost involved in evaluating the performance of machine learning models. [Brief explanation of the drawing]

[0010] [Figure 1] This block diagram shows the configuration of the model evaluation device in Embodiment 1 of this disclosure. [Figure 2] This figure shows the processing performed by the model evaluation device disclosed in Figure 1. [Figure 3] This figure shows the processing performed by the model evaluation device disclosed in Figure 1. [Figure 4]It is a diagram showing how processing is performed by the model evaluation device disclosed in FIG. 1 [Figure 5] It is a diagram showing how processing is performed by the model evaluation device disclosed in FIG. 1 [Figure 6] It is a diagram showing how processing is performed by the model evaluation device disclosed in FIG. 1 [Figure 7] It is a diagram showing how processing is performed by the model evaluation device disclosed in FIG. 1 [Figure 8] It is a flowchart showing the operation of the model evaluation device disclosed in FIG. 1 [Figure 9] It is a block diagram showing the hardware configuration of the model evaluation device according to Embodiment 2 of the present disclosure [Figure 10] It is a block diagram showing the configuration of the model evaluation device according to Embodiment 2 of the present disclosure DETAILED DESCRIPTION OF EMBODIMENTS

[0011] <Embodiment 1> A first embodiment of the present disclosure will be described with reference to FIGS. 1 to 8. FIG. 1 is a diagram for explaining the configuration of a model evaluation device, and FIGS. 2 to 8 are diagrams for explaining the processing operation of the model evaluation device

[0012] [Configuration] The model evaluation device 10 according to the present embodiment is used to evaluate the prediction performance of an operational model that predicts correct labels from already generated input data. When evaluating prediction performance, if data with correct labels corresponding to input data is used, it may take time and cost to obtain such data. Therefore, the model evaluation device according to the present embodiment performs evaluation using input data to which no correct label is attached

[0013] The model evaluation apparatus 10 according to the present embodiment is constituted by one or a plurality of information processing apparatuses including an arithmetic device and a storage device. As shown in FIG. 1, the model evaluation apparatus 10 includes a check model generation unit 11, a check model selection unit 12, and a performance evaluation unit 13. Each function of the check model generation unit 11, the check model selection unit 12, and the performance evaluation unit 13 can be realized by the arithmetic device executing a program stored in the storage device for realizing each function. The model evaluation apparatus 10 further includes a model storage unit 16, a training data storage unit 17, and an estimation target data storage unit 18. The model storage unit 16, the training data storage unit 17, and the estimation target data storage unit 18 are constituted by storage devices. Each configuration will be described in detail below.

[0014] The model storage unit 16 stores an operation model (also referred to as a "first machine learning model") generated by executing a machine learning algorithm using pre-prepared training data. The operation model is a model used by a predetermined prediction system, and is an object whose prediction performance is to be evaluated. The model is described here. A model is information representing the relationship between explanatory variables and an objective variable. For example, a model is a component for estimating a result of an estimation target by calculating the objective variable based on explanatory variables. A prediction model is generated by inputting training data for which values of an objective variable (also referred to as "labels") have already been obtained and arbitrary parameters, and executing a machine learning algorithm. For example, the prediction model may be represented by a function c that maps an input x to a correct answer y. Note that the prediction model may also be described as a "learning model", an "analysis model", an "AI model", a "trained model", an "inference model", a "prediction formula", or the like. Note that an explanatory variable is a variable used as an input in a prediction model. An explanatory variable may be described as a "feature quantity" or a "feature", or the like. Furthermore, the machine learning algorithm used to generate the model is not particularly limited and any existing learning algorithm is acceptable. For example, the learning algorithm may be a random forest, support vector machine, naive Bayes, neural network, a piecewise linear model using FAB inference (Factorized Asymptotic Bayesian Inference), or a neural network. Furthermore, the piecewise linear model method using FAB inference is, for example, described in U.S. Patent Publication US201. This information is disclosed in the official gazette, No. 4 / 0222741A1, etc.

[0015] The training data storage unit 17 stores training data to which labels have been pre-assigned to the input data as described above. The training data may be used to generate the operational model described above, or it may be used to generate a check model as described later. In addition, the training data may include separate data used to generate a check model, which is different from the training data used to generate the operational model.

[0016] The estimation target data storage unit 18 stores estimation target data for which labels have not been assigned to the input data. In other words, the estimation target data consists only of the input data, and as will be described later, it is the data that is input to the operational model and the check model when evaluating the operational model, and is used to estimate the prediction labels. For example, the estimation target data may be data measured from a prediction system equipped with the operational model, or data suitable for prediction by the operational model or the check model.

[0017] The check model generation unit 11 (generation unit) generates a check model (also called a "second machine learning model") that outputs a label according to the input data by executing a machine learning algorithm using the training data to which labels have been assigned to the input data stored in the training data storage unit 17. At this time, the check model generation unit 11 learns the training data multiple times and generates multiple check models that are different from the operational model stored in the model storage unit 16, and each is different from the others. In particular, the check model generation unit 11 generates multiple check models so that they are diverse, that is, so that the multiple check models are dissimilar to each other. As an example, the check model generation unit 11 can generate multiple check models by changing the seed of the random numbers used in machine learning and learning multiple times. For example, the check model generation unit 11 can generate multiple check models that are dissimilar to each other and diverse by learning by randomly extracting training data with replacement, or by learning by randomly changing hyperparameters. The check model generation unit 11 may generate multiple check models that are different to each other by any method. For example, the checkmodel generation unit 11 may assign pseudo-labels to the estimation target data that does not have labels assigned to it and learn from it, or it may change the weights of the data during the learning process and generate multiple checkmodels.

[0018] Here, Figure 2 shows a conceptual diagram of operational model A (solid black line) and check model B (solid gray line). In Figure 2, the symbol C represents the true model (dotted black line), the circles indicated by the symbol Pt represent positive example labels in the training data, and the squares indicated by the symbol Pf represent negative example labels in the training data. In this example, the diagram shows a conceptual case for classifying binary labels of positive and negative examples. For example, if multiple check models B are generated without considering diversity, as shown in the left figure of Figure 2, it is possible to generate multiple check models B that are densely clustered and similar to each other. On the other hand, as in this embodiment, if multiple check models B are generated by changing the random number seed and performing multiple training iterations, as shown in the right figure of Figure 2, it is possible to generate multiple check models B that are dispersed, dissimilar to each other, and possess diversity.

[0019] The check model selection unit 12 (selection unit) deletes some check models from the multiple check models B generated as described above, and selects a predetermined number (e.g., M) of check models B that is less than the initial number (e.g., T). At this time, the check model selection unit 12 selects check models based on the dissimilarity between each check model B, which is determined by a pre-set criterion for the predicted labels output by the check models. In particular, the check model selection unit 12 selects check models such that the dissimilarity between the predetermined number of check models to be selected becomes larger. That is, as shown in the conceptual diagram of Figure 3 from left to right, the check model selection unit 12 deletes check models B that are similar to each other from a state where multiple check models B are densely clustered, and selects a predetermined number of check models such that the distance between the remaining predetermined number of check models B becomes larger. The specific processing method for selecting check models will be explained below with reference to Figures 4 to 7.

[0020] First, we have multiple check models (g1, g2, ..., g T We will represent this diversity as shown in equation 1, and as an indicator to evaluate this diversity, we will set a diversity criterion shown in equation 1.

number

number

number

[0021] Here, each check model is represented as a node (code V), and the dissimilarity between each check model is represented as the weight (length) of the edge (code E) connecting the nodes, as shown in graph G in Figure 4. Here, we show the case where there are 7 nodes V corresponding to check models. In graph G, the longer the length of the edge E between nodes V, that is, the greater the distance between connected nodes V, the higher the dissimilarity between the check models corresponding to each node V. In other words, the shorter the length of the edge E between nodes V, that is, the closer the distance between connected nodes V, the more similar the check models corresponding to each node V are. The left side of Figure 4 shows the overall diagram of the undirected fully weighted graph G, and the minimum spanning tree, which is the spanning tree composed of subtrees containing all nodes V in graph G and has the minimum length (weight) of edge E, is shown by the black line edge Em in the right side of Figure 4.

[0022] Furthermore, in equation 1 above, MST(G) is defined as a function that returns the edges included in the minimum spanning tree of graph G shown in Figure 4 (right). Therefore, the diversity criterion shown in equation 1 above is the sum of the edge weights (lengths) of the minimum spanning tree of graph G, and the larger this value, the greater the dissimilarity between each checkmodel corresponding to each node. Using this, the checkmodel selection unit 12 selects a predetermined number of checkmodels by deleting some of them so that the diversity criterion value of graph G is larger for the selected number of checkmodels. Here, we will explain an example in which four checkmodels are selected from the seven checkmodels corresponding to nodes B in graph G.

[0023] First, the check model selection unit 12 extracts two check models whose dissimilarity is small compared to others. For example, as shown within the dotted line in Figure 5 (left), it extracts two nodes Vi and Vj connected to the shortest edge Em in graph G. Then, by deleting one of these two nodes Vi and Vj, it selects the check model corresponding to the remaining node. At this time, since the two nodes with short edges Em in graph G have a high degree of similarity, deleting one of them can increase the diversity of the remaining check models. For example, as shown along the dotted line in Figure 5 (right), the check model selection unit 12 examines the lengths (weights) of edges Eki and Ekj from another node Vk for the two extracted nodes Vi and Vj, for example, from the node Vk closest to these two nodes Vi and Vj, deletes the node connected to the shorter edge, and selects the remaining node. In the example in Figure 5 (right), of the two extracted nodes Vi and Vj, node Vj is deleted and node Vi is selected. Then, as shown in Figure 6 (left), node Vj is deleted and node Vi is selected in graph G, and the edge Em is reconstructed to form a minimum spanning tree.

[0024] As described above, the check model selection unit 12 repeats the process of extracting two nodes, deleting one of them, and selecting the other, until a predetermined number of nodes are selected. As a result, for example, as shown in the right diagram of Figure 6, four nodes are selected from the seven nodes V in graph G, and a minimum spanning tree is generated using these four nodes. At this time, the value of the diversity criterion for the minimum spanning tree formed by the four nodes can be improved, for example, by approximately maximizing it. The check model selection unit 12 then selects four check models corresponding to the four selected nodes and stores these selected check models in the model storage unit 16.

[0025] The method by which the check model selection unit 12 selects check models is not limited to the method described above. The check model selection unit 12 may select a predetermined number of check models from those generated by the check model generation unit 11 using a different diversity index. For example, the Q statistic (similarity index; a higher Q indicates lower diversity) may be used as the diversity index. As an example, the check model selection unit 12 may evaluate the similarity of two check models based on the degree of agreement in the correct and incorrect answer trends, and select a predetermined number of check models with diversity by removing one of the similar check models.

[0026] The performance evaluation unit 13 (evaluation unit) reads the operational model and each selected check model stored in the model storage unit 16, and the target data to be estimated stored in the target data storage unit 18, and uses these to evaluate the predictive performance of the operational model. Specifically, the performance evaluation unit 13 inputs the target data to be estimated, which has no labels assigned to it, into the operational model and each selected check model, and obtains the predictive labels that are the outputs of each. Then, it evaluates the predictive performance of the operational model based on the degree of agreement between the predictive labels output from the operational model and the predictive labels output from each check model. At this time, the performance evaluation unit 13 treats the predictive labels of each check model as ground truth labels, and evaluates the predictive performance of the operational model based on the degree of agreement between the predictive labels output from the operational model and these ground truth labels. For example, the performance evaluation unit 13 evaluates the predictive performance of the operational model as higher the number of check models whose predictive labels match those output from the operational model.

[0027] Here, we will explain the specific performance evaluation method used by the performance evaluation unit 13. For example, let's assume that the performance index is calculated using the following equation 4. Note that N is the number of data points to be estimated as input.

number

number

number

[0028] The performance indicator may represent the accuracy with which the predicted labels of the operational model match those of the check model, as shown in Equation 7 below, or it may be precision, recall, or F1 score, as shown in Equation 8 below. However, the performance indicator is not limited to these values, and other values ​​may be used.

number

number

[0029] Here, the relationship between the operational model and each check model when evaluating the operational model described above is illustrated and explained in Figure 7. The left side of Figure 7 shows a conceptual diagram of the similarity relationship between operational model A and each check model B, indicating that the closer the distance, the more similar the prediction behavior. The left side of Figure 7 also shows the training data with positive example labels assigned by the code Pt, and the training data with negative example labels assigned by the code Pf, illustrating a conceptual diagram for classifying binary labels of positive and negative examples. The right side of Figure 7 further shows the data to be estimated, assigned by the code P. As shown in this figure, the data to be estimated P does not necessarily result in the same prediction result, i.e., the predicted label, for operational model A and all of the check models B. For example, the check model assigned by the code B will have the same prediction result as operational model A, but the check model assigned by the code B' will have a different prediction result than operational model A. In this case, the data to be estimated P that produces different output results for the operational model and the check model is considered to be data that is difficult to predict in the first place, and therefore the prediction in the operational model is considered to be wrong.

[0030] Given the situation described above, if multiple check models are clustered together in a conceptual diagram like the one shown in Figure 7, resulting in similar predictive behavior, it can lead to biased and inappropriate evaluation of the operational model. To address this problem, in this embodiment, as described above, multiple check models are generated in a diversified manner and then selected, thereby suppressing biased and inappropriate evaluation of the operational model. As a result, the operational model can be evaluated using readily available, unlabeled estimation target data, enabling rapid, low-cost, and appropriate evaluation of the operational model.

[0031] [Operation] Next, the operation of the model evaluation device 10 described above will be explained with reference to the flowchart in Figure 8. It should be assumed that the model evaluation device 10 stores the operational model to be evaluated, the training data used to generate the operational model, and the estimation target data used to evaluate the operational model.

[0032] First, the model evaluation device 10 generates multiple checkmodels by executing a machine learning algorithm using training data to which correct labels have been assigned (step S1). At this time, the model evaluation device 10 generates multiple checkmodels that are dissimilar and diverse to one another by randomly extracting training data with replacement and learning, or by randomly changing hyperparameters and learning.

[0033] Next, the model evaluation device 10 selects a predetermined number of check models from the multiple generated check models (step S2). At this time, the model evaluation device 10 selects the check models based on the dissimilarity between each check model, which is determined by a pre-set criterion for the predicted labels output by the check models. In particular, the model evaluation device 10 selects the check models such that the dissimilarity between the predetermined number of check models to be selected is greater. As an example, as explained with reference to Figures 4 to 6, the model evaluation device 10 uses a minimum spanning tree of graph G, where each check model is a node (code V) and the dissimilarity between each check model is the weight of the edge (code E) connecting the nodes, to select the check model corresponding to the node.

[0034] Subsequently, the model evaluation device 10 inputs unlabeled estimation target data to the operational model and each selected check model, and obtains the prediction labels which are the outputs of each (step S3). Then, the model evaluation device 10 evaluates the prediction performance of the operational model based on the degree of agreement between the prediction labels output from the operational model and the prediction labels output from each check model (step S4). For example, the model evaluation device 10 treats the prediction labels of each check model as ground truth labels and evaluates the prediction accuracy of the operational model based on the degree of agreement between the prediction labels output from the operational model and these ground truth labels.

[0035] As described above, the model evaluation device 10 in this embodiment generates and selects from a variety of check models, and uses these check models to evaluate the predictive performance of the operational model. Therefore, it is possible to suppress inappropriate evaluations of the operational model, such as bias due to a high density of similar check models. As a result, the operational model can be evaluated using readily available, unlabeled estimation target data, enabling rapid, low-cost, and appropriate evaluation of the operational model.

[0036] <Examples of application> Here, as an example of the application of the present disclosure described above, we will explain an example of its application in the medical and healthcare field. In this example, the operational model (first machine learning model) is a model that classifies chest X-ray images as either healthy (positive case) or diseased (negative case), and its predictive performance is evaluated using the model evaluation device 10 described above. By applying such an operational model to the present disclosure, the operational model can be evaluated using unlabeled chest X-ray images, and the operational model can be evaluated quickly, at low cost, and appropriately. Furthermore, by using the model evaluation device 10 of the present disclosure, it is possible to effectively support physician decision-making.

[0037] <Embodiment 2> Next, a second embodiment of the present disclosure will be described with reference to Figures 9 to 10. Figures 9 to 10 are block diagrams showing the configuration of the model evaluation apparatus in Embodiment 2. In this embodiment, the configuration of the model evaluation apparatus described in the above-described embodiment is shown in outline.

[0038] First, with reference to Figure 9, the hardware configuration of the model evaluation device 100 in this embodiment will be described. The model evaluation device 100 is composed of a general information processing device, and as an example, it is equipped with the following hardware configuration. ·CPU(Central Processing Unit)101(Arithmetic unit) ROM (Read Only Memory) 102 (Storage Device) • RAM (Random Access Memory) 103 (Storage Device) • Program group 104 loaded into RAM 103 • Storage device 105 for storing the program group 104 • Drive device 106 for reading and writing to external storage medium 110 of the information processing device. • Communication interface 107 connecting to a communication network 111 outside the information processing device. • Input / output interface 108 for data input and output. • Bus 109 connecting each component

[0039] Figure 9 shows an example of the hardware configuration of the information processing device, which is the model evaluation device 100, and the hardware configuration of the information processing device is not limited to the case described above. For example, the information processing device may consist of only a part of the configuration described above, such as not having the drive device 106. In addition, the information processing device may use a GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof instead of the CPU described above.

[0040] The model evaluation device 100 can be equipped with the generation unit 121 and evaluation unit 122 shown in Figure 10 by having the CPU 101 acquire the program group 104 and execute it. The program group 104 is, for example, stored in advance in a storage device 105 or ROM 102, and the CPU 101 loads it into RAM 103 and executes it as needed. The program group 104 may also be supplied to the CPU 101 via a communication network 111, or it may be stored in advance in a storage medium 110, and the drive device 106 reads the program and supplies it to the CPU 101. However, the generation unit 121 and evaluation unit 122 described above may be constructed with dedicated electronic circuits to realize such means.

[0041] The generation unit 121 generates multiple second machine learning models that are different from the first machine learning model that is the subject of performance evaluation. Furthermore, the generation unit 121 may select a predetermined number from the multiple second machine learning models that have been generated, based on the dissimilarity between each second machine learning model, which is determined by a set of predefined criteria for the prediction labels output by the second machine learning models.

[0042] The evaluation unit 122 evaluates the first machine learning model based on the predicted labels output by inputting the same data into both the first machine learning model and the second machine learning model.

[0043] As described above, this disclosure enables the generation of multiple diversified second machine learning models, and the evaluation of the predictive performance of the first machine learning model using these diversified second machine learning check models, thereby enabling rapid, low-cost, and appropriate evaluation of the first machine learning model.

[0044] The programs described above can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0045] Although the present disclosure has been described above with reference to the embodiments described above, the present disclosure is not limited to the embodiments described above. Various modifications to the configuration and details of the present disclosure can be made that can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, at least one of the functions of the generation unit 121 and evaluation unit 122 described above may be performed on an information processing device installed and connected at any location on the network, that is, it may be performed using so-called cloud computing.

[0046] <Note> Some or all of the above embodiments may also be described as follows. The following outlines the configuration of the model evaluation apparatus, model evaluation method, and program in this disclosure. However, this disclosure is not limited to the following configuration. (Note 1) A generation unit that generates multiple second machine learning models, which are different from the first machine learning model that is the subject of performance evaluation, An evaluation unit evaluates the first machine learning model based on the predicted labels output by inputting the same data into both the first machine learning model and the second machine learning model. A model evaluation device equipped with the following features. (Note 2) The model evaluation device described in Appendix 1, The system includes a selection unit that further selects a predetermined number of the generated second machine learning models based on the degree of dissimilarity between each of the second machine learning models, which is determined by a predetermined criterion for the predicted labels output by the second machine learning model. The evaluation unit evaluates the first machine learning operation model based on the prediction labels output by inputting the same data to both the first machine learning and the selected second machine learning model. Model evaluation device. (Note 3) The model evaluation device described in Appendix 2, The selection unit selects the second machine learning models such that the dissimilarity among a predetermined number of the second machine learning models becomes larger. Model evaluation device. (Note 4) The model evaluation device described in Appendix 2, The selection unit extracts two second machine learning models from the generated plurality of second machine learning models in which the dissimilarity between the second machine learning models is smaller compared to the others, and selects one of the two extracted second machine learning models to select the second machine learning model. Model evaluation device. (Note 5) The model evaluation device described in Appendix 4, The selection unit selects the one of the two extracted second machine learning models that has a larger dissimilarity with the other second machine learning model. Model evaluation device. (Note 6) The model evaluation device described in Appendix 2, The selection unit further selects from the generated second machine learning models based on a minimum spanning tree in a graph where each generated second machine learning model is a node, and the dissimilarity between each second machine learning model is represented as the weight of the edge connecting the nodes. Model evaluation device. (Note 7) The model evaluation device described in Appendix 1, The evaluation unit uses the predicted label output from the second machine learning model as the ground truth label and evaluates the first machine learning model based on the degree of agreement between the ground truth label and the predicted label output from the first machine learning model. Model evaluation device. (Note 8) Multiple second machine learning models, different from the first machine learning model that is the subject of performance evaluation, are generated. The first machine learning model is evaluated based on the predicted labels output by inputting the same data into both the first and second machine learning models. Model evaluation methods. (Note 9) The model evaluation method described in Appendix 8, Based on the dissimilarity between each of the second machine learning models, which is determined by a set of criteria for the predicted labels output by the second machine learning model, a predetermined number of the generated second machine learning models are further selected. The first machine learning model is evaluated based on the predicted labels output by inputting the same data into both the first machine learning model and the selected second machine learning model. Model evaluation methods. (Note 10) Multiple second machine learning models, different from the first machine learning model that is the subject of performance evaluation, are generated. The first machine learning model is evaluated based on the predicted labels output by inputting the same data into both the first and second machine learning models. A computer-readable storage medium that stores a program that causes a computer to execute a process. [Explanation of Symbols]

[0047] 10 Model evaluation device 11. Check Model Generation Unit 12. Check Model Selection Section 13 Performance Evaluation Department 16 Model Memory Unit 17 Training data storage unit 18 Estimated target data storage unit 100 Model Evaluation Devices 101 CPU 102 ROM 103 RAM 104 Program Groups 105 Storage device 106 Drive unit 107 Communication Interface 108 Input / Output Interfaces 109 Bus 110 Storage medium 111 Communication Network 121 Generation part 122 Evaluation Department

Claims

1. A generation unit that generates multiple second machine learning models, which are different from the first machine learning model that is the subject of performance evaluation, A selection unit further selects a predetermined number of the generated second machine learning models based on the dissimilarity between each of the second machine learning models, based on a predetermined criterion for the predicted labels output by the second machine learning model. An evaluation unit evaluates the first machine learning model based on the prediction labels output by inputting the same data into both the first machine learning model and the selected second machine learning model. A model evaluation device equipped with the following features.

2. A model evaluation apparatus according to claim 1, The selection unit selects the second machine learning models such that the dissimilarity among a predetermined number of the second machine learning models becomes larger. Model evaluation device.

3. A model evaluation apparatus according to claim 1, The selection unit extracts two second machine learning models from the generated plurality of second machine learning models in which the dissimilarity between the second machine learning models is smaller compared to the others, and selects one of the two extracted second machine learning models to select the second machine learning model. Model evaluation device.

4. A model evaluation apparatus according to claim 3, The selection unit selects the one of the two extracted second machine learning models that has a larger dissimilarity with the other second machine learning model. Model evaluation device.

5. A model evaluation apparatus according to claim 1, The selection unit further selects from the generated second machine learning models based on a minimum spanning tree in a graph where each generated second machine learning model is a node, and the dissimilarity between each second machine learning model is represented as the weight of the edge connecting the nodes. Model evaluation device.

6. A model evaluation apparatus according to claim 1, The evaluation unit uses the predicted label output from the second machine learning model as the ground truth label and evaluates the first machine learning model based on the degree of agreement between the ground truth label and the predicted label output from the first machine learning model. Model evaluation device.

7. Multiple second machine learning models, different from the first machine learning model that is the subject of performance evaluation, are generated. Based on the dissimilarity between each of the second machine learning models, which is determined by a set of criteria for the predicted labels output by the second machine learning model, a predetermined number of the generated second machine learning models are further selected. The first machine learning model is evaluated based on the predicted labels output by inputting the same data into both the first machine learning model and the selected second machine learning model. Model evaluation methods.

8. Multiple second machine learning models, different from the first machine learning model that is the subject of performance evaluation, are generated. Based on the dissimilarity between each of the second machine learning models, which is determined by a set of criteria for the predicted labels output by the second machine learning model, a predetermined number of the generated second machine learning models are further selected. The first machine learning model is evaluated based on the predicted labels output by inputting the same data into both the first machine learning model and the selected second machine learning model. A program that instructs a computer to perform a process.

Images