Model evaluation device, model evaluation method, program

Abstract

A model evaluation device 100 according to the present disclosure comprises: a learning unit 121 that, on the basis of a prediction output from a first learning model with respect to input of training data, trains a second learning model that outputs the probability of the prediction with respect to the input of the training data; and an evaluation unit 122 that evaluates the first learning model on the basis of the prediction output from the first learning model with respect to the input of prescribed input data and the probability of the prediction output from the second learning model with respect to the input of the prescribed input data.

Description

[Technical Field]

[0001] The present disclosure relates to a model evaluation apparatus, a model evaluation method, and a program. [Background Art]

[0002] Based on known explanatory variable data and objective variable data, a prediction model for predicting objective variable data from new explanatory variable data is created, and the created prediction model is put into operation. In this case, it is necessary to manage the performance of the prediction model, and the prediction performance of the prediction model is also evaluated. When evaluating the prediction performance of a prediction model, explanatory variable data serving as new input data and objective variable data serving as corresponding correct answer data are required. [Prior Art Document] [Patent Document]

[0003] [Patent Document 1] International Publication No. 2020 / 225923 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, depending on the situation in which the prediction model is operated, it may take time and cost to obtain correct answer data corresponding to new input data to be predicted by the prediction model. For example, when correct answer data cannot be obtained until a predetermined time has elapsed, or when correct answer data is generated through research by experts or the like, it takes time and cost to obtain the data. Further, Patent Document 1 describes that data without correct answers is labeled to generate pseudo-correct answer data, but it takes time and cost to generate the pseudo-correct answer data.

[0005] Thus, when evaluating a predictive model, if we use ground truth data corresponding to the new input data to be predicted, the acquisition of such ground truth data is time-consuming and costly, which creates a problem in that the predictive model cannot be evaluated quickly and at low cost.

[0006] Therefore, the purpose of this disclosure is to provide a model evaluation device that can solve the above-mentioned problem of not being able to evaluate the quality of a predictive model quickly and at low cost. [Means for solving the problem]

[0007] A model evaluation device, which is one form of this disclosure, A learning unit that learns a second learning model that outputs the probability of prediction for the same training data input, based on the prediction output from the first learning model for the same training data input. An evaluation unit evaluates the first learning model based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for a given input of predetermined input data. Equipped with, This is the structure it takes.

[0008] Furthermore, the model evaluation method, which is one form of this disclosure, Based on the predictions output by the first learning model for the input training data, a second learning model is trained to output the probability of prediction for the same input training data. The first learning model is evaluated based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for the given input of predetermined input data. This is the structure it takes.

[0009] Furthermore, one form of this disclosure is a program, Based on the predictions output by the first learning model for the input training data, a second learning model is trained to output the probability of prediction for the same input training data. The first learning model is evaluated based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for the given input of predetermined input data. To have the computer perform the process. This is the structure it takes. [Effects of the Invention]

[0010] This disclosure, configured as described above, enables rapid and low-cost evaluation of the quality of predictive models. [Brief explanation of the drawing]

[0011] [Figure 1] This block diagram shows the configuration of the model evaluation device in Embodiment 1 of this disclosure. [Figure 2] This figure shows the processing performed by the model evaluation device disclosed in Figure 1. [Figure 3] This figure shows the processing performed by the model evaluation device disclosed in Figure 1. [Figure 4] This figure shows the processing performed by the model evaluation device disclosed in Figure 1. [Figure 5] This figure shows the processing performed by the model evaluation device disclosed in Figure 1. [Figure 6] Figure 1 is a flowchart showing the operation of the model evaluation device. [Figure 7] This block diagram shows the hardware configuration of the model evaluation device in Embodiment 2 of this disclosure. [Figure 8] This block diagram shows the configuration of the model evaluation device in Embodiment 2 of this disclosure. [Modes for carrying out the invention]

[0012] <Embodiment 1> A first embodiment of the present disclosure will be described with reference to FIGS. 1 to 6. FIG. 1 is a diagram for explaining the configuration of a model evaluation apparatus, and FIGS. 2 to 6 are diagrams for explaining the processing operation of the model evaluation apparatus.

[0013] [Configuration] The model evaluation apparatus 10 according to the present embodiment is used to evaluate the prediction performance of an operational model (first learning model) that predicts a correct label from input data. Here, if data with a correct label corresponding to input data is used when evaluating the prediction performance of an operational model, it may take time and cost to obtain such data. In contrast, the model evaluation apparatus according to the present embodiment can perform evaluation even using input data to which no correct label is assigned, as will be described later.

[0014] The model evaluation apparatus 10 according to the present embodiment is configured by one or a plurality of information processing apparatuses including an arithmetic device and a storage device. As shown in FIG. 1, the model evaluation apparatus 10 includes a check model generation unit 11 and a performance evaluation unit 12. Each function of the check model generation unit 11 and the performance evaluation unit 12 can be realized by the arithmetic device executing a program stored in the storage device for realizing each function. The model evaluation apparatus 10 further includes a model storage unit 16, a training data storage unit 17, and an estimation target data storage unit 18. The model storage unit 16, the training data storage unit 17, and the estimation target data storage unit 18 are configured by storage devices. Each configuration will be described in detail below.

[0015] The model storage unit 16 stores an operation model (first learning model) generated by machine learning on training data prepared in advance. Here, the training data is data obtained by assigning a correct label to be predicted from the input data to the input data. Therefore, the operation model generated by performing machine learning on such training data is trained to output a correct label to be predicted from the input data in accordance with the input of the input data. In this case, the input data of the training data is, for example, data representing features such as an image of a target for which a correct label is to be predicted, and the target's color, weight, shape, etc., and the correct label of the training data is a specific class among a plurality of classes related to a preset target. For example, in the present embodiment, when vegetables are targeted as will be described later, examples of the input data include image data of vegetables. In this case, a plurality of vegetable classes (types) are set, the correct data is one specific class (type) among them, and examples include cabbage, carrot, bell pepper, and tomato.

[0016] The operation model is installed and operated in a predetermined prediction system, or may be operated in the future, and when estimation target data is input, the operation model outputs a prediction label predicted from the data. Therefore, the operation model generated by performing machine learning on the above-mentioned training data, for example, outputs any one of cabbage, carrot, bell pepper, and tomato among a plurality of preset vegetable classes as a prediction label (specific class) in accordance with an input of image data of a vegetable that is input data. Furthermore, as will be described later, the operation model is a target for which prediction performance is to be evaluated.

[0017] It should be noted that the operational model in this embodiment is not necessarily limited to predicting the classification (type) of vegetables. In other words, the input data entered into the operational model may be data on any subject, and the operational model may predict any classification corresponding to the subject. For example, the operational model may predict a person's health status, and as one example, it may take a person's current condition as input data and output a prediction of the person's health status in the future.

[0018] The training data storage unit 17 stores training data to which the input data has been pre-assigned correct labels, as described above. The training data may be used to generate the operational model described above, or it may be used to generate a check model, as will be described later. In addition, the training data may include separate data used to generate a check model, which is different from the training data used to generate the operational model.

[0019] The estimation target data storage unit 18 stores estimation target data for which no correct labels have been assigned to the input data. In other words, the estimation target data consists only of the input data, and as will be described later, it is input to the operational model and the check model when evaluating the operational model, and is used to estimate the predicted labels. For example, the estimation target data may be image data of the appearance of a vegetable whose classification to be estimated is unknown, or data representing the characteristics of the vegetable whose classification is unknown, such as color, weight, and shape. However, the estimation target data is not necessarily limited to vegetable data, and may be any data with content appropriate to the target of estimation.

[0020] The check model generation unit 11 (learning unit) generates a check model (second learning model) that outputs the probability of each label, including the correct label, in response to the input data, by performing machine learning using the training data to which the correct labels have been assigned to the input data stored in the training data storage unit 17. For example, in this embodiment, as described above, each classification of vegetables is set as each label, and the check model generation unit 11 learns the check model to output the predicted probability for each of the multiple classifications of vegetables for the input data. As a result, the generated check model outputs the probability that each classification of vegetable (for example, cabbage, carrot, bell pepper, tomato) corresponding to the input data, such as image data of vegetables, is applicable.

[0021] In this embodiment, the check model generation unit 11, when generating a check model using training data, also inputs the same training data to the operational model. The check model generation unit 11 then learns to generate a check model using the training data based on the predicted labels output from the operational model. Specifically, the check model generation unit 11 first inputs the training data to be used when generating the check model into the operational model and identifies the predicted labels (specific classifications) output from the operational model at that time. Then, the check model generation unit 11 learns the check model by changing the learning weights for each label in the training data according to the predicted labels output from the operational model. In particular, the check model generation unit 11 learns the check model by making the learning weight of the labels in the training data corresponding to the predicted labels output from the operational model higher than the weights for other labels.

[0022] Here, an example of the check model learning process by the check model generation unit 11 will be explained with reference to Figures 3 to 5. In this example, as mentioned above, the target for classification prediction is vegetables, and a check model is generated that outputs the probability of each classification (each type) of vegetable by inputting image data of vegetables as input data.

[0023] First, as shown in Figure 2, let's assume that the predicted label for the vegetable classification output when the training data input data T1 is input to the operational model is "tomato". In this case, the check model generation unit 11 learns the check model by setting the weights for learning the probability that the input data T1 predicts the vegetable classification "tomato" higher than the weights for learning the other classifications "cabbage, carrot, bell pepper". Accordingly, the check model generation unit 11 learns the check model by setting the weights for learning the probability that the input data T1 predicts the vegetable classification "cabbage, carrot, bell pepper" lower than the weights for learning the other classification "tomato". As a result, the check model generation unit 11 learns the check model in such a way that the accuracy of predicting the probability that the training data input data T1 predicts the vegetable classification "tomato" improves.

[0024] More specifically, the check model generation unit 11 learns the check model using the loss function shown in equation 1. In this case, the operational model f and the check model h are represented by equation 2, and the loss is represented by equations 3 and 4, with an example of the loss shown in equation 5. D represents the training data, and h(f(x)|x) represents the probability of the check model h predicting the label f(x) for the input data x. Equation 6 illustrates the loss function in a learning example different from this embodiment.

number

number

number

number

number

number

[0025] Here, equation 3 represents the loss of prediction probability by the check model for the predicted label of the operational model f, and takes a minimum value of 1 at p=1. Also, equation 4 represents the loss of prediction probability by the check model for labels different from the predicted label of the operational model f, and takes a minimum value of 0 at p=0.

[0026] Furthermore, the loss function in this embodiment, shown in equation 1, has a term w added to that of equation 6. Here, w1 represents the weight of the learning of the prediction probability in the check model for the prediction label of the operational model f, and w0 represents the weight of the learning of the prediction probability in the check model for labels other than the prediction label of the operational model f. In this case, for example, by setting "w1=1, w0=0", the check model will be learned only for the prediction label of the operational model. To elaborate further, the loss function in this case is shown in equation 7, and only h(f(x)|x) appears in the loss function, meaning that only h(f(x)|x) is learned, and labels other than f(x) are ignored during learning. For this reason, in the example in Figure 2, only the prediction probability of f(x) = tomato is learned, and the prediction probabilities of other classifications (cabbage, carrot, bell pepper) are learned randomly.

number

[0027] As another example, by setting "w1=1-α, w0=α" and, for example, "α=0.1", the check model will be trained primarily on the predicted labels of the operational model, similar to the above example. In particular, it is desirable to set "w1>w0" so that the loss weight corresponding to the training data label corresponding to the predicted label output from the operational model is higher than the loss weight for other labels. In this way, the check model will be trained intensively on the predicted labels of the operational model, while training will also be performed on other labels.

[0028] The accuracy of the check model when it is generated by the check model generation unit 11 as described above will be explained with reference to Figures 3 and 4. First, the training data column in Figures 3 and 4 shows the true probability that the training data is each vegetable classification, i.e., each label. In the example in Figure 3, when the operational model predicts the label "tomato" for the input training data T2, the learning weight for the label "tomato" is set high for the input training data T2, and the check model is trained accordingly. Therefore, in this embodiment, the accuracy of the probability of predicting each label by the check model can be improved. On the other hand, when the check model is trained using the loss function by equation 6 described above, which is different from this embodiment, learning is performed equally for the predicted label by the operational model and for all other labels, so the accuracy of the probability of predicting each label by the check model may be low.

[0029] Furthermore, the example in Figure 4 shows the case where the predicted label for the training data input T2 in the operational model is "carrot". In this case, the learning weight for the label "carrot" is set high for the training data input T2, and the check model is trained accordingly. Therefore, in this embodiment, it is possible to improve the accuracy of the probability of predicting each label by the check model. On the other hand, unlike this embodiment, if the check model is trained using the loss function by equation 6 described above, learning is performed equally for the predicted label by the operational model and for all other labels, which may result in a lower accuracy of the probability of predicting each label by the check model.

[0030] The check model generation unit 11 then stores the check model it has generated through learning, as described above, in the model storage unit 16. The check model generation unit 11 may generate one check model or multiple check models.

[0031] The performance evaluation unit 12 (evaluation unit) reads the operational model and check model stored in the model storage unit 16 and the target data to be estimated stored in the target data storage unit 18, and uses these to evaluate the predictive performance of the operational model. Specifically, the performance evaluation unit 12 inputs the target data to be estimated, for which no correct labels have been assigned, into the operational model and the check model, respectively, obtains the predicted labels output from the operational model, and obtains the predicted probability of each label output from the check model. Then, it evaluates the operational model based on the predicted probabilities of the labels output from the check model that correspond to the predicted labels output from the operational model. For example, it calculates statistical values ​​such as the average value of the predicted probabilities of the labels output from the check label that correspond to the predicted labels output from the operational model, and uses this as the accuracy estimate value, which is a value used to evaluate the operational model. In this case, the higher the accuracy evaluation value, the higher the predictive performance of the operational model is evaluated to be.

[0032] Here, an example of how the performance evaluation unit 12 calculates evaluation values ​​will be explained with reference to Figure 5. In Figure 5, the example of vegetable classification described above is represented as classes 1, 2, 3, and 4. In this case, for input "x1", the performance evaluation unit 12 adopts the prediction probability of class 3 of "60%" from the check model, since the operational model outputs the predicted label "class 3". Then, the performance evaluation unit 12 calculates multiple estimation target data (x1~x n The same process is performed for the selected multiple prediction probabilities, and statistical values ​​of these probabilities are calculated as accuracy estimates. At this time, the performance evaluation unit 12 may output the calculated accuracy evaluation values ​​to the display of an information processing device operated by an evaluator who evaluates the operational model, as shown in Figure 5.

[0033] [Operation] Next, the operation of the model evaluation device 10 described above will be explained with reference to the flowchart in Figure 6. It should be assumed that the model evaluation device 10 stores the operational model to be evaluated, the training data used to generate the operational model, and the estimation target data used to evaluate the operational model.

[0034] First, the model evaluation device 10 generates a check model by machine learning using training data to which correct labels have been assigned (step S1). At this time, when the model evaluation device 10 generates a check model using the training data, it also inputs this training data into the operational model and generates a check model using the training data based on the predicted labels output from the operational model. Specifically, the model evaluation device 10 learns the check model by changing the learning weights for each label in the training data according to the predicted labels output from the operational model. In particular, the model evaluation device 10 learns the check model by using the loss function shown in equation 1 above, setting the loss weight of the training data corresponding to the predicted labels output from the operational model to be higher than the weights for other labels.

[0035] In this way, the model evaluation device 10 focuses on training the check model for the predicted labels of the operational model, while also training for other labels, thereby generating a check model. As a result, the trained check model may improve the accuracy of its probability prediction for each label.

[0036] Next, the model evaluation device 10 inputs the target data to be estimated, for which no correct labels have been assigned, into the operational model and the check model, respectively (step S2). Then, the model evaluation device 10 obtains the predicted labels, which are the output from the operational model, and the predicted probabilities of each label, which are the output from the check model, and evaluates the operational model based on these (step S3). Specifically, the model evaluation device 10 calculates statistical values ​​such as the average value of the predicted probabilities of the labels output from the check label that correspond to the predicted labels output from the operational model, and uses these as accuracy estimates to evaluate the operational model. Then, as shown in Figure 5, the model evaluation device 10 outputs the calculated accuracy estimates by displaying them on the display of an information processing device operated by the evaluator who is evaluating the operational model (step S4). At this time, the model evaluation device 10 may also categorize the evaluation values ​​into pre-set ranks and output the corresponding rank.

[0037] <Embodiment 2> Next, a second embodiment of the present disclosure will be described with reference to Figures 7 to 8. Figures 7 to 8 are block diagrams showing the configuration of the model evaluation apparatus in Embodiment 2. In this embodiment, the configuration of the model evaluation apparatus described in the above-described embodiment is shown in outline.

[0038] First, with reference to Figure 7, the hardware configuration of the model evaluation device 100 in this embodiment will be described. The model evaluation device 100 is composed of a general information processing device, and as an example, it is equipped with the following hardware configuration. ·CPU(Central Processing Unit)101(Arithmetic unit) ROM (Read Only Memory) 102 (Storage Device) • RAM (Random Access Memory) 103 (Storage Device) • Program group 104 loaded into RAM 103 • Storage device 105 for storing the program group 104 • Drive device 106 for reading and writing to external storage medium 110 of the information processing device. • Communication interface 107 connecting to a communication network 111 outside the information processing device. • Input / output interface 108 for data input and output. • Bus 109 connecting each component

[0039] Figure 7 shows an example of the hardware configuration of the information processing device, which is the model evaluation device 100, and the hardware configuration of the information processing device is not limited to the case described above. For example, the information processing device may consist of only a part of the configuration described above, such as not having the drive device 106. In addition, the information processing device may use a GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof instead of the CPU described above.

[0040] The model evaluation device 100 can be equipped with the learning unit 121 and evaluation unit 122 shown in Figure 8 by having the CPU 101 acquire the program group 104 and execute it. The program group 104 is, for example, stored in advance in a storage device 105 or ROM 102, and the CPU 101 loads it into RAM 103 and executes it as needed. The program group 104 may also be supplied to the CPU 101 via a communication network 111, or it may be stored in advance in a storage medium 110, and the drive device 106 reads the program and supplies it to the CPU 101. However, the learning unit 121 and evaluation unit 122 described above may be constructed with dedicated electronic circuits to realize such means.

[0041] The learning unit 121 learns a second learning model that outputs a prediction probability for the input training data, based on the prediction output from the first learning model for the input training data. At this time, the learning unit 121 learns the second learning model so that it outputs a prediction probability for each of the multiple classifications from which the input training data is predicted, based on the classification output from the first learning model for the input training data. Furthermore, the learning unit 121 learns the second learning model by changing the learning weights for each classification of the training data used to learn the second learning model, according to the classification output from the first learning model for the input training data.

[0042] The evaluation unit 122 evaluates the first learning model based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for a given input of predetermined input data.

[0043] As described above, this disclosure allows for the training of a second learning model based on the prediction results output from the first learning model in response to the input training data, thereby improving the accuracy of the output from the second learning model. Furthermore, by evaluating the first learning model using the second learning model, the first learning model can be evaluated quickly, inexpensively, and appropriately.

[0044] The programs described above can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0045] Although the present disclosure has been described above with reference to the embodiments described above, the present disclosure is not limited to the embodiments described above. Various modifications to the configuration and details of the present disclosure can be made that are understandable to those skilled in the art within the scope of the present disclosure. Furthermore, at least one of the functions of the learning unit 121 and evaluation unit 122 described above may be performed on an information processing device installed and connected at any location on the network, that is, it may be performed using so-called cloud computing.

[0046] <Note> Some or all of the above embodiments may also be described as follows. The following outlines the configuration of the model evaluation apparatus, model evaluation method, and program in this disclosure. However, this disclosure is not limited to the following configuration. (Note 1) A learning unit that learns a second learning model that outputs the probability of prediction for the same training data input, based on the prediction output from the first learning model for the same training data input. An evaluation unit evaluates the first learning model based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for a given input of predetermined input data. A model evaluation device equipped with the following features. (Note 2) The model evaluation device described in Appendix 1, The learning unit trains the second learning model so that, based on the classification predicted by the first learning model for the input training data, the second learning model outputs the probability of prediction for each of the multiple classifications from among the multiple classifications set in advance, for the input training data. Model evaluation device. (Note 3) The model evaluation device described in Appendix 2, The learning unit changes the learning weights for each classification of the training data used to train the second learning model, according to the classification, which is the prediction output from the first learning model in response to the input of the training data, and trains the second learning model. Model evaluation device. (Note 4) The model evaluation device described in Appendix 3, The learning unit, when a specific classification is output as a prediction from the first learning model in response to the input of the training data, learns the second learning model by making the learning weight for that specific classification in the training data used for learning the second learning model higher than that for other classifications. Model evaluation device. (Note 5) The model evaluation device described in Appendix 4, The learning unit learns the second learning model by setting the weight of the loss during learning for the specific classification of the training data used to learn the second learning model to be higher than that for other classifications. Model evaluation device. (Note 6) The model evaluation device described in Appendix 1, The evaluation unit evaluates the first learning model based on the probability of the prediction output from the second learning model for the input of the predetermined input data, which corresponds to the prediction output from the first learning model for the input of the predetermined input data. Model evaluation device. (Note 7) The model evaluation device described in Appendix 6, The evaluation unit evaluates the first learning model based on the probabilities of the predictions output from the second learning model for each of the multiple predetermined input data, which correspond to the predictions output from the first learning model for each of the multiple predetermined input data. Model evaluation device. (Note 8) Based on the predictions output by the first learning model for the input training data, a second learning model is trained to output the probability of prediction for the same input training data. The first learning model is evaluated based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for the given input of predetermined input data. Model evaluation method. (Note 9) The model evaluation method described in Appendix 8, Based on the classification predicted by the first learning model for the input training data, among a set of pre-configured classifications, the second learning model is trained to output the probability of prediction for each of the multiple classifications for the input training data. Model evaluation method. (Note 10) The model evaluation method described in Appendix 9, The second learning model is trained by changing the learning weights for each classification of the training data used in training the second learning model, according to the classification, which is the prediction output from the first learning model in response to the input of the training data. Model evaluation method. (Note 11) The model evaluation method described in Appendix 10, If the first learning model outputs a specific classification as a prediction in response to the input of the training data, the second learning model is trained by setting the learning weight for that specific classification in the training data used for training the second learning model to be higher than that for other classifications. Model evaluation method. (Note 12) Based on the predictions output by the first learning model for the input training data, a second learning model is trained to output the probability of prediction for the same input training data. The first learning model is evaluated based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for the given input of predetermined input data. A computer-readable storage medium that stores a program to cause a computer to execute a process. [Explanation of Symbols]

[0047] 10 Model evaluation device 11. Check Model Generation Unit 12 Performance Evaluation Department 16 Model Memory Unit 17 Training data storage unit 18 Estimated target data storage unit 100 Model Evaluation Devices 101 CPU 102 ROM 103 RAM 104 Program Groups 105 Storage device 106 Drive unit 107 Communication Interface 108 Input / Output Interfaces 109 Bus 110 Storage medium 111 Communication Network 121 Learning Department 122 Evaluation Department

Claims

1. A learning unit that learns a second learning model that outputs the probability of prediction for the same training data input, based on the prediction output from the first learning model for the same training data input. An evaluation unit evaluates the first learning model based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for a given input of predetermined input data. A model evaluation device equipped with the following features.

2. A model evaluation apparatus according to claim 1, The learning unit trains the second learning model so that, based on the classification predicted by the first learning model for the input training data, the second learning model outputs the probability of prediction for each of the multiple classifications from among the multiple classifications set in advance, for the input training data. Model evaluation device.

3. A model evaluation apparatus according to claim 2, The learning unit changes the learning weights for each classification of the training data used to train the second learning model, according to the classification, which is the prediction output from the first learning model in response to the input of the training data, and trains the second learning model. Model evaluation device.

4. A model evaluation apparatus according to claim 3, The learning unit, when a specific classification is output as a prediction from the first learning model in response to the input of the training data, learns the second learning model by making the learning weight for that specific classification in the training data used for learning the second learning model higher than that for other classifications. Model evaluation device.

5. A model evaluation apparatus according to claim 4, The learning unit learns the second learning model by setting the weight of the loss during learning for the specific classification of the training data used to learn the second learning model to be higher than that for other classifications. Model evaluation device.

6. A model evaluation apparatus according to claim 1, The evaluation unit evaluates the first learning model based on the probability of the prediction output from the second learning model for the input of the predetermined input data, which corresponds to the prediction output from the first learning model for the input of the predetermined input data. Model evaluation device.

7. A model evaluation apparatus according to claim 6, The evaluation unit evaluates the first learning model based on the probabilities of the predictions output from the second learning model for each of the multiple predetermined input data, which correspond to the predictions output from the first learning model for each of the multiple predetermined input data. Model evaluation device.

8. Based on the predictions output by the first learning model for the input training data, a second learning model is trained to output the probability of prediction for the same input training data. The first learning model is evaluated based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for the given input of predetermined input data. Model evaluation method.

9. A model evaluation method according to claim 8, Based on the classification predicted by the first learning model for the input training data, among a set of pre-configured classifications, the second learning model is trained to output the probability of prediction for each of the multiple classifications for the input training data. Model evaluation method.

10. Based on the predictions output by the first learning model for the input training data, a second learning model is trained to output the probability of prediction for the same input training data. The first learning model is evaluated based on the prediction output from the first learning model for a given input of predetermined input data and the probability of the prediction output from the second learning model for the given input of predetermined input data. A program that causes a computer to perform a process.

Images