Model evaluation device, model evaluation method, and program

Abstract

A model evaluation device 100 according to the present disclosure is provided with: a learning unit 121 that machine-learns a second machine learning model so as to predict the error between first prediction data output from a first machine learning model in response to input of explanatory variable data of training data and objective variable data corresponding to the explanatory variable data of the training data, on the basis of the first prediction data and second prediction data output from the second machine learning model in response to input of the explanatory variable data of the training data; and an evaluation unit 122 that evaluates the prediction performance of the first machine learning model on the basis of the error between third prediction data output from the first machine learning model in response to input of evaluation data and fourth prediction data output from the second machine learning model in response to input of the evaluation data. Thus, the model evaluation device 100 can assist users in decision making.

Description

Model evaluation device, model evaluation method, and program

[0001] The present disclosure relates to a model evaluation device, a model evaluation method, and a program.

[0002] There are technologies for evaluating and managing the performance of machine learning models. For example, in Patent Literature 1, labels are assigned to data to generate pseudo-correct data, and the performance of a machine learning model can be evaluated using the pseudo-correct data.

[0003] International Publication No. 2020 / 225923

[0004] However, when evaluating the performance of a machine learning model, a problem arises in that it takes time and cost to obtain the necessary data. For example, when labeled data cannot be obtained until a predetermined time has passed or when labeled data is generated through research by experts, obtaining the data takes time and costs money. Furthermore, even when pseudo-correct data is generated using the technology of Patent Document 1, it takes time and costs money.

[0005] Therefore, an object of the present disclosure is to provide a model evaluation device that can solve the above-mentioned problem of the time and cost required when evaluating the performance of a machine learning model.

[0006] A model evaluation device according to one embodiment of the present disclosure includes: a learning unit that trains a second machine learning model to predict an error between first prediction data output from a first machine learning model in response to input explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first prediction data and second prediction data output from the second machine learning model in response to input explanatory variable data of the training data; and an evaluation unit that evaluates the predictive performance of the first machine learning model based on an error between third prediction data output from the first machine learning model in response to input evaluation data and fourth prediction data output from the second machine learning model in response to input evaluation data. Furthermore, a model evaluation method according to one embodiment of the present disclosure is configured to: train a second machine learning model to predict an error between first prediction data output from a first machine learning model in response to input explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first prediction data and second prediction data output from the second machine learning model in response to input explanatory variable data of the training data; and evaluate the predictive performance of the first machine learning model based on an error between third prediction data output from the first machine learning model in response to input evaluation data and fourth prediction data output from the second machine learning model in response to input evaluation data.Furthermore, a program according to one embodiment of the present disclosure has a configuration that causes a computer to execute the following processes: train a second machine learning model to predict an error between first predicted data output from a first machine learning model in response to input explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first predicted data and second predicted data output from the second machine learning model in response to input explanatory variable data of the training data; and evaluate the predictive performance of the first machine learning model based on an error between third predicted data output from the first machine learning model in response to input evaluation data and fourth predicted data output from the second machine learning model in response to input evaluation data.

[0007] By being configured as described above, the present disclosure can reduce the time and cost required when evaluating the performance of a machine learning model.

[0008] Fig. 1 is a block diagram showing a configuration of a model evaluation device according to the present disclosure; Fig. 2 is a flowchart showing a processing operation of a model evaluation device according to the present disclosure; Fig. 3 is a block diagram showing a hardware configuration of a model evaluation device according to the present disclosure; Fig. 4 is a block diagram showing a configuration of a model evaluation device according to the present disclosure.

[0009] First Embodiment A first embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to any embodiment.

[0010] [Configuration] The model evaluation device 10 in this embodiment is used to evaluate the predictive performance of an operational model (first machine learning model) that has already been generated and is being used in a predetermined situation and that predicts prediction data from input data. Here, if correct answer data corresponding to the input data is used to evaluate the predictive performance of the operational model, obtaining such correct answer data may require time and cost. Therefore, the model evaluation device in this embodiment evaluates the predictive performance of the operational model using input data for which correct answer data corresponding to the input data is not available.

[0011] The model evaluation device 10 in this embodiment is configured with one or more information processing devices each including a calculation device and a storage device. As shown in FIG. 1 , the model evaluation device 10 includes a learning unit 11 and an evaluation unit 12. The functions of the learning unit 11 and the evaluation unit 12 can be realized by the calculation device executing a program for realizing each function stored in the storage device. The model evaluation device 10 also includes a model storage unit 16, a training data storage unit 17, and an evaluation data storage unit 18. The model storage unit 16, the training data storage unit 17, and the evaluation data storage unit 18 are each configured with a storage device. Each component will be described in detail below.

[0012] The model storage unit 16 stores an operational model (first machine learning model) generated by executing a machine learning algorithm using previously prepared training data. The operational model is a model used by a predetermined prediction system, and is a target for evaluating prediction performance.

[0013] Here, the operational model f is generated in advance through machine learning using training data consisting of explanatory variable data x and objective variable data y corresponding to the explanatory variable data x. Specifically, the operational model f is generated by machine learning so that when explanatory variable data x is input, objective variable data y corresponding to the explanatory variable data x is output. As a result, the operational model f can input explanatory variable data x', whose objective variable data is unknown, and output prediction data f(x') that predicts objective variable data.

[0014] The model storage unit 16 also stores a check model h (second machine learning model) generated as described below.

[0015] The training data storage unit 17 stores explanatory variable data x ｉ and the i-th explanatory variable data x ｉ The objective variable data y corresponding to ｉ and the training data (x ｉ , y ｉ) is stored. Note that i indicates the order of the training data consisting of multiple sets. The training data may be used to generate the above-mentioned operational model, or may be data different from that used to generate the operational model. The training data is then used to generate the check model h, as will be described later.

[0016] The evaluation data storage unit 18 stores explanatory variable data x' ｊ That is, the evaluation data is the explanatory variable data x' ｊ The corresponding objective variable data is not assigned, and the explanatory variable data x' ｊ It is composed of only j. Note that j indicates the order of the multiple evaluation data. As will be described later, the evaluation data is input to the operational model f and the check model h when evaluating the operational model f, and is used to predict the prediction data. For example, the evaluation data is data measured from a prediction system in which the operational model f is operated, or data suitable for prediction by the operational model f or the check model h.

[0017] As described above, the learning unit 11 uses the operational model f stored in the model storage unit 16 and the training data (x ｉ , y ｉ ) and performs machine learning, which will be described later, to generate a check model h. Specifically, the learning unit 11 performs machine learning using a loss L shown in the following equation (1). Here, x ｉ , y ｉ is the i-th explanatory variable data x in the training data ｉ and the objective variable data y ｉ represents l １ , l ２ , l ３ represents the loss function, f represents the operational model (first machine learning model), and h represents the check model (second machine learning model). １ , l ２ , l ３For example, squared error, absolute error, squared logarithmic error, absolute logarithmic error, Huber loss, squared percentage error, absolute percentage error, etc. When applying the operational model to a classification task, the loss function may be O1 loss, cross entropy, etc.

[0018] In the above-mentioned formula 1, l ２ (f(x ｉ ), y ｉ ) is the explanatory variable data x of the training data for the operational model f. ｉ Predicted data f(x ｉ ) (first prediction data) and the explanatory variable data x of the training data ｉ The objective variable data y corresponding to ｉ In addition, in equation 1, l ３ (f(x ｉ ), h(x ｉ )) is the predicted data f(x) for the training data by the operational model f ｉ ) and the explanatory variable data x of the training data for the check model h ｉ The predicted data h(x ｉ ) (second predicted data). Therefore, the loss L described above can be said to be represented by the difference between the prediction error of the operational model f for the training data and the prediction error between the operational model f and the check model h.

[0019] Then, the learning unit 11 performs machine learning on the check model h so as to minimize the above-mentioned loss L. In other words, the learning unit 11 performs machine learning on the check model h so as to minimize the loss L corresponding to the difference between the prediction error of the above-mentioned operation model f for the training data and the prediction error between the operation model f and the check model h. ｉ Prediction data h(x ｉ In other words, the learning unit 11 performs machine learning on the check model h so that the check model h outputs the explanatory variable data x of the training data so that the value of the prediction error between the operation model f and the check model h approaches the value of the prediction error for the training data of the operation model f. ｉ Prediction data h(x ｉThe check model h is machine-learned to output the

[0020] Here, the reason why the loss L is set as in the above-described formula 1 in this embodiment will be explained. First, the following formula 2 can be considered as a loss L' that is different from the loss L exemplified in this embodiment.

[0021] Equation 2 expresses the loss L' corresponding to the difference between the prediction error of the operational model f for the training data and the prediction of the check model h. In this case, if the operational model f is overtrained, for example, the error of the operational model f for the training data becomes 0 (f(x ｉ ) = y ｉ ), the loss L' is expressed by the following equation (3).

[0022] When the check model h is machine-learned using the loss L' expressed by the above formula 3, h(x) always becomes 0. Therefore, when the loss L' shown in the above formula 2 is set, the check model h cannot be machine-learned appropriately.

[0023] In contrast, when the loss L shown in Equation 1 is set as in this embodiment, when the operational model f is over-learned as described above, the error of the operational model f with respect to the training data becomes 0 (f(x ｉ ) = y ｉ )), the loss L is expressed as in Equation 4.

[0024] From Equation 4, the prediction of the check model h (h(x ｉ )) is the objective variable data of the training data (y ｉ ), the check model h is machine-learned so that it approaches the loss function l. １ , l ３ If each of the above is a square error, then Equation 4 can be expressed as Equation 5, and is appropriately learned.

[0025] Here, a regularization term may be added to the loss L as shown in Equation 6. This makes it possible to suppress overlearning of the check model h and ensure appropriate learning, even when the error of the operational model f becomes a small value due to overlearning. Note that θ represents a parameter of the check model h, and λ represents the importance of the regularization term.

[0026] In addition, the loss L described above is assigned a weight w ｉ In this case, the weight w ｉ may be a fixed constant or may be a value that changes dynamically during the course of machine learning.

[0027] For example, the weight w ｉ is a value that changes dynamically during the machine learning process, the prediction error l for the training data of the operational model f ２ (f(x ｉ ), y ｉ ) and the prediction error l between the operational model f and the check model h ３ (f(x ｉ ), h(x ｉ ) and may be a value that changes dynamically depending on the value of ｉ may be set as shown in the following formula 8. Here, η is a preset parameter.

[0028] In the above case, if the prediction error between the operational model f and the check model h is optimistic, the weight can be increased, making it possible to perform learning that avoids optimistic predictions.

[0029] The learning unit 11 performs machine learning to generate a check model h that minimizes the loss L set as above, and stores the generated check model h in the model storage unit 16.

[0030] The evaluation unit 12 evaluates the prediction performance of the operation model f using the check model h generated as described above and the evaluation data. Specifically, the evaluation unit 12 evaluates the prediction performance P using the following equation (9).

[0031] Here, x' ｊis explanatory variable data of the j-th evaluation data, and there is no corresponding response variable data in the evaluation data. ４ is a function that calculates the performance to be predicted, for example, the mean square error. ４ is the above-mentioned l ２ Ya l ３ It may be the same function as or a different function.

[0032] In this way, the evaluation unit 12 calculates the explanatory variable data x' of the evaluation data for each of the operation model f and the check model h. ｊ Each predicted data (f(x' ｊ ), h(x' ｊ )) based on the error, the predictive performance P of the operational model f can be evaluated.

[0033] [Operation] Next, the operation of the above-mentioned model evaluation device 10 will be described. The model evaluation device 10 stores a pre-generated operational model f and also stores training data (x ｉ , y ｉ ) and evaluation data (x' ｊ ) is stored.

[0034] The model evaluation device 10 evaluates the operational model f and the training data (x ｉ , y ｉ ) to calculate the loss L as shown in Equation 1, and machine-learn the check model h so as to minimize the loss L (step S1 in FIG. 2). Specifically, the model evaluation device 10 machine-learns the check model h so as to minimize the loss L corresponding to the difference between the prediction error of the operation model f for the training data and the prediction error between the operation model f and the check model h when the explanatory variable data of the training data are input.

[0035] Thereafter, the model evaluation device 10 evaluates the predictive performance of the operational model f using the generated check model h. Specifically, the model evaluation device 10 evaluates the predictive performance of the operational model f using explanatory variable data x' of the evaluation data for the operational model f and the check model h. ｊ(Step S2 in FIG. 2). Then, the model evaluation device 10 evaluates the prediction performance P of the operation model f based on the error in the prediction data of the operation model f and the check model h, using a formula for calculating the prediction evaluation P as shown in FIG. 9 (Step S3 in FIG. 2).

[0036] [Application Example] Here, as an application example of the above-mentioned model evaluation device 10, a case where it is used in the healthcare field in a hospital, etc. will be described. For example, assume that the operation model f, the prediction performance of which is to be evaluated by the model evaluation device 10, is operated to predict the hospital bed occupancy rate one week from now as dependent variable data. In this case, the explanatory variables input to the operation model f include the date of the month, day of the week, season, current bed occupancy rate, moving average of past bed occupancy rates, local infectious disease outbreak status, number of visiting patients, etc.

[0037] The model evaluation device 10 generates a check model h by machine learning based on training data consisting of explanatory variable data and objective variable data as described above, and an operation model f, so as to minimize the loss L shown in the above-described equation 1. The model evaluation device 10 then inputs explanatory variable data of the evaluation data into the operation model f and the generated check model h, and evaluates the predictive performance of the operation model f from the output.

[0038] In this way, by using the model evaluation device 10 according to the present disclosure in a hospital and calculating the bed occupancy rate using an operational model, medical staff such as doctors can make decisions about admitting patients.

[0039] Second Embodiment Next, a second embodiment of the present disclosure will be described with reference to the drawings. This embodiment shows an outline of the configuration of the model evaluation device described in the above embodiment. Note that Figures 3 and 4 are diagrams for explaining the configuration, and these drawings may be relevant to any of the embodiments.

[0040] First, the hardware configuration of the model evaluation device 100 will be described with reference to Fig. 3. The model evaluation device 100 is configured as a general information processing device, and is equipped with the following hardware configuration, for example: - CPU (Central Processing Unit) 101 (arithmetic device) - ROM (Read Only Memory) 102 (storage device) - RAM (Random Access Memory) 103 (storage device) - Programs 104 loaded into the RAM 103 - Storage device 105 storing the programs 104 - Drive device 106 for reading and writing data from and to a storage medium 110 external to the information processing device - Communication interface 107 for connecting to a communication network 111 external to the information processing device - Input / output interface 108 for inputting and outputting data - Bus 109 for connecting the various components

[0041] 3 shows an example of the hardware configuration of the information processing device that is the model evaluation device 100, and the hardware configuration of the information processing device is not limited to the above-described case. For example, the information processing device may be configured with a part of the above-described configuration, such as not including the drive device 106. Furthermore, instead of the above-described CPU, the information processing device may use a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point Number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.

[0042] The model evaluation device 100 can be equipped with the learning unit 121 and evaluation unit 122 shown in FIG. 4 by having the CPU 101 acquire and execute the program group 104. The program group 104 is, for example, stored in advance in the storage device 105 or the ROM 102, and is loaded into the RAM 103 and executed by the CPU 101 as needed. The program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in advance in the storage medium 110, and the drive device 106 may read out the programs and supply them to the CPU 101. However, the learning unit 121 and evaluation unit 122 described above may be constructed using dedicated electronic circuits for realizing such means.

[0043] The learning unit 121 trains the second machine learning model to predict an error between first predicted data output from a first machine learning model in response to input of explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first predicted data and second predicted data output from a second machine learning model in response to input of the explanatory variable data of the training data. For example, the learning unit 121 trains the second machine learning model by using a loss based on the error between the first predicted data and the dependent variable data and the error between the first predicted data and the second predicted data.

[0044] The evaluation unit 122 evaluates the predictive performance of the first machine learning model based on the error between third predicted data output from the first machine learning model in response to input of evaluation data and fourth predicted data output from the second machine learning model in response to input of the evaluation data.

[0045] With the above configuration, the present disclosure can evaluate the performance of a machine learning model without obtaining ground truth data for evaluation data, thereby reducing the time and cost required for performance evaluation. Furthermore, by performing machine learning using the loss described above, even if the machine learning model to be evaluated is overfitted, it is possible to train and generate a second machine learning model that can appropriately serve as a check model, thereby enabling appropriate evaluation of the machine learning model to be evaluated.

[0046] In addition, at least one of the functions of the learning unit 121 and the evaluation unit 122 described above may be executed by an information processing device installed and connected anywhere on the network, that is, they may be executed by so-called cloud computing.

[0047] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-RWs, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program can also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0048] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each of the above-described embodiments can be combined with other embodiments as appropriate.

[0049] <Supplementary Notes> Some or all of the above embodiments may be described as in the following supplementary notes. Below, an outline of the configurations of a model evaluation device, a model evaluation method, and a program according to the present disclosure will be described. However, the present disclosure is not limited to the following configurations. (Supplementary Note 1) A model evaluation device comprising: a learning unit that trains a second machine learning model by machine learning to predict an error between first prediction data output from a first machine learning model in response to input of explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first prediction data and second prediction data output from a second machine learning model in response to input of the explanatory variable data of the training data; and an evaluation unit that evaluates the predictive performance of the first machine learning model based on an error between third prediction data output from the first machine learning model in response to input of evaluation data and fourth prediction data output from the second machine learning model in response to input of the evaluation data. (Supplementary Note 2) The model evaluation device according to Supplementary Note 1, wherein the learning unit trains the second machine learning model by machine learning using a loss based on an error between the first prediction data and the dependent variable data and an error between the first prediction data and the second prediction data. (Supplementary Note 3) The model evaluation device according to Supplementary Note 2, wherein the learning unit trains the second machine learning model by machine learning using a loss according to a difference between the error between the first prediction data and the dependent variable data and an error between the first prediction data and the second prediction data. (Supplementary Note 4) The model evaluation device according to Supplementary Note 2, wherein the learning unit trains the second machine learning model by machine learning using a loss weighted based on the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data. (Supplementary Note 5) The model evaluation device according to Supplementary Note 4, wherein the learning unit performs machine learning of the second machine learning model using a loss weighted according to a difference between an error between the first prediction data and the dependent variable data and an error between the first prediction data and the second prediction data.(Supplementary Note 6) A model evaluation method comprising: training a second machine learning model to predict an error between first predicted data output from a first machine learning model in response to input explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first predicted data and second predicted data output from the second machine learning model in response to input of the explanatory variable data of the training data; and evaluating the predictive performance of the first machine learning model based on an error between third predicted data output from the first machine learning model in response to input of evaluation data and fourth predicted data output from the second machine learning model in response to input of the evaluation data. (Supplementary Note 7) A model evaluation method according to Supplementary Note 6, comprising: training the second machine learning model by machine learning using losses based on an error between the first predicted data and the dependent variable data and an error between the first predicted data and the second predicted data. (Supplementary Note 8) The model evaluation method according to Supplementary Note 7, wherein the second machine learning model is trained using a loss according to the difference between the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data. (Supplementary Note 9) The model evaluation method according to Supplementary Note 7, wherein the second machine learning model is trained using a loss weighted based on the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data. (Supplementary Note 10) The model evaluation method according to Supplementary Note 9, wherein the second machine learning model is trained using a loss weighted based on the difference between the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data.(Supplementary Note 11) A computer-readable storage medium storing a program that causes a computer to execute the following processes: training a second machine learning model to predict an error between first predicted data output from a first machine learning model in response to input explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first predicted data and second predicted data output from the second machine learning model in response to input explanatory variable data of the training data; and evaluating the predictive performance of the first machine learning model based on an error between third predicted data output from the first machine learning model in response to input evaluation data and fourth predicted data output from the second machine learning model in response to input evaluation data.

[0050] REFERENCE SIGNS LIST 10 Model evaluation device 11 Learning unit 12 Evaluation unit 16 Model storage unit 17 Training data storage unit 18 Evaluation data storage unit 100 Model evaluation device 101 CPU 102 ROM 103 RAM 104 Program group 105 Storage device 106 Drive device 107 Communication interface 108 Input / output interface 109 Bus 110 Storage medium 111 Communication network 121 Learning unit 122 Evaluation unit

Claims

1. A model evaluation device comprising: a learning unit that trains a second machine learning model by machine learning to predict an error between first predicted data output from a first machine learning model in response to input of explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first predicted data and second predicted data output from a second machine learning model in response to input of the explanatory variable data of the training data; and an evaluation unit that evaluates the predictive performance of the first machine learning model based on an error between third predicted data output from the first machine learning model in response to input of evaluation data and fourth predicted data output from the second machine learning model in response to input of the evaluation data.

2. A model evaluation device as described in claim 1, wherein the learning unit machine-trains the second machine learning model using a loss based on an error between the first prediction data and the dependent variable data and an error between the first prediction data and the second prediction data.

3. A model evaluation device as described in claim 2, wherein the learning unit machine-trains the second machine learning model using a loss corresponding to the difference between the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data.

4. A model evaluation device as described in claim 2, wherein the learning unit trains the second machine learning model using a loss weighted based on the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data.

5. A model evaluation device as described in claim 4, wherein the learning unit machine-trains the second machine learning model using a loss weighted according to the difference between the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data.

6. A model evaluation method comprising: training a second machine learning model to predict an error between first predicted data output from a first machine learning model in response to input of explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first predicted data and second predicted data output from a second machine learning model in response to input of the explanatory variable data of the training data; and evaluating the predictive performance of the first machine learning model based on the error between third predicted data output from the first machine learning model in response to input of evaluation data and fourth predicted data output from the second machine learning model in response to input of the evaluation data.

7. A model evaluation method as described in claim 6, comprising: machine learning the second machine learning model using a loss based on an error between the first prediction data and the dependent variable data and an error between the first prediction data and the second prediction data.

8. A model evaluation method as described in claim 7, comprising: machine learning the second machine learning model using a loss corresponding to the difference between the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data.

9. A model evaluation method as described in claim 7, comprising: machine learning the second machine learning model using a loss weighted based on the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data.

10. A model evaluation method as described in claim 9, comprising: machine learning the second machine learning model using a loss weighted according to the difference between the error between the first prediction data and the dependent variable data and the error between the first prediction data and the second prediction data.

11. A computer-readable storage medium storing a program for causing a computer to execute a process of: training a second machine learning model to predict an error between first predicted data output from a first machine learning model in response to input of explanatory variable data of training data and dependent variable data corresponding to the explanatory variable data of the training data, based on the first predicted data and second predicted data output from the second machine learning model in response to input of the explanatory variable data of the training data; and evaluating the predictive performance of the first machine learning model based on the error between third predicted data output from the first machine learning model in response to input of evaluation data and fourth predicted data output from the second machine learning model in response to input of the evaluation data.

Images