Model evaluation device, model evaluation method, and program
The model evaluation device efficiently evaluates predictive models by learning a second model to output prediction probabilities, addressing the cost and time issues of obtaining correct data, thereby improving prediction accuracy and reducing evaluation costs.
Patent Information
- Application Number
- PCT/JP2023/017589
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-14
AI Technical Summary
Evaluating the quality of predictive models is time-consuming and costly due to the need for correct data, which can be difficult to obtain, especially when pseudo-correct answers require significant time and resources to generate.
A model evaluation device and method that learns a second learning model to output prediction probabilities based on the output of a first learning model, allowing for the evaluation of the first learning model using input data without correct labels, by adjusting learning weights to focus on predicted labels and generating a check model that improves prediction accuracy.
Enables quick and cost-effective evaluation of predictive model quality by using input data without correct labels, improving prediction accuracy and reducing the need for expensive and time-consuming correct data generation.
Smart Images

Figure JP2023017589_14112024_PF_FP_ABST
Abstract
Description
Model evaluation device, model evaluation method, and program
[0001] The present disclosure relates to a model evaluation device, a model evaluation method, and a program.
[0002] A prediction model is created that predicts dependent variable data from new explanatory variable data based on known explanatory variable data and dependent variable data, and the created prediction model is operated. In this case, it is necessary to manage the performance of the prediction model, and the predictive performance of the prediction model is also evaluated. When evaluating the predictive performance of a prediction model, explanatory variable data that serves as new input data and dependent variable data that serves as corresponding ground truth data are required.
[0003] International Publication No. 2020 / 225923
[0004] However, depending on the circumstances under which a predictive model is operated, it may take time and cost to obtain correct data corresponding to new input data that is the target of prediction by the predictive model. For example, when correct data cannot be obtained until a predetermined time has passed or when correct data is generated through research by experts, etc., obtaining the data may take time and cost. Furthermore, Patent Document 1 describes assigning a label to non-correct data to generate pseudo-correct data, but generating pseudo-correct data takes time and costs money.
[0005] Thus, when evaluating a predictive model, if correct answer data corresponding to new input data to be predicted is used, it takes time and money to obtain such correct answer data, which creates the problem that the predictive model cannot be evaluated quickly and at low cost.
[0006] Therefore, an object of the present disclosure is to provide a model evaluation device that can solve the above-mentioned problem of being unable to evaluate the quality of a predictive model quickly and at low cost.
[0007] A model evaluation device according to one embodiment of the present disclosure is configured to include: a learning unit that learns a second learning model that outputs a probability of a prediction for input of training data based on a prediction output from a first learning model for input of the training data; and an evaluation unit that evaluates the first learning model based on the prediction output from the first learning model for input of predetermined input data and the probability of the prediction output from the second learning model for input of the predetermined input data.
[0008] Furthermore, a model evaluation method that is one embodiment of the present disclosure is configured to: learn a second learning model that outputs a probability of a prediction for input of training data based on a prediction output from a first learning model for input of the training data; and evaluate the first learning model based on the prediction output from the first learning model for input of specified input data and the probability of the prediction output from the second learning model for input of the specified input data.
[0009] Furthermore, a program that is one form of the present disclosure is configured to cause a computer to execute the following processes: learn a second learning model that outputs a probability of a prediction for input of training data based on a prediction output from a first learning model for input of the training data; and evaluate the first learning model based on the prediction output from the first learning model for input of specified input data and the probability of the prediction output from the second learning model for input of the specified input data.
[0010] As configured as described above, the present disclosure makes it possible to evaluate the quality of a predictive model quickly and at low cost.
[0011] FIG. 1 is a block diagram showing the configuration of a model evaluation device according to a first embodiment of the present disclosure. FIG. 2 is a diagram showing the processing state of the model evaluation device disclosed in FIG. 1. FIG. 3 is a diagram showing the processing state of the model evaluation device disclosed in FIG. 1. FIG. 4 is a diagram showing the processing state of the model evaluation device disclosed in FIG. 1. FIG. 5 is a flowchart showing the operation of the model evaluation device disclosed in FIG. 1. FIG. 6 is a block diagram showing the hardware configuration of a model evaluation device according to a second embodiment of the present disclosure. FIG. 7 is a block diagram showing the configuration of a model evaluation device according to a second embodiment of the present disclosure.
[0012] <First Embodiment> A first embodiment of the present disclosure will be described with reference to Fig. 1 to Fig. 6. Fig. 1 is a diagram for explaining the configuration of a model evaluation device, and Fig. 2 to Fig. 6 are diagrams for explaining the processing operation of the model evaluation device.
[0013] [Configuration] The model evaluation device 10 in this embodiment is used to evaluate the predictive performance of an operational model (first learning model) that predicts a correct label from input data. Here, if data with a correct label corresponding to the input data is used to evaluate the predictive performance of the operational model, it may take time and cost to obtain such data. In contrast, the model evaluation device in this embodiment can perform evaluation even using input data that does not have a correct label, as described below.
[0014] The model evaluation device 10 in this embodiment is configured with one or more information processing devices each including a calculation device and a storage device. As shown in FIG. 1 , the model evaluation device 10 includes a check model generation unit 11 and a performance evaluation unit 12. The functions of the check model generation unit 11 and the performance evaluation unit 12 can be realized by the calculation device executing a program for realizing each function stored in the storage device. The model evaluation device 10 also includes a model storage unit 16, a training data storage unit 17, and an estimation target data storage unit 18. The model storage unit 16, the training data storage unit 17, and the estimation target data storage unit 18 are each configured with a storage device. Each component will be described in detail below.
[0015] The model storage unit 16 stores an operational model (first learning model) generated by machine learning of previously prepared training data. Here, the training data is data in which a correct label to be predicted from input data is assigned to the input data. Therefore, the operational model generated by machine learning of the training data is trained to output a correct label to be predicted from the input data in response to the input of the input data. In this case, the input data of the training data is, for example, data representing an image of a target for which a correct label is to be predicted, or characteristics of the target, such as color, weight, and shape, and the correct label of the training data is a specific classification from among multiple classifications of the target that have been set. For example, in this embodiment, when targeting vegetables as described below, the input data may include image data of vegetables. In this case, multiple vegetable classifications (types) are set, and the correct data is one specific classification (type) from among the multiple vegetable classifications, such as cabbage, carrot, bell pepper, and tomato, as examples.
[0016] The operational model is installed in a predetermined prediction system and is currently in operation, or may be in operation in the future, and when estimation target data is input, it outputs a predicted label predicted from the data. Therefore, the operational model generated by machine learning the above-mentioned training data will output one of cabbage, carrot, bell pepper, or tomato as a predicted label (specific classification) from among multiple preset vegetable classifications, in response to input of image data of vegetables as input data. Furthermore, the operational model is subject to evaluation of its predictive performance, as described below.
[0017] The operational model in this embodiment is not necessarily limited to predicting the classification (type) of vegetables. In other words, the input data input to the operational model may be data of any target, and the operational model may predict any classification according to the target. For example, the operational model may predict a person's health condition, and as an example, the operational model may use the person's current condition as input data and output the person's predicted health condition at a later date as a prediction.
[0018] The training data storage unit 17 stores the training data in which the above-described input data is previously assigned with a correct answer label. The training data may be used to generate the above-described operation model, or may be used when generating a check model as described below. Furthermore, the training data may include other data used to generate a check model, different from that used to generate the operation model.
[0019] The estimation target data storage unit 18 stores estimation target data to which no correct label has been assigned for the input data. In other words, the estimation target data is only input data, and as described below, is data that is input to the operation model and the check model when evaluating the operation model and is used to estimate a predicted label. For example, the estimation target data may be image data capturing the appearance of a vegetable whose classification is unknown, or data representing characteristics such as color, weight, and shape of a vegetable whose classification is unknown. However, the estimation target data is not necessarily limited to data on vegetables, and may be any data corresponding to the estimation target.
[0020] The check model generation unit 11 (learning unit) performs machine learning using training data in which correct labels are assigned to the above-mentioned input data stored in the training data storage unit 17, thereby generating a check model (second learning model) that outputs the probability of each label, including the correct label, in response to input of input data. For example, in this embodiment, each vegetable classification is set as each label as described above, and the check model generation unit 11 trains the check model to output a predicted probability for each of multiple vegetable classifications for the input data. As a result, in response to input of input data such as vegetable image data, the generated check model outputs the probability that the input data will fall into each vegetable classification (for example, cabbage, carrot, bell pepper, and tomato) corresponding to the input data.
[0021] Here, in this embodiment, when generating a check model using training data, the check model generation unit 11 also inputs the training data into the operational model. Then, the check model generation unit 11 learns to generate a check model using the training data based on the predicted label output from the operational model. Specifically, the check model generation unit 11 first inputs the training data used to generate the check model into the operational model and identifies the predicted label (specific classification) output from the operational model at that time. Then, the check model generation unit 11 learns the check model by changing the learning weight for each label of the training data according to the predicted label output from the operational model. In particular, the check model generation unit 11 learns the check model by increasing the learning weight for the label of the training data corresponding to the predicted label output from the operational model compared to the weights for other labels.
[0022] An example of the check model learning process by the check model generation unit 11 will now be described with reference to Figures 3 to 5. Here, as described above, the target for predicting classification is vegetables, and image data of vegetables is input as input data to generate a check model that outputs the probability of each classification (each type) of vegetables.
[0023] First, as shown in FIG. 2 , assume that the vegetable classification, which is the predicted label output when input data T1 of training data is input to the operational model, is "tomato." In this case, the check model generation unit 11 trains the check model by setting a higher weight when learning the probability that the input data T1 will be predicted as the vegetable classification "tomato" than the weight when learning other classifications "cabbage, carrot, bell pepper." Accordingly, the check model generation unit 11 trains the check model by setting a lower weight when learning the probability that the input data T1 will be predicted as the vegetable classification "cabbage, carrot, bell pepper" than the weight when learning other classifications "tomato." As a result, the check model generation unit 11 trains the check model so as to improve the accuracy of the probability that the input data T1 of training data will be predicted as the vegetable classification "tomato."
[0024] More specifically, when learning the check model, the check model generation unit 11 learns using the loss function shown in Equation 1. At this time, the operation model f and the check model h are assumed to be expressed by Equation 2, and the loss is assumed to be expressed by Equation 3 and Equation 4, with an example of the loss being shown in Equation 5. D represents the training data, and h(f(x)|x) represents the prediction probability for the label f(x) of the check model h in the input data x. Furthermore, Equation 6 illustrates an example of a loss function in a learning example different from this embodiment.
[0025] Here, Equation 3 represents the loss of prediction probability by the check model for the predicted label of the operational model f, and takes a minimum value of 1 when p = 1. Furthermore, Equation 4 represents the loss of prediction probability by the check model for a label different from the predicted label of the operational model f, and takes a minimum value of 0 when p = 0.
[0026] The loss function in this embodiment shown in Equation 1 has a term w added to Equation 6. 1 represents the learning weight of the prediction probability in the check model for the predicted label of the operational model f, and w0 represents the learning weight of the prediction probability in the check model for a label different from the predicted label of the operational model f. In this case, for example, "w 1 = 1, w 0 By setting "f(x)|x" to "0", the check model will be trained only for the predicted labels of the operational model. More specifically, the loss function in this case is expressed by equation 7, and only h(f(x)|x) appears in the loss function, so only h(f(x)|x) is learned, and labels other than f(x) are ignored during learning. For this reason, in the example of FIG. 2, only the predicted probability of f(x) = tomato is learned, and the predicted probabilities of the other categories (cabbage, carrot, bell pepper) are learned randomly.
[0027] Another example is "w 1 = 1 - α, w 0 By setting "α=0.1" as an example, the check model will be trained with a focus on the predicted labels of the operational model, as described above. 1 >w 0 " and it is desirable to set the weight of the loss corresponding to the label of the training data that corresponds to the predicted label output from the operational model higher than the weight of the loss for other labels. In this way, the check model is trained intensively on the predicted label of the operational model, while also training on other labels.
[0028] The accuracy of the check model when the check model is generated by the check model generation unit 11 as described above will be described with reference to FIGS. 3 and 4. First, the training data columns in FIGS. 3 and 4 indicate the true probability that the training data corresponds to each vegetable classification, i.e., each label. In the example of FIG. 3, if the predicted label for input data T2 of the training data in the operational model is "tomato," the learning weight for the label "tomato" is set high for the input data T2 of the training data, and the check model is trained. Therefore, in this embodiment, the accuracy of the probability of predicting each label by the check model can be improved. On the other hand, if the check model is trained using a loss function based on the above-described formula (6), which is different from this embodiment, learning is performed equally for the predicted label by the operational model and other labels, which may result in a lower accuracy of the probability of predicting each label by the check model.
[0029] 4 shows a case where the predicted label for input data T2 of the training data in the operational model is "carrot." In this case, for input data T2 of the training data, the learning weight for the label "carrot" is set high and the check model is trained. Therefore, in this embodiment, it is possible to improve the accuracy of the probability of predicting each label by the check model. On the other hand, unlike this embodiment, when the check model is trained using the loss function according to Equation 6 above, learning is performed equally for the predicted label by the operational model and other labels, which may result in a lower accuracy of the probability of predicting each label by the check model.
[0030] Then, the check model generation unit 11 stores the check model generated by learning as described above in the model storage unit 16. Note that the check model generation unit 11 may generate one or more check models.
[0031] The performance evaluation unit 12 (evaluation unit) reads the operation model and check model stored in the model storage unit 16 and the estimation target data stored in the estimation target data storage unit 18, and uses these to evaluate the predictive performance of the operation model. Specifically, the performance evaluation unit 12 inputs estimation target data to which no correct answer label has been assigned to the operation model and the check model, respectively, obtains predicted labels output from the operation model, and obtains the predicted probability of each label output from the check model. The performance evaluation unit 12 then evaluates the operation model based on the predicted probability of the label output from the check model corresponding to the predicted label output from the operation model. For example, a statistical value such as the average value of the predicted probabilities of the labels output from the check labels corresponding to the predicted labels output from the operation model is calculated, and this is used as an accuracy estimation value to evaluate the operation model. In this case, the higher the accuracy evaluation value, the higher the predictive performance of the operation model is evaluated to be.
[0032] Here, an example of calculation of the evaluation value by the performance evaluation unit 12 will be described with reference to FIG. 5. In FIG. 5, an example of the classification of vegetables is represented as classes 1, 2, 3, and 4. At this time, the performance evaluation unit 12 calculates the evaluation value by inputting "x 1 For ", the operational model outputs the predicted label "Class 3", so the prediction probability of Class 3 by the check model is "60%". Then, the performance evaluation unit 12 uses a plurality of estimation target data (x 1 ~x n ) and calculates the statistical value of the multiple prediction probabilities used as the accuracy estimate. At this time, the performance evaluation unit 12 may display and output the calculated accuracy evaluation value on a display of an information processing device operated by an evaluator who evaluates the operation model, as shown in FIG.
[0033] [Operation] Next, the operation of the above-mentioned model evaluation device 10 will be described with reference to the flowchart in Fig. 6. It is assumed that the model evaluation device 10 stores the above-mentioned operation model to be evaluated, the training data used to generate the operation model, and the estimation target data to be used to evaluate the operation model.
[0034] First, the model evaluation device 10 generates a check model by machine learning using training data to which correct labels have been assigned (step S1). When generating the check model using the training data, the model evaluation device 10 also inputs the training data into the operational model, and generates the check model using the training data based on the predicted labels output from the operational model. Specifically, the model evaluation device 10 trains the check model by changing the learning weights for each label in the training data according to the predicted labels output from the operational model. In particular, the model evaluation device 10 trains the check model by using the loss function shown in Equation 1 above, setting the loss weights for the training data corresponding to the predicted labels output from the operational model higher than the weights for other labels.
[0035] In this way, the model evaluation device 10 generates a check model by concentrating on learning the predicted label of the operation model while also learning other labels, which can improve the accuracy of the probability of predicting each label in the learned check model.
[0036] Next, the model evaluation device 10 inputs estimation target data to which no correct answer label has been assigned to the operation model and the check model (step S2). The model evaluation device 10 then obtains the predicted labels output from the operation model and the predicted probabilities of each label output from the check model, and evaluates the operation model based on these (step S3). Specifically, the model evaluation device 10 calculates statistical values, such as the average value of the predicted probabilities of the labels output from the check labels corresponding to the predicted labels output from the operation model, and uses these as accuracy estimates to evaluate the operation model. The model evaluation device 10 then outputs the calculated accuracy estimates, as shown in FIG. 5, by displaying them on the display of an information processing device operated by an evaluator who evaluates the operation model (step S4). At this time, the model evaluation device 10 may classify the evaluation values into predetermined ranks and display the corresponding ranks.
[0037] <Embodiment 2> Next, a second embodiment of the present disclosure will be described with reference to Fig. 7 and Fig. 8. Fig. 7 and Fig. 8 are block diagrams showing the configuration of a model evaluation device in embodiment 2. Note that this embodiment shows an outline of the configuration of the model evaluation device described in the above-mentioned embodiment.
[0038] First, the hardware configuration of the model evaluation device 100 in this embodiment will be described with reference to Fig. 7. The model evaluation device 100 is configured as a general information processing device, and is equipped with the following hardware configuration, for example: - CPU (Central Processing Unit) 101 (arithmetic unit) - ROM (Read Only Memory) 102 (storage device) - RAM (Random Access Memory) 103 (storage device) - Programs 104 loaded into the RAM 103 - Storage device 105 storing the programs 104 - Drive device 106 for reading and writing from / to a storage medium 110 external to the information processing device - Communication interface 107 for connecting to a communication network 111 external to the information processing device - Input / output interface 108 for inputting and outputting data - Bus 109 for connecting the various components
[0039] 7 shows an example of the hardware configuration of the information processing device that is the model evaluation device 100, and the hardware configuration of the information processing device is not limited to the above-described case. For example, the information processing device may be configured with a part of the above-described configuration, such as not including the drive device 106. Furthermore, instead of the above-described CPU, the information processing device may use a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point Number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.
[0040] The model evaluation device 100 can be equipped with a learning unit 121 and an evaluation unit 122 shown in FIG. 8 by having the CPU 101 acquire and execute the program group 104. The program group 104 is stored in advance in the storage device 105 or the ROM 102, for example, and is loaded into the RAM 103 and executed by the CPU 101 as needed. The program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in advance in the storage medium 110, and the drive device 106 may read out the programs and supply them to the CPU 101. However, the learning unit 121 and evaluation unit 122 described above may be constructed using dedicated electronic circuits for realizing such means.
[0041] The learning unit 121 learns a second learning model that outputs a probability of prediction for input training data based on a prediction output from a first learning model for input training data. At this time, the learning unit 121 learns the second learning model so that the second learning model outputs a probability of prediction for each of multiple classifications for input training data based on a classification that is a prediction output from the first learning model for input training data among multiple preset classifications. Furthermore, the learning unit 121 learns the second learning model by changing the learning weight for each classification of the training data used for learning the second learning model, depending on the classification that is a prediction output from the first learning model for input training data.
[0042] The evaluation unit 122 evaluates the first learning model based on the prediction output from the first learning model for the input of specified input data and the probability of the prediction output from the second learning model for the input of the specified input data.
[0043] With the above configuration, the present disclosure trains the second learning model based on the prediction results output from the first learning model for input training data, thereby improving the accuracy of the output from the second learning model. Furthermore, by evaluating the first learning model using the second learning model, the first learning model can be appropriately evaluated quickly and at low cost.
[0044] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.
[0045] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, at least one or more of the functions of the learning unit 121 and the evaluation unit 122 described above may be executed by an information processing device installed and connected anywhere on a network, that is, may be executed by so-called cloud computing.
[0046] <Supplementary Notes> Some or all of the above embodiments may also be described as in the following supplementary notes. Below, an outline of the configurations of a model evaluation device, a model evaluation method, and a program according to the present disclosure will be described. However, the present disclosure is not limited to the following configurations. (Supplementary Note 1) A model evaluation device comprising: a learning unit that learns a second learning model that outputs a probability of a prediction for input of training data based on a prediction output from the first learning model for input of the training data; and an evaluation unit that evaluates the first learning model based on a prediction output from the first learning model for input of predetermined input data and the probability of the prediction output from the second learning model for input of the predetermined input data. (Supplementary Note 2) The model evaluation device according to Supplementary Note 1, wherein the learning unit learns the second learning model based on a classification that is a prediction output from the first learning model for input of the training data among a plurality of preset classifications, such that the second learning model outputs a probability of prediction for each of a plurality of classifications for input of the training data. (Supplementary Note 3) The model evaluation device according to Supplementary Note 2, wherein the learning unit learns the second learning model by changing a learning weight for each classification of the training data used for learning the second learning model in accordance with a classification that is a prediction output from the first learning model for the input of the training data. (Supplementary Note 4) The model evaluation device according to Supplementary Note 3, wherein, when a specific classification is output as a prediction from the first learning model for the input of the training data, the learning unit learns the second learning model by increasing the learning weight for the specific classification of the training data used for learning the second learning model compared to other classifications. (Supplementary Note 5) The model evaluation device according to Supplementary Note 4, wherein the learning unit learns the second learning model by increasing the weight of the loss during learning for the specific classification of the training data used for learning the second learning model compared to other classifications.(Supplementary Note 6) The model evaluation device according to Supplementary Note 1, wherein the evaluation unit evaluates the first learning model based on the probability of a prediction output from the second learning model for input of specified input data, which corresponds to the prediction output from the first learning model for input of the specified input data. (Supplementary Note 7) The model evaluation device according to Supplementary Note 6, wherein the evaluation unit evaluates the first learning model based on the probability of each prediction output from the second learning model for input of multiple specified input data, which corresponds to each prediction output from the first learning model for input of the multiple specified input data. (Supplementary Note 8) A model evaluation method comprising: based on the predictions output from the first learning model for input of training data, training a second learning model that outputs a probability of a prediction for input of the training data; and evaluating the first learning model based on the predictions output from the first learning model for input of specified input data and the probability of the predictions output from the second learning model for input of the specified input data. (Supplementary Note 9) The model evaluation method according to Supplementary Note 8, wherein the second learning model is trained so that the second learning model outputs a probability of prediction for each of a plurality of classifications for the input of the training data, based on a classification that is a prediction output from the first learning model for the input of the training data from among a plurality of predetermined classifications. (Supplementary Note 10) The model evaluation method according to Supplementary Note 9, wherein the second learning model is trained by changing a learning weight for each classification of the training data used for training the second learning model, according to the classification that is a prediction output from the first learning model for the input of the training data. (Supplementary Note 11) The model evaluation method according to Supplementary Note 10, wherein, when the first learning model outputs a specific classification as a prediction for the input of the training data, the second learning model is trained by increasing the learning weight for the specific classification of the training data used for training the second learning model compared to other classifications.(Appendix 12) A computer-readable storage medium storing a program for causing a computer to execute the following process: based on the predictions output from a first learning model for input of training data, learn a second learning model that outputs the probability of a prediction for input of the training data; and evaluate the first learning model based on the predictions output from the first learning model for input of specified input data and the probability of the predictions output from the second learning model for input of the specified input data.
[0047] REFERENCE SIGNS LIST 10 Model evaluation device 11 Check model generation unit 12 Performance evaluation unit 16 Model storage unit 17 Training data storage unit 18 Estimation target data storage unit 100 Model evaluation device 101 CPU 102 ROM 103 RAM 104 Program group 105 Storage device 106 Drive device 107 Communication interface 108 Input / output interface 109 Bus 110 Storage medium 111 Communication network 121 Learning unit 122 Evaluation unit
Claims
1. A model evaluation device comprising: a learning unit that learns a second learning model that outputs a probability of a prediction for input of training data based on a prediction output from a first learning model for input of the training data; and an evaluation unit that evaluates the first learning model based on the prediction output from the first learning model for input of specified input data and the probability of the prediction output from the second learning model for input of the specified input data.
2. A model evaluation device as described in claim 1, wherein the learning unit learns the second learning model so that the second learning model outputs a probability of prediction for each of a plurality of classifications for the input of the training data based on a classification that is a prediction output from the first learning model for the input of the training data among a plurality of predetermined classifications.
3. A model evaluation device as described in claim 2, wherein the learning unit learns the second learning model by changing the learning weight for each classification of the training data used for learning the second learning model in accordance with the classification, which is a prediction output from the first learning model in response to the input of the training data.
4. A model evaluation device as described in claim 3, wherein when a specific classification is output as a prediction from the first learning model in response to the input of the training data, the learning unit trains the second learning model by giving a higher learning weight to the specific classification of the training data used for training the second learning model than to other classifications.
5. A model evaluation device as described in claim 4, wherein the learning unit learns the second learning model by weighting the loss during learning for the specific classification of the training data used to learn the second learning model higher than for other classifications.
6. A model evaluation device as described in claim 1, wherein the evaluation unit evaluates the first learning model based on the probability of a prediction output from the second learning model for input of the specified input data, which corresponds to the prediction output from the first learning model for input of the specified input data.
7. A model evaluation device as described in claim 6, wherein the evaluation unit evaluates the first learning model based on the probability of each prediction output from the second learning model for each of a plurality of specified input data, which corresponds to each prediction output from the first learning model for each of the plurality of specified input data.
8. A model evaluation method comprising: learning a second learning model that outputs a probability of a prediction for input of training data based on a prediction output from a first learning model for input of the training data; and evaluating the first learning model based on the prediction output from the first learning model for input of specified input data and the probability of the prediction output from the second learning model for input of the specified input data.
9. A model evaluation method as described in claim 8, comprising: learning the second learning model so that the second learning model outputs a probability of prediction for each of a plurality of classifications for the input of the training data based on a classification that is a prediction output from the first learning model for the input of the training data among a plurality of predetermined classifications.
10. A model evaluation method as described in claim 9, comprising: learning the second learning model by changing the learning weight for each classification of the training data used for training the second learning model in accordance with the classification, which is a prediction output from the first learning model in response to the input of the training data.
11. A model evaluation method as described in claim 10, wherein, when a specific classification is output as a prediction from the first learning model in response to the input of the training data, the second learning model is trained by giving a higher learning weight to the specific classification of the training data used for training the second learning model than to other classifications.
12. A computer-readable storage medium storing a program for causing a computer to execute a process of: learning a second learning model that outputs a probability of a prediction for input of training data based on a prediction output from a first learning model for input of the training data; and evaluating the first learning model based on the prediction output from the first learning model for input of specified input data and the probability of the prediction output from the second learning model for input of the specified input data.
Citation Information
Patent Citations
Information processor, information processing method, and program
JP2023008028A
Self-training technique for generating neural network models
US20220083840A1
Evaluation program, evaluation method, and accuracy evaluation device
WO2023013024A1
