Model evaluation apparatus, method, and program

Abstract

To provide a model evaluating technology capable of executing a different evaluation from that of a background technology.SOLUTION: An estimating model learning unit 1 learns an estimating model in such a way that an index representing a matching probability between a response corresponding to each situation obtained from the estimating model and a response corresponding to each situation of a system becomes close to an index representing the matching probability between a first response corresponding to each situation obtained from the estimating model and a second response. An evaluation value calculating unit 2 is configured to: set, as a first value, the value of an index representing the matching probability between the first response corresponding to each situation of the system and the second response; set, as a second value, the value of an index (first index) representing the matching probability between the response corresponding to each situation obtained from the estimating model and the response corresponding to each situation of the system, the value of an index (second index) representing the matching probability between the first response corresponding to each situation obtained from the estimating model and the second response, or an average value of the first index and the second index; and calculate a difference between the first value and the second value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosure concerns techniques for evaluating models. [Background technology]

[0002] A technique called Confidence Calibration is known for evaluating models that estimate the discrete response of a system containing uncertainty as a probability distribution (see, for example, Non-Patent Document 1).

[0003] Confidence Calibration is a method for matching the machine accuracy of a model with the machine confidence level of that model. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, "On Calibration of Modern Neural Networks", ICML'17: Proceedings of the 34th International Conference on Machine Learning, Volume 70, pp.1321-1330, 2017. [Non-Patent Document 2] Li HH., Ma WJ, "Confidence reports in decision-making with multiple alternatives violate the Bayesian confidence hypothesis", Nat Commun 11, 2004, 2020. [Non-Patent Document 3] Shiro Kumano, Ryo Ishii, Kazuhiro Otsuka, "Computational Model of Idiosyncratic Perception of Others' Emotions", In Proc. Int'l Conf. Affective Computing and Intelligent Interaction (ACII 2017), pp. 42-49, 2017. [Non-Patent Document 4] Samejima F., "Estimation of Latent Ability Using a Response Pattern of Graded Scores", Psychometric Monograph No. 17, Richmond, VA: Psychometric Society, 1969. Summary of the Invention Problems to be Solved by the Invention

[0005] However, in the method described in the background art, since the upper limit of the evaluation value of the model is unknown, there has been a problem that it is impossible to know how much room for improvement the model still has. Additionally, in the method described in the background art, there has been a problem that the goodness of estimation of the entire distribution cannot be evaluated.

[0006] An object of the disclosed technology is to provide a model evaluation apparatus, method, and program that enable evaluation different from that of the background art. Means for Solving the Problems

[0007] One aspect of the disclosed technology includes an estimation model learning unit that learns an estimation model using the learning data, assuming that responses corresponding to each situation of the system are obtained as learning data, and that learns the estimation model so that an index indicating the probability that the responses corresponding to each situation obtained from the estimation model, which is a model obtained by learning, match the responses corresponding to each situation of the system, and an index indicating the probability that the first response and the second response corresponding to each situation obtained from the estimation model match, become close to each other; and an evaluation value calculation unit that takes the value of the index indicating the probability that the first response and the second response corresponding to each situation of the system match as the first value, and takes the value of the index indicating the probability that the responses corresponding to each situation obtained from the estimation model match the responses corresponding to each situation of the system (hereinafter referred to as the first index), the value of the index indicating the probability that the first response and the second response corresponding to each situation obtained from the estimation model match (hereinafter referred to as the second index), or the average of the first and second indexes as the second value, and takes the difference between the first and second values ​​as the evaluation value of the estimation model.

[0008] One aspect of the disclosed technology includes an estimation model learning unit that, assuming that responses corresponding to each situation of the system are obtained as training data, uses the training data to learn the estimation model such that an index indicating the probability that the responses corresponding to each situation obtained from the estimation model, which is a model obtained through learning, match the responses corresponding to each situation of the system, and an index indicating the probability that the first response and the second response corresponding to each situation obtained from the estimation model match, become close to each other. [Effects of the Invention]

[0009] Disclosure technology allows for evaluations that differ from those of the background technology. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows an example of the functional configuration of a model evaluation device. [Figure 2] Figure 2 shows an example of the processing procedure for a model evaluation method. [Figure 3]Figure 3 is a diagram illustrating an experimental example. [Figure 4] Figure 4 is a diagram illustrating an experimental example. [Figure 5] Figure 5 shows an example of a computer's functional configuration. [Modes for carrying out the invention]

[0011] Embodiments of the disclosed technology will be described below with reference to the drawings. Note that components having the same function are numbered identically in the drawings, and redundant explanations are omitted.

[0012] [Model evaluation apparatus and method] As shown in Figure 1, the model evaluation device includes, for example, an estimated model learning unit 1 and an evaluation value calculation unit 2.

[0013] The model evaluation method can be implemented, for example, by having each component of the model evaluation device perform the processing shown in steps S1 and S2 in Figure 2.

[0014] The following describes each component of the model evaluation device.

[0015] <Estimation Model Learning Unit 1> Training data is input to the estimation model learning unit 1.

[0016] The training data consists of responses corresponding to each state of the system. The response corresponding to each state m is y. (m) Let ∈{1,…,C}. M is the total number of situations, m is the index representing a situation, and m∈{1,…,M}. C is the total number of responses, c is the index representing a response, and c∈{1,…,C}. The training data includes the first response y corresponding to the same situation m. (m) and the second response y' (m) It may include the first response y. (m) and the second response y' (m) Let M' be the set of situations m for which a certain condition has been obtained. M' is all or part of the M situations.

[0017] The estimation model learning unit 1 uses input training data to train an estimation model, which is a model obtained through training, such that an index indicating the probability that a response corresponding to each situation obtained from the estimation model matches a response corresponding to each situation of the system and an index indicating the probability that a first response and a second response corresponding to each situation obtained from the estimation model match are close to each other (step S1).

[0018] The estimation model obtained through training is output to the evaluation value calculation unit 2.

[0019] An example of the index indicating the probability that a response corresponding to each situation obtained from the estimation model matches a response corresponding to each situation of the system is CCP shown below. This index is calculated using the training data and the estimation model.

Mathematics

[0020] Note that the "index indicating probability" may be the probability itself, or may be a value that increases in accordance with the probability.

[0021] An example of the index indicating the probability that a first response and a second response corresponding to each situation obtained from the estimation model match is PCP shown below. This index is calculated using the estimation model.

number

number

[0022] The background technology, Confidence Calibration, only looks at the peak (summit) of q (an index representing probability) and does not provide an index that considers the shape of the entire distribution of q in C dimensions. In contrast, the index calculated by the estimation model learning unit 1 is influenced by the entire distribution, and therefore can evaluate the goodness of the estimation of the entire distribution. For this reason, the index calculated by the estimation model learning unit 1 allows for a different evaluation from the background technology in that it can evaluate the goodness of the estimation of the entire distribution.

[0023] The metrics calculated by the estimation model learning unit 1 may also be output from the estimation model learning unit 1.

[0024] The estimation model learning unit 1 may also learn the model to minimize the following cost function L. Here, N is the total number of systems to be simulated, n is the index representing a system, and n ∈ {1, ..., N}. For example, the model is learned in this way when we want to create a system model that integrates systems simulating the responses corresponding to each situation of multiple people (N people).

number

[0025] y c (mn) is, y (mn) This is the c-th element of the one-hot vector corresponding to y. (mn) The corresponding one-hot vector is y (mn) If = d, then the vector has 1 as its d-th element and 0 for all other elements.

[0026] CCP (n) This is an example of an index that shows the probability that the response corresponding to each situation obtained from the estimation model matches the response corresponding to each situation in the system, as shown below. This index is calculated using the training data and the estimation model.

number

[0027] An example of an index that shows the probability that the first and second responses corresponding to each situation obtained from the estimation model match is the PCP shown below. (n) This index is calculated using an estimation model.

number

[0028] In this case, the estimation model learning unit 1 first learns the model to minimize a cost function consisting only of the first term on the right-hand side of equations (1) and (2) above. That is, the estimation model learning unit 1 generates a provisional estimation model by learning the model to minimize a cost function L defined by, for example, one of the following equations.

number

number

[0029] For information on Temperature Scaling (TS), see, for example, Non-Patent Document 1.

[0030] For example, the output of the hypothetical estimation model is q=(q1,…,q C ) and the corrected output is q'=(q1',…,q C Let ') be the softmax function, let T be the temperature in Temperature Scaling (TS), and q c '=σ(q c As / T), the estimation model learning unit 1 outputs the corrected output q'=(q1',…,q CThe temperature T is estimated such that the CCP and PCP determined by ') satisfy CCP=PCP. Then, the estimation model learning unit 1 uses the estimated temperature T to q c '=σ(q c A provisional estimation model corrected by / T) may be obtained as the final estimation model.

[0031] <Evaluation Value Calculation Unit 2> The evaluation value calculation unit 2 receives the training data and the estimated model trained by the estimated model learning unit 1 as input.

[0032] The evaluation value calculation unit 2 sets the first value as the value of an index that indicates the probability that the first response and the second response corresponding to each situation of the system will match, the second value as the value of an index that indicates the probability that the response corresponding to each situation obtained from the estimation model will match the response corresponding to each situation of the system (hereinafter referred to as the first index), the second value as an index that indicates the probability that the first response and the second response corresponding to each situation obtained from the estimation model will match (hereinafter referred to as the second index), or the average of the first and second indexes, and sets the value based on the difference between the first and second values ​​as the evaluation value of the estimation model (step S2).

[0033] An example of a value based on the difference between the first and second values ​​is the value obtained by subtracting the second value from the first value.

[0034] An example of a metric that shows the probability that the first and second responses corresponding to each state of the system match is TCP, shown below. This metric is calculated using training data. (y (m) =y' (m) ) is y (m) =y' (m) If so, then it is 1, and y (m) =y' (m) Otherwise, it is 0. |M'| is the number of elements in set M'.

number

[0035] An example of an index (second index) that shows the probability that the first response and the second response corresponding to each situation obtained from the estimation model will match is the PCP or PCP shown earlier. (n) This index is calculated using an estimation model.

[0036] The processing of the estimation model learning unit 1 is performed so that the index (first index) which shows the probability that the response corresponding to each situation obtained from the estimation model matches the response corresponding to each situation of the system matches, and the index (second index) which shows the probability that the first response and the second response corresponding to each situation obtained from the estimation model match, are brought closer together. For this reason, the evaluation value calculation unit 2 may use the value of the first index, the value of the second index, or the average value of the first and second indices as the second value.

[0037] An index (e.g., TCP) that shows the probability that the first and second responses corresponding to each state of the system will match can be considered the true collision probability derived from the true distribution.

[0038] Therefore, when the estimation model is trained so that an index indicating the probability that the response corresponding to each situation obtained from the estimation model matches the response corresponding to each situation of the system (e.g., CCP) and an index indicating the probability that the first response corresponding to each situation obtained from the estimation model matches the second response (e.g., PCP) are close together, the difference between the first and second values ​​(e.g., the TCP value minus the CCP value or PCP value) becomes the squared error between the true distribution and the distribution obtained from the estimation model. Since it is a squared error, the upper limit of the estimation model's performance is reached when the difference between the first and second values ​​equals 0.

[0039] Therefore, by obtaining an evaluation value for the estimation model based on the difference between the first and second values, it becomes possible to determine how much further improvement there is in the model (estimation model). In this respect, a different evaluation from the background technology becomes possible.

[0040] <Example of experiment> Fifty Japanese students (equal numbers of males and females) were presented with 30 randomly selected CG facial images from a set of 120 images, each for one second. They were asked to rate (i) emotional valence (unpleasant, 1 to pleasant, 5) and (ii) arousal level (calming, 1 to excited, 5) twice. Using these ratings as training data, a first estimation model (a graded response model) was trained using the disclosed technique, where the presentation of the CG facial images was the "situation," the Japanese students were the "system," and emotional valence and arousal level were the "responses." M=120, |M'|=30, and N=50. The Hamiltonian Monte Carlo algorithm was used as the training algorithm. This first estimation model can also be considered a true distribution.

[0041] From the first estimation model, random rating values ​​are calculated, and from these calculated rating values, the Adam optimalizer in PyTourch is determined with a learning rate of 2e. -4 A second estimation model was trained using the disclosed technique with a mini-batch size of 6000.

[0042] Figures 3(a) and 3(b) show the relationship between the number of iterations in the training of the second estimation model and the values ​​of MA, CCP, PCP, MC, and TCP. Figure 3(a) shows this relationship in emotional valence. Figure 3(b) shows this relationship in arousal. In Figures 3(a) and 3(b), the horizontal axis represents the number of iterations in the training of the second estimation model, and the vertical axis represents the values ​​of MA, CCP, PCP, MC, and TCP.

[0043] MA stands for Machine Accuracy, and MC stands for Machine Confidence.

[0044] Figures 3(a) and 3(b) show that the relationship CCP=PCP is nearly satisfied for both emotional valence and arousal. As a byproduct, the relationship MA=MC is also nearly satisfied.

[0045] Furthermore, when a portion of the 120 total situations were randomly selected and the second estimation model was trained in the same manner (10 trials), the relationship between the number of randomly selected situations and the difference (error) between ε directly calculated from the first estimation model (considered to be the true distribution) and ε calculated using the second estimation model is shown in Figures 4(a) and 4(b). Figure 4(a) shows this relationship in emotional valence. Figure 4(b) shows this relationship in arousal. In Figures 4(a) and 4(b), the horizontal axis represents the number of randomly selected situations, and the vertical axis represents the error.

[0046] Figures 4(a) and 4(b) show that the error in ε is almost zero when the number of situations is around 120. In other words, if there is a sufficient number of situations, the disclosed technique can be used to obtain results that are in line with theory.

[0047] [Differentiation] The specific configuration of the embodiments of the disclosed technology is not limited to the configuration described above. The specific configuration of the embodiments of the disclosed technology can be modified as appropriate, without departing from the spirit of the embodiments of the disclosed technology.

[0048] The various processes described in the embodiments of the disclosed technology may be performed not only in chronological order according to the order described, but also in parallel or individually as required by the processing capacity of the device performing the processes.

[0049] For example, data exchange between components of the model evaluation device may be performed directly, or it may be performed via a storage unit (not shown).

[0050] [Programs, recording media] The processing of each part of the above-mentioned devices may be implemented by a computer. In this case, the processing content of the functions that each device should have is described by a program. This program is then loaded into the memory unit 1020 of the computer 1000 shown in Figure 5, and the arithmetic processing unit 1010, input unit 1030, output unit 1040, display unit 1060, etc. are operated, thereby realizing the various processing functions of each of the above-mentioned devices on the computer.

[0051] The program describing this process can be recorded on a computer-readable recording medium. Computer-readable recording media are, for example, non-temporary recording media, specifically magnetic recording devices, optical discs, etc.

[0052] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0053] A computer executing such a program first stores the program recorded on a portable recording medium or transferred from a server computer in its own non-temporary storage device, the auxiliary recording unit 1050. Then, when processing is to be executed, the computer reads the program stored in the auxiliary recording unit 1050 into the storage unit 1020 and executes the processing according to the loaded program. Alternatively, the computer may directly read the program from the portable recording medium into the storage unit 1020 and execute the processing according to that program. Furthermore, each time a program is transferred to this computer from a server computer, it may sequentially execute the processing according to the received program. Alternatively, the above processing may be executed by a so-called ASP (Application Service Provider) type service, where the server computer does not transfer programs to this computer, but the processing function is realized only by execution instructions and result acquisition. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0054] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware. For example, the estimation model learning unit 1 and the evaluation value calculation unit 2 may be configured by processing circuits.

[0055] It goes without saying that the invention may be modified as appropriate without departing from its spirit.

[0056] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually described as being incorporated by reference.

Claims

1. Assuming that responses corresponding to each state of the system are obtained as training data, the estimation model learning unit learns the estimation model using the training data such that an index indicating the probability that the responses corresponding to each state obtained from the estimation model, which is a model obtained through learning, match the responses corresponding to each state of the system, and an index indicating the probability that the first response and the second response corresponding to each state obtained from the estimation model match, become close to each other. An evaluation value calculation unit takes the value of an index that indicates the probability that the first response and the second response corresponding to each situation of the system will match as the first value, the value of an index that indicates the probability that the response corresponding to each situation obtained from the estimation model will match the response of the system corresponding to each situation (hereinafter referred to as the first index), the value of an index that indicates the probability that the first response and the second response corresponding to each situation obtained from the estimation model will match (hereinafter referred to as the second index), or the average of the first index and the second index as the second value, and uses the difference between the first value and the second value as the evaluation value of the estimation model. A model evaluation device that includes this.

2. A model evaluation apparatus according to claim 1, The estimation model learning unit generates a provisional estimation model using the training data, and learns the estimation model by correcting the output of the provisional estimation model so that an index indicating the probability that the response corresponding to each situation obtained from the estimation model matches the response of the system corresponding to each situation, and an index indicating the probability that the first response and the second response corresponding to each situation obtained from the estimation model match, become close to each other. Model evaluation device.

3. Assuming that responses corresponding to each state of the system are obtained as training data, the estimation model learning unit learns the estimation model using the training data such that an index indicating the probability that the responses corresponding to each state obtained from the estimation model, which is a model obtained by learning, match the responses corresponding to each state of the system, and an index indicating the probability that the first response and the second response corresponding to each state obtained from the estimation model match, become close to each other. A model evaluation device that includes this.

4. The estimation model learning unit assumes that responses corresponding to each situation of the system have been obtained as training data, and uses the training data to learn the estimation model such that an index indicating the probability that the responses corresponding to each situation obtained from the estimation model obtained through learning match the responses corresponding to each situation of the system, and an index indicating the probability that the first response and the second response corresponding to each situation obtained from the estimation model match, become close together. The evaluation value calculation unit takes the value of an index that indicates the probability that the first response and the second response corresponding to each situation of the system will match as the first value, the value of an index that indicates the probability that the response corresponding to each situation obtained from the estimation model will match the response of the system corresponding to each situation (hereinafter referred to as the first index), the value of an index that indicates the probability that the first response and the second response corresponding to each situation obtained from the estimation model will match (hereinafter referred to as the second index), or the average of the first index and the second index as the second value, and takes the value based on the difference between the first value and the second value as the evaluation value of the estimation model. A model evaluation method that includes this.

5. The estimation model learning unit assumes that responses corresponding to each situation of the system have been obtained as training data, and uses the training data to learn the estimation model such that an index indicating the probability that the responses corresponding to each situation obtained from the estimation model obtained through learning match the responses corresponding to each situation of the system, and an index indicating the probability that the first response and the second response corresponding to each situation obtained from the estimation model match, become close together. A model evaluation method that includes this.

6. A program for causing a computer to function as a component of any of the model evaluation devices according to claims 1 to 3.

Images