Learning device, learning method, and program

Abstract

To provide a technology capable of obtaining a model with high interpretability and accuracy, while suppressing human cost.SOLUTION: A learning apparatus includes: an input unit which acquires first input data and a ground truth value of the first input data; an inference unit which outputs first inference values corresponding to each of a plurality of inference models, on the basis of the first input data and the inference models; an explanation unit which outputs first explanatory information corresponding to each of the inference models that indicates a degree of contribution of the first input data to the first inference values; an inference evaluation unit which obtains an inference evaluation result on the basis of the ground truth value and the first inference value; an explanation evaluation unit which obtains an explanation evaluation result on the basis of a matching degree between the pieces of first explanatory information corresponding to each of the inference models; and an update unit which updates a first weight parameter of the inference models, on the basis of the inference evaluation result and the explanation evaluation result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning device, a learning method, and a program. [Background technology]

[0002] Neural networks (hereafter referred to as "NN") have high performance in areas such as image recognition. However, NNs are generally composed of a huge number of parameters and complex models, making it difficult to interpret the relationship between the NN parameters and the output results from the NN. To solve this problem, several methods have been proposed to obtain NNs with high interpretability. Note that "high interpretability" can also be expressed as "high degree of agreement with human senses."

[0003] For example, a method is known in which a heat map indicating areas that an NN model should pay attention to for making a decision is manually labeled, and the model is trained to match the heat map, thereby obtaining a model that is easy for humans to interpret (see, for example, Non-Patent Document 1). Also, if the interpretability of the heat map obtained from the model is low, a model with higher interpretability can be obtained by retraining the model so that it does not match the heat map.

[0004] Also, a method is known for improving the accuracy of a NN by introducing into the network a mechanism for extracting regions of interest from input data that the NN uses to make decisions (see, for example, Non-Patent Document 2). The regions of interest obtained by such a method are manually corrected, and the NN is retrained so that the corrected regions of interest match the regions of interest of the NN, thereby improving the interpretability and accuracy of the NN. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Andrew Ross and two others, "Right for the Right Reasons: Training Differentiable Models by Constraining Their Explanations," [online], [Retrieved September 8, 2021], Internet<https: / / arxiv.org / abs / 1703.03717> [Non-patent document 2] Masahiro Mitsuhara and 6 others, "Embedding Human Knowledge into Deep Neural Network via Attention Map," [online], [Retrieved September 8, 2021], Internet<https: / / arxiv.org / abs / 1905.03540> [Non-patent document 3] "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization", [online], [Retrieved September 8, 2021], Internet<https: / / arxiv.org / abs / 1610.02391v3> Summary of the Invention [Problem to be solved by the invention]

[0006] However, the methods described in Non-Patent Documents 1 and 2, in which labels for heat maps are prepared manually, require a large amount of human labor for labeling.

[0007] On the other hand, a method that does not require labeling is to retrain a model so that it does not match the heat map of a trained model, as described in Non-Patent Document 1. However, this method has a problem in that the retraining is likely to result in a decrease in accuracy. Furthermore, this method has a problem in that the degree of match of the heat map decreases uniformly for all data, which may actually reduce the interpretability of the heat map for individual data.

[0008] Therefore, it is desirable to provide a technology that can obtain a highly interpretable and accurate model while reducing human costs. [Means for solving the problem]

[0009] In order to solve the above problem, according to one aspect of the present invention, there is provided an input unit that acquires first input data and a correct value of the first input data; an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and a plurality of inference models; an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models that indicates the magnitude of contribution of the first input data to the first inference value; an inference evaluation unit that obtains an inference evaluation result based on the correct value and the first inference value; an explanation evaluation unit that obtains an explanation evaluation result based on the degree of agreement between the first explanation information corresponding to each of the plurality of inference models; and so that the degree of coincidence between the first explanatory information pieces is small, an update unit that updates first weight parameters of the plurality of inference models, the input unit acquires second input data; the inference unit outputs a second inferred value corresponding to each of the plurality of trained models based on the second input data and a plurality of trained models that are a plurality of estimation models after updating the first weight parameter; the explanation unit outputs second explanation information corresponding to each of the plurality of trained models that indicates a magnitude of contribution of the second input data to the second inferred value; and the learning device comprises: a presentation control unit that controls presentation of the second inferred value and the second explanation information to a user; and an operation unit that receives, from the user, selected model information that indicates one or more trained models selected by the user from the plurality of trained models. is provided.

[0012] The learning device may include a recording control unit that controls recording of information indicating one or more trained models selected by the user from the plurality of trained models.

[0013] The explanation evaluation result may take a smaller value as the degree of agreement between the first explanation information corresponding to each of the plurality of inference models increases.

[0014] The explanation evaluation unit may obtain the explanation evaluation result based on an inner product of vectors normalized from first explanation information corresponding to each of the plurality of inference models.

[0015] The explanation evaluation unit may binarize the first explanation information to generate a mask for each of the multiple inference models, calculate the product of the mask generated from the first explanation information corresponding to an inference model other than itself and the first explanation information corresponding to its own inference model, and obtain the explanation evaluation result based on the sum of the products for each of the multiple inference models.

[0016] The explanation section may include a function capable of backpropagation.

[0017] The explanation unit may have a second weight parameter, and the update unit may update the second weight parameter by backpropagation.

[0018] At least one of the plurality of inference models may include a neural network. Note that a neural network is merely one example of a machine learning algorithm. Therefore, other machine learning algorithms may be used instead of a neural network.

[0019] The update unit may update the first weight parameter based on a result of adding the inference evaluation result and the explanation evaluation result.

[0020] The first explanatory information may be a heat map indicating the magnitude of the contribution of the first input data to the first inferred value.

[0021] According to another aspect of the present invention, a method for evaluating a variance of a first input data and a correct answer value of the first input data is provided. Based on the first input data and a plurality of inference models, a first inference value corresponding to each of the plurality of inference models is output. First explanation information corresponding to each of the plurality of inference models indicating the magnitude of contribution of the first input data to the first inference value is output. An inference evaluation result is obtained based on the correct answer value and the first inference value. An explanation evaluation result is obtained based on the degree of agreement between the first explanation information corresponding to each of the plurality of inference models. Based on the inference evaluation result and the explanation evaluation result, so that the degree of coincidence between the first explanatory information pieces is small, Updating first weight parameters of the plurality of inference models; acquiring second input data; outputting second inferred values ​​corresponding to each of the plurality of trained models based on the second input data and a plurality of trained models that are a plurality of estimation models after updating the first weight parameter; outputting second explanation information corresponding to each of the plurality of trained models indicating the magnitude of contribution of the second input data to the second inferred value; controlling presentation of the second inferred value and the second explanation information to a user; and receiving selected model information from the user indicating one or more trained models selected by the user from the plurality of trained models; A learning method is provided, including:

[0022] According to another aspect of the present invention, a computer includes an input unit that acquires first input data and a correct answer value of the first input data, an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and a plurality of inference models, an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models that indicates the magnitude of contribution of the first input data to the first inference value, an inference evaluation unit that obtains an inference evaluation result based on the correct answer value and the first inference value, an explanation evaluation unit that obtains an explanation evaluation result based on the degree of agreement between the first explanation information corresponding to each of the plurality of inference models, and so that the degree of coincidence between the first explanatory information pieces is small, an update unit that updates first weight parameters of the plurality of inference models, the input unit acquires second input data; the inference unit outputs a second inferred value corresponding to each of the plurality of trained models based on the second input data and a plurality of trained models that are a plurality of estimation models after updating the first weight parameter; the explanation unit outputs second explanation information corresponding to each of the plurality of trained models that indicates a magnitude of contribution of the second input data to the second inferred value; and the learning device comprises: a presentation control unit that controls presentation of the second inferred value and the second explanation information to a user; and an operation unit that receives, from the user, selected model information that indicates one or more trained models selected by the user from the plurality of trained models. A program is provided to function as a [Effects of the Invention]

[0023] As described above, the present invention provides a technique that makes it possible to obtain a model with high interpretability and accuracy while suppressing human costs. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device according to an embodiment of the present invention. [Figure 2] 10A and 10B are diagrams for explaining a method for obtaining explanation evaluation results by multiplying a mask obtained by binarizing a heat map with another heat map. [Figure 3] 10 is a flowchart illustrating an example of an operation in a learning stage of the learning device according to the embodiment. [Figure 4] 10 is a flowchart illustrating an example of operation of the learning device according to the embodiment in a test stage. [Figure 5] FIG. 1 is a diagram illustrating a hardware configuration of an information processing device as an example of a learning device. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant explanations will be omitted.

[0026] Furthermore, in this specification and drawings, multiple components having substantially the same functional configuration may be distinguished by adding different numbers after the same reference numeral. However, if there is no particular need to distinguish between multiple components having substantially the same functional configuration, only the same reference numeral will be used. Furthermore, similar components in different embodiments may be distinguished by adding different letters after the same reference numeral. However, if there is no particular need to distinguish between similar components in different embodiments, only the same reference numeral will be used.

[0027] (0. Overview of the embodiment) An overview of an embodiment of the present invention will be described. In the embodiment of the present invention, a learning device that performs neural network learning based on a combination of input data (learning data) and a correct answer value will be described. However, a neural network is merely one example of a machine learning algorithm. Therefore, other machine learning algorithms may be used instead of a neural network. For example, a support vector machine (SVM) may be used as another example of a machine learning algorithm.

[0028] (1. Details of the embodiment) An embodiment of the present invention will now be described in detail.

[0029] (1.1. Example of learning device configuration) 1 is a diagram showing an example of the functional configuration of a learning device 10 according to an embodiment of the present invention. As shown in Fig. 1, the learning device 10 according to an embodiment of the present invention includes an input unit 101, an inference unit 102, an explanation unit 103, an inference evaluation unit 104, an explanation evaluation unit 105, an update unit 106, a presentation control unit 107, a recording control unit 108, a display unit 121, and an operation unit 122.

[0030] In an embodiment of the present invention, it is mainly assumed that the inference unit 102 includes n inference models (n is an integer greater than 1), i.e., a "first inference model" through an "nth inference model." Furthermore, in an embodiment of the present invention, it is mainly assumed that each of the first to nth inference models includes a neural network. Hereinafter, a neural network will also be referred to as "NN."

[0031] The NNs included in each of the first to n-th inference models use weight parameters 110 (first weight parameters). In this case, the NNs included in each of the first to n-th inference models may have a common structure and use different weight parameters 110 (first weight parameters). Alternatively, the NNs included in each of the first to n-th inference models may have different structures.

[0032] At least one of the first to n-th inference models may include a neural network. For example, some of the first to n-th inference models may include a neural network, and other of the first to n-th inference models may include another machine learning algorithm instead of a neural network.

[0033] Furthermore, in the embodiment of the present invention, it is mainly assumed that the explanation unit 103 is configured to include a neural network. The neural network included in the explanation unit 103 uses weight parameters (second weight parameters).

[0034] The data set 100, the weight parameters 110 (first weight parameters) of the first to n-th inference models, and the weight parameters (second weight parameters) possessed by the explanation unit 103 are stored in a storage unit (not shown). Such a storage unit may be configured by a memory such as a RAM (Random Access Memory), a hard disk drive, or a flash memory.

[0035] The input unit 101, the inference unit 102, the explanation unit 103, the inference evaluation unit 104, the explanation evaluation unit 105, the update unit 106, the presentation control unit 107, and the recording control unit 108 each include a computing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and their functions can be realized by the computing device expanding a program stored in a ROM (Read Only Memory) into a RAM and executing it. A computer-readable recording medium on which the program is recorded may also be provided. Alternatively, these blocks may be configured with dedicated hardware or a combination of multiple hardware components. Data required for the computation by the computing device is appropriately stored in a storage unit (not shown).

[0036] In the initial state, initial values ​​are set for the weight parameters 110 of the first to n-th inference models and the weight parameters held by the explanation unit 103. For example, the initial values ​​set for these may be random values, but may also be any values. For example, the initial values ​​set for these may be learned values ​​obtained in advance by learning.

[0037] (Dataset 100) The dataset 100 includes a plurality of pieces of input data (first input data) used in the learning stage and correct answers for each of the plurality of pieces of input data. The plurality of pieces of input data used in the learning stage may correspond to training data. Furthermore, the dataset 100 includes a plurality of pieces of input data (second input data) used in the testing stage. The plurality of pieces of input data used in the testing stage may correspond to testing data.

[0038] It is generally assumed that the test data is prepared as separate data from the training data, but the test data may include part of the training data.

[0039] Furthermore, in the embodiments of the present invention, it is mainly assumed that the input data is image data (especially still image data). However, the type of input data is not particularly limited, and data other than image data can also be used as input data. For example, the input data may be moving image data including multiple frames, or may be audio data.

[0040] (input unit 101) In the learning stage, the input unit 101 sequentially acquires combinations of input data and correct values ​​to be used in the learning stage from the dataset 100. The input unit 101 sequentially outputs the combinations of input data and correct values ​​to be used in the learning stage to the inference unit 102. In addition, in the testing stage, the input unit 101 sequentially acquires input data to be used in testing from the dataset 100. The input unit 101 sequentially outputs the input data to be used in the testing stage to the inference unit 102.

[0041] For example, when the input unit 101 has acquired and output all combinations of input data and correct values ​​to be used in the learning stage from the dataset 100, it may repeat the operation of acquiring the combinations again from the beginning and outputting them again a predetermined number of times. In such a case, the blocks subsequent to the input unit 101 may also sequentially repeat their respective processes based on the re-input. On the other hand, for example, when the input unit 101 has acquired and output all input data to be used in the testing stage from the dataset 100, it may end the acquisition of input data.

[0042] (Inference section 102) In the learning stage, the inference unit 102 obtains inference values ​​(first inference values) corresponding to each of the first to n-th inference models based on the input data input from the input unit 101 and the first to n-th inference models. Similarly, in the testing stage, the inference unit 102 obtains inference values ​​(second inference values) corresponding to each of the first to n-th inference models based on the input data input from the input unit 101 and the first to n-th inference models.

[0043] The weight parameters 110 used by the first to nth inference models are stored in a memory unit (not shown). Therefore, the inference unit 102 acquires the weight parameters 110 from the memory unit (not shown), and performs inference using the first to nth inference models based on the acquired weight parameters 110 and the input data input from the input unit 101.

[0044] In this specification, obtaining an output from a NN based on an input to the NN is broadly referred to as "inference."

[0045] As an example, if the function representing the i-th inference model is Fi (i is an integer from 1 to n) and the input to the i-th inference model is x, the output from the i-th inference model can be expressed as Fi(x).

[0046] As will be explained later, the explanation method (i.e., the method for generating explanation information) used by the explanation unit 103 may include an explanation method that requires information such as features (intermediate features) output from each of the first to nth inference models in addition to the inference value. In such a case, the inference unit 102 may output to the explanation unit 103 the features output from each of the intermediate layers from the first to nth inference models together with the inference value.

[0047] The specific configuration of the first to n-th inference models is not particularly limited. However, it is preferable that the format of each output from the first to n-th inference models be set in accordance with the format of the correct value corresponding to the input data. For example, if the correct value is the class of a classification problem, it is preferable that each output from the first to n-th inference models be a one-hot vector having a length equal to the number of classes.

[0048] In the learning stage, the inference unit 102 outputs inference values ​​corresponding to each of the first to n-th inference models to the explanation unit 103 and the inference evaluation unit 104. On the other hand, in the test stage, the inference unit 102 outputs inference values ​​corresponding to each of the first to n-th inference models to the explanation unit 103 and the presentation control unit 107.

[0049] (Explanatory section 103) The explanation unit 103 generates explanation information that explains the basis for determining the inference value input from the inference unit 102 for each of the first to n-th inference models.

[0050] Here, the explanation information is information indicating the magnitude of contribution of input data to the inferred value input from the inference unit 102. Below, a case will be mainly described in which the explanation information is a heat map indicating the magnitude of contribution of input data to the inferred value for each region (for example, pixels constituting an image) or variable. The heat map can indicate important regions or variables of the input data that contributed to the judgment.

[0051] If the input data is image data or the like, the heat map may be represented by a two-dimensional vector. Alternatively, if the input data is tabular data or the like, the heat map may be represented by a one-dimensional vector.

[0052] The heat map may be generated in any manner. For example, the explanation unit 103 may generate a heat map based on the inferred values ​​input from the inference unit 102. Alternatively, as described above, not only the inferred values ​​but also feature quantities may be input from the inference unit 102 to the explanation unit 103. In such a case, the explanation unit 103 may generate a heat map based on the inferred values ​​and feature quantities input from the inference unit 102.

[0053] For example, the explanation unit 103 may include a function capable of backpropagation. In this case, as will be described later, the weight parameters held by the explanation unit 103 may be updated by the update unit 106 using the backpropagation method. That is, the explanation unit 103 may generate a heat map using the weight parameters updated using the backpropagation method.

[0054] As an explanation method for generating a heat map using weight parameters updated by the backpropagation method, the so-called Grad-CAM described in Non-Patent Document 3 can be applied. Grad-CAM is an explanation method that outputs a heat map showing regions of the input to the NN that have a high contribution to the inferred value. In addition, various explanation methods such as Vanilla Gradient and SmoothGrad can also be applied.

[0055] As described above, the inference value corresponding to the i-th inference model can be expressed as Fi(x). As an example, if the function indicating the heat map generation process is G, the heat map Ti(x) corresponding to the i-th inference model generated by the explanation unit 103 can be expressed as follows:

[0056] Ti(x)=G(Fi(x)) (1)

[0057] In the learning stage, the explanation unit 103 outputs the generated n heat maps (first explanation information) to the explanation evaluation unit 105. On the other hand, in the test stage, the explanation unit 103 outputs the generated n heat maps (second explanation information) to the presentation control unit 107.

[0058] (Inference evaluation unit 104) The inference evaluation unit 104 obtains an inference evaluation result based on the inference values ​​corresponding to each of the first to nth inference models input from the inference unit 102 and the correct answer value acquired by the input unit 101. More specifically, the inference evaluation unit 104 obtains an inference evaluation result by comparing the inference values ​​corresponding to each of the first to nth inference models with the correct answer value acquired by the input unit 101.

[0059] In an embodiment of the present invention, it is assumed that the inference evaluation unit 104 calculates the sum of loss functions corresponding to the inferred value and the correct answer value for the first to nth inference models as a loss function L1 as an example of an inference evaluation result. Here, the loss function corresponding to the inferred value and the correct answer value is not limited to a specific function, and a loss function similar to a loss function used in a general neural network may be used. For example, the loss function corresponding to the inferred value and the correct answer value may be a cross-entropy error based on the difference between the correct answer value and the inferred value.

[0060] The inference evaluation unit 104 outputs the inference evaluation result to the update unit 106 .

[0061] (Explanation Evaluation Section 105) The explanation evaluation unit 105 obtains an explanation evaluation result based on the heat maps corresponding to the first to n-th inference models input from the explanation unit 103. More specifically, the explanation evaluation unit 105 compares the heat maps corresponding to the first to n-th inference models with each other. Then, the explanation evaluation unit 105 obtains an explanation evaluation result based on the degree of agreement between the n heat maps as the comparison results.

[0062] In the embodiment of the present invention, a loss function is mainly assumed in which the explanation evaluation result takes a smaller value as the degree of agreement between n heat maps increases. Note that the degree of agreement between n heat maps may be rephrased as the degree of discrepancy indicating the degree of discrepancy between the n heat maps. In such a case, the loss function may take a smaller value as the degree of discrepancy between the n heat maps decreases.

[0063] There are no limitations on the method for obtaining an explanation evaluation result from n heat maps. Here, we will explain two methods for obtaining an explanation evaluation result: a method for obtaining an explanation evaluation result by multiplying a mask obtained by binarizing a heat map with another heat map, and a method for obtaining an explanation evaluation result by taking the dot product of normalized heat maps.

[0064] 2 is a diagram illustrating a method for obtaining an explanation evaluation result by multiplying a mask obtained by binarizing a heat map with another heat map. In the example shown in FIG. 2, for ease of explanation, it is assumed that n=2, i.e., that the inference unit 102 has a first inference model and a second inference model.

[0065] Referring to Figure 2, the first inference model outputs an inferred value and a heat map H1. Meanwhile, the second inference model outputs an inferred value and a heat map H2. In Figure 2, in heat maps H1 and H2, areas of the input data that contribute more to the inferred value are shown in darker colors.

[0066] The explanation evaluation unit 105 binarizes the heat map H1 to generate a mask M1, and binarizes the heat map H2 to generate a mask M2. Note that the binarization can be performed by setting the value of elements (e.g., pixels constituting the heat map) that are equal to or greater than a threshold c to 1, and the value of elements that are less than the threshold c to 0. In FIG. 2, the binary values ​​1 and 0 are represented by black and white, respectively.

[0067] The explanation evaluation unit 105 calculates, for each element, the product of the heat map H1 output from the first inference model and the mask M2 generated from the heat map H2 output from the second inference model. Similarly, the explanation evaluation unit 105 calculates, for each element, the product of the heat map H2 output from the second inference model and the mask M1 generated from the heat map H1 output from the first inference model. This allows a set of products corresponding to each element to be obtained for each inference model.

[0068] The explanation evaluation unit 105 calculates the sum of products by adding up the products corresponding to each element for all inference models.The explanation evaluation unit 105 then calculates a total value by adding up the sums of products calculated in this way for all elements.The explanation evaluation unit 105 sets this total value as a loss function L2 as an example of the explanation evaluation result.

[0069] The case where n=2 has been described with reference to Figure 2. If n is an arbitrary integer greater than 1, the following description will be given.

[0070] That is, the explanation evaluation unit 105 generates a mask Mi(x) by binarizing the values ​​of each element of the heat map Ti(x) for i = 1 to n. Next, for each inference model, the explanation evaluation unit 105 calculates the product of the heat map Ti(x) output from its own inference model and the sum of the masks M1(x) to Mi-1(x) and Mi+1(x) to Mn(x) generated from the heat maps corresponding to inference models other than its own, for each element.

[0071] The explanation evaluation unit 105 calculates the sum of products by adding up the products corresponding to each element for the first inference model through the nth inference model. The explanation evaluation unit 105 then obtains an explanation evaluation result based on the sum of products calculated in this manner. More specifically, the explanation evaluation unit 105 calculates a total value by adding up the sums of products for all elements. The explanation evaluation unit 105 uses this total value as a loss function L2, an example of the explanation evaluation result.

[0072] This loss function L2 is the sum of the regions in each heat map that have values ​​equal to or greater than a threshold in heat maps other than the target. By performing training to reduce the value of this loss function L2, n inference models with low heat map matching can be obtained. The loss function L2 in this case can be expressed as in the following equation (2). In equation (2), e represents the element number. Here, the heat map Ti(x) may be normalized by dividing it by the size |Ti(x)| of the heat map Ti(x). The heat map Ti(x) may also be multiplied by an activation function such as sigmoid.

[0073]

number

[0074] We have explained a method for obtaining explanation evaluation results by multiplying a mask obtained by binarizing a heat map with another heat map, with reference to Figure 2. Next, we will explain a method for obtaining explanation evaluation results by taking the dot product of normalized heat maps.

[0075] The explanation evaluation unit 105 normalizes the heat map Ti(x) by dividing it by the size |Ti(x)| of the heat map Ti(x) for i=1 to n to generate a normalized vector for i=1 to n. Then, the explanation evaluation unit 105 obtains an explanation evaluation result based on the inner product of the normalized vector for i=1 to n. More specifically, the explanation evaluation unit 105 calculates a total value by adding up the inner products for all elements. The explanation evaluation unit 105 sets this total value as a loss function L2, which is an example of the explanation evaluation result.

[0076] The larger the dot product of the normalized vectors, the larger the value of this loss function L2. A large dot product of normalized vectors means that the degree of match between heat maps is high. Therefore, by performing training to reduce the value of this loss function L2, n inference models with low degrees of match between heat maps can be obtained. Note that the loss function L2 in this case can be expressed as in the following equation (3). In equation (3), e represents the element number.

[0077]

number

[0078] The explanation evaluation unit 105 outputs the explanation evaluation result to the update unit 106 .

[0079] (Updated part 106) The update unit 106 updates the weight parameters 110 used by each of the first to n-th inference models based on the inference evaluation result input from the inference evaluation unit 104 and the explanation evaluation result input from the explanation evaluation unit 105. This allows the weight parameters 110 to be updated so that the inference values ​​output from each of the first to n-th inference models approach the correct value and so that the degree of agreement between the n heat maps output from the explanation unit 103 decreases. The weight parameters 110 may be updated by backpropagation.

[0080] For example, the update unit 106 may add the inference evaluation result input from the inference evaluation unit 104 and the explanation evaluation result input from the explanation evaluation unit 105, and update the weight parameter 110 based on the addition result. At this time, the update unit 106 may update the weight parameter 110 by backpropagation using the calculated addition result as an error. As described above, when the inference evaluation result is expressed as a loss function L1 and the explanation evaluation result is expressed as a loss function L2, the addition result is L1+L2.

[0081] Furthermore, the update unit 106 may update the weight parameters of the explanation unit 103. More specifically, if the explanation unit 103 includes a function capable of backpropagation, the update unit 106 may update the weight parameters of the explanation unit 103 by backpropagation based on the inference evaluation result and the explanation evaluation result.

[0082] The learning termination condition (i.e., the weight parameter update termination condition) is not particularly limited and may be any condition indicating that learning from the first inference model to the nth inference model has been completed to a certain extent. Specifically, the learning termination condition may include a condition that the value of the loss function L1+L2 is smaller than a threshold. Alternatively, the learning termination condition may include a condition that the change in the value of the loss function L1+L2 is smaller than a threshold (a condition that the value of the loss function L1+L2 has converged). Alternatively, the learning termination condition may include a condition that the weight parameters have been updated a predetermined number of times. Alternatively, if the inference evaluation unit 104 calculates accuracy (e.g., accuracy rate) based on the correct answer value and the inferred value, the learning termination condition may include a condition that the accuracy exceeds a predetermined rate (e.g., 90%).

[0083] (Presentation control unit 107) In the test stage, the presentation control unit 107 controls the display unit 121 so that inference values ​​corresponding to the first to nth inference models input from the inference unit 102 and n heat maps input from the explanation unit 103 are presented to the user. More specifically, the presentation control unit 107 controls the display unit 121 so that inference values ​​corresponding to the first to nth inference models and n heat maps are displayed. Note that a configuration in which n heat maps are displayed but inference values ​​corresponding to the first to nth inference models are not displayed may also be envisioned.

[0084] (Display section 121) The display unit 121 is configured by a display and has a function of displaying various information under the control of the presentation control unit 107. For example, the display unit 121 can display n inferred values ​​and n heat maps. Here, the form of the display unit 121 is not particularly limited. For example, the display unit 121 may be a liquid crystal display (LCD) device, an OLED (organic light emitting diode) device, or a display device such as a lamp.

[0085] (Operation unit 122) The operation unit 122 accepts operations by the user. For example, suppose the user refers to n inference values ​​and n heat maps and finds one or more inference models (hereinafter also referred to as "selection models") with high interpretability from the n inference models. At this time, the user inputs information indicating the selection model (hereinafter also referred to as "selection model information") into the operation unit 122, and the operation unit 122 accepts selection model information 123. For example, the selection model information 123 may be a number indicating the selection model.

[0086] In the embodiment of the present invention, it is mainly assumed that the operation unit 122 is a mouse and a keyboard. However, the form of the operation unit 122 is not particularly limited. For example, the operation unit 122 may be a touch panel or another input device.

[0087] (Recording control unit 108) The recording control unit 108 controls the recording of selection model information 123 received from the user by the operation unit 122. More specifically, the recording control unit 108 stores the selection model information 123 received from the user by the operation unit 122 in a storage unit (not shown). The selection model information 123 is later acquired from the storage unit (not shown), and the selection model indicated by the selection model information 123 can be used as a trained model with high interpretability.

[0088] The test termination condition is not particularly limited as long as it indicates that the user has taken the test a sufficient number of times. Specifically, the test termination condition may include a condition that the user has confirmed the inference result a predetermined number of times or more during the test phase.

[0089] An example of the configuration of the learning device 10 according to the embodiment of the present invention has been described above.

[0090] (1.2. Behavior during the learning phase) The flow of operations in the learning stage of the learning device 10 according to the embodiment of the present invention will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of operations in the learning stage of the learning device 10 according to the embodiment of the present invention.

[0091] 3, the input unit 101 acquires a combination of input data (i.e., learning data) and correct values ​​from the dataset 100. Furthermore, the inference unit 102 acquires weight parameters 110 corresponding to each of the n inference models (S11). The inference unit 102 performs inference based on the input data acquired by the input unit 101 and the n inference models (S12), and outputs the n inference values ​​obtained by the inference to the inference evaluation unit 104 and the explanation unit 103, respectively.

[0092] The explanation unit 103 generates a heat map that explains the basis for determining each of the n inference values ​​based on the n inference values ​​input from the inference unit 102 (S13). The explanation unit 103 outputs the generated n heat maps to the explanation evaluation unit 105.

[0093] The inference evaluation unit 104 evaluates the n inference values ​​input from the inference unit 102 based on the correct answer value acquired by the input unit 101 to obtain an inference evaluation result. More specifically, the inference evaluation unit 104 calculates a loss function according to the correct answer value and the n inference values ​​as the inference evaluation result. The inference evaluation unit 104 outputs the calculated inference evaluation result to the update unit 106.

[0094] The explanation evaluation unit 105 obtains an explanation evaluation result based on the degree of coincidence between the n heat maps input from the explanation unit 103. More specifically, the explanation evaluation unit 105 calculates, as the explanation evaluation result, a loss function according to the degree of coincidence between the n heat maps input from the explanation unit 103. The explanation evaluation unit 105 outputs the calculated explanation evaluation result to the update unit 106 (S14).

[0095] The update unit 106 updates the weight parameters 110 corresponding to each of the first to n-th inference models based on the inference evaluation result input from the inference evaluation unit 104 and the explanation evaluation result input from the explanation evaluation unit 105 (S15). More specifically, the update unit 106 updates the weight parameters 110 by backpropagation based on the inference evaluation result and the explanation evaluation result. Furthermore, the update unit 106 updates the weight parameters held by the explanation unit 103 by backpropagation based on the inference evaluation result and the explanation evaluation result.

[0096] Each time the update unit 106 finishes updating the weight parameters based on the input data, it determines whether the learning termination condition has been satisfied (S16). If it determines that the learning termination condition has not been satisfied ("NO" in S16), the operation proceeds to S11, the input unit 101 acquires the next input data, and the inference unit 102, explanation unit 103, inference evaluation unit 104, explanation evaluation unit 105, and update unit 106 each execute their own processing based on the next input data again. On the other hand, if the update unit 106 determines that the learning termination condition has been satisfied ("YES" in S16), the learning is terminated.

[0097] The flow of operations in the learning stage of the learning device 10 according to an embodiment of the present invention has been described above.

[0098] (1.3. Operation during testing) The flow of operations in the test stage of the learning device 10 according to the embodiment of the present invention will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of operations in the test stage of the learning device 10 according to the embodiment of the present invention.

[0099] 4, the input unit 101 acquires a combination of input data (i.e., test data) and a correct answer value from the dataset 100. Furthermore, the inference unit 102 acquires weight parameters 110 corresponding to each of the n inference models (S21). The inference unit 102 performs inference based on the input data acquired by the input unit 101 and the n inference models (S22), and outputs the n inference values ​​obtained by the inference to the explanation unit 103 and the presentation control unit 107, respectively.

[0100] The explanation unit 103 generates a heat map that explains the basis for determining each of the n inferred values ​​based on the n inferred values ​​input from the inference unit 102 (S23). The explanation unit 103 outputs the generated n heat maps to the presentation control unit 107.

[0101] The presentation control unit 107 controls the display unit 121 so that the n inferred values ​​input from the inference unit 102 and the n heat maps input from the explanation unit 103 are presented to the user. The display unit 121 displays the n inferred values ​​and the n heat maps in accordance with the control of the presentation control unit 107 (S24).

[0102] The operation unit 122 receives from the user information (selected model information 123) indicating one or more inference models determined to have high interpretability from the n inference models. The recording control unit 108 controls the recording of the selected model information 123 received from the user by the operation unit 122 (S25). A storage unit (not shown) stores the selected model information 123 in accordance with the control of the recording control unit 108.

[0103] The recording control unit 108 determines whether the test termination condition is satisfied (S26) each time the recording control of the selection model information 123 based on the input data is completed. If it is determined that the test termination condition is not satisfied ("NO" in S26), the operation proceeds to S21, the input unit 101 acquires the next input data, and the inference unit 102, the explanation unit 103, the presentation control unit 107, and the recording control unit 108 each execute their own processing based on the next input data again. On the other hand, if the recording control unit 108 determines that the test termination condition is satisfied ("YES" in S26), the test is terminated.

[0104] The flow of operations in the test stage of the learning device 10 according to an embodiment of the present invention has been described above.

[0105] (1.4. Effects of the embodiment) As described above, according to an embodiment of the present invention, learning can be performed so that the inference values ​​output from each of the first to nth inference models approach the correct value and so that the degree of similarity between the n heat maps output as explanatory information decreases. This makes it possible to obtain inference models that output multiple heat maps that are different from each other. This allows a user to select and use a model that outputs a heat map with higher interpretability from among the n models.

[0106] The effects achieved by the embodiments of the present invention have been described above.

[0107] (2. Hardware configuration example) Next, an example of the hardware configuration of the learning device 10 according to an embodiment of the present invention will be described. Below, an example of the hardware configuration of an information processing device 900 will be described as an example of the hardware configuration of the learning device 10 according to an embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the learning device 10. Therefore, the hardware configuration of the learning device 10 may be such that unnecessary components are deleted from the hardware configuration of the information processing device 900 described below, or new components are added.

[0108] 5 is a diagram showing the hardware configuration of an information processing device 900 as an example of the learning device 10 according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0109] The CPU 901 functions as an arithmetic processing unit and control unit, and controls the overall operation of the information processing device 900 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, etc. These are interconnected by a host bus 904 that is composed of a CPU bus, etc.

[0110] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not necessary to configure the host bus 904, bridge 905, and external bus 906 separately, and these functions may be implemented on a single bus.

[0111] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers that allow the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. By operating this input device 908, the user operating the information processing device 900 can input various data to the information processing device 900 and instruct the information processing device 900 to perform processing operations.

[0112] The output device 909 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp, and an audio output device such as a speaker.

[0113] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is configured, for example, with an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

[0114] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network, etc. The communication device 911 may be compatible with either wireless communication or wired communication.

[0115] An example of the hardware configuration of the learning device 10 according to an embodiment of the present invention has been described above.

[0116] (3. Summary) Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0117] For example, the above example mainly assumes that the learning device 10 simultaneously learns n inference models. However, the learning device 10 does not have to simultaneously learn all n inference models. For example, a trained inference model may be used as part of the n inference models. In this case, the weight parameters of the trained inference model may be fixed to a constant value without being updated.

[0118] Furthermore, in the above example, it is mainly assumed that the explanation unit 103 uses one type of heat map generation method. However, the explanation unit 103 may use multiple types of heat map generation methods. In this case, the explanation unit 103 may output the total value of the losses for the multiple types of heat map generation methods based on the degree of agreement between the heat maps to the update unit 106 as an example of the explanation evaluation result. [Explanation of symbols]

[0119] 10 Learning Device 100 datasets 101 Input section 102 Reasoning part 103 Explanation Section 104 Inference Evaluation Unit 105 Explanation and Evaluation Department 106 Update Department 107 Presentation control unit 108 Recording control section 110 Weight Parameter 121 Display section 122 Operation section 123 Selection Model Information

Claims

1. an input unit that acquires first input data and a correct answer value of the first input data; an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and a plurality of inference models; an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models, the first explanation information indicating the magnitude of contribution of the first input data to the first inferred value; an inference evaluation unit that obtains an inference evaluation result based on the correct answer value and the first inference value; an explanation evaluation unit that obtains an explanation evaluation result based on the degree of agreement between first explanation information corresponding to each of the plurality of inference models; an update unit that updates first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result so as to reduce the degree of agreement between the first explanation information; A learning device comprising: the input unit acquires second input data; the inference unit outputs second inference values ​​corresponding to the plurality of trained models, which are the plurality of estimation models after updating the first weight parameters, based on the second input data and the plurality of trained models; the explanation unit outputs second explanation information corresponding to each of the plurality of trained models, the second explanation information indicating a magnitude of contribution of the second input data to the second inferred value; the learning device includes: a presentation control unit that controls presentation of the second inferred value and the second explanation information to a user; an operation unit that receives, from the user, selected model information indicating one or more trained models selected by the user from the plurality of trained models; A learning device comprising:

2. The learning device A recording control unit that controls recording of information indicating the one or more trained models, The learning device according to claim 1 .

3. the explanation evaluation result takes a smaller value as the degree of agreement between the first explanation information corresponding to each of the plurality of inference models increases; The learning device according to claim 1 or 2.

4. the explanation evaluation unit obtains the explanation evaluation result based on an inner product of vectors normalized from first explanation information corresponding to each of the plurality of inference models; The learning device according to claim 3 .

5. The explanation evaluation unit binarizes the first explanation information to generate a mask for each of the multiple inference models, calculates the product of the mask generated from the first explanation information corresponding to an inference model other than itself and the first explanation information corresponding to its own inference model, and obtains the explanation evaluation result based on the sum of the products for each of the multiple inference models. The learning device according to claim 3 .

6. The explanation section includes a function capable of backpropagation. The learning device according to any one of claims 1 to 5.

7. the explanation section has a second weighting parameter; the update unit updates the second weight parameters by backpropagation. The learning device according to claim 6.

8. At least one of the plurality of inference models includes a neural network. The learning device according to any one of claims 1 to 7.

9. the update unit updates the first weight parameter based on an addition result of the inference evaluation result and the explanation evaluation result. The learning device according to any one of claims 1 to 8.

10. the first explanatory information is a heat map showing the magnitude of contribution of the first input data to the first inferred value; The learning device according to any one of claims 1 to 9.

11. Obtaining first input data and a correct answer value of the first input data; outputting a first inference value corresponding to each of the plurality of inference models based on the first input data and the plurality of inference models; outputting first explanatory information corresponding to each of the plurality of inference models indicating the magnitude of contribution of the first input data to the first inferred value; obtaining an inference evaluation result based on the correct answer value and the first inference value; obtaining an explanation evaluation result based on the degree of agreement between first explanation information corresponding to each of the plurality of inference models; updating first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result so that the degree of agreement between the first explanation information becomes smaller; obtaining second input data; outputting second inference values ​​corresponding to each of the plurality of trained models based on the second input data and the plurality of trained models, which are the plurality of estimation models after updating the first weight parameters; and outputting second explanation information corresponding to each of the plurality of trained models, the second explanation information indicating a magnitude of contribution of the second input data to the second inferred value; controlling presentation of the second inference value and the second explanatory information to a user; receiving selected model information from the user indicating one or more trained models selected by the user from the plurality of trained models; including, learning methods.

12. Computer, an input unit that acquires first input data and a correct answer value of the first input data; an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and a plurality of inference models; an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models, the first explanation information indicating the magnitude of contribution of the first input data to the first inferred value; an inference evaluation unit that obtains an inference evaluation result based on the correct answer value and the first inference value; an explanation evaluation unit that obtains an explanation evaluation result based on the degree of agreement between first explanation information corresponding to each of the plurality of inference models; an update unit that updates first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result so as to reduce the degree of agreement between the first explanation information; A learning device comprising: the input unit acquires second input data; the inference unit outputs second inference values ​​corresponding to the plurality of trained models, which are the plurality of estimation models after updating the first weight parameters, based on the second input data and the plurality of trained models; the explanation unit outputs second explanation information corresponding to each of the plurality of trained models, the second explanation information indicating a magnitude of contribution of the second input data to the second inferred value; the learning device includes: a presentation control unit that controls presentation of the second inferred value and the second explanation information to a user; an operation unit that receives, from the user, selected model information indicating one or more trained models selected by the user from the plurality of trained models; A program that functions as a learning device.

Images