Explanation information output program, explanation information output method and explanation information output device

Abstract

To output proper explanatory information regarding a prediction result of a machine learning model.SOLUTION: An explanatory information output program is configured to: use an explanatory model generated based on a prediction result outputted by a machine learning model according to input of multiple data containing multiple variables respectively and the multiple data to calculate respective contributions of the multiple data with regard to the prediction result; select specific variables among the multiple variables; determine specific data among the multiple data on the basis of values of the respective specific variables of the multiple data, and the contributions; output the specific data as explanatory information of the prediction result, thereby, outputting proper explanatory information regarding the prediction result of the machine learning model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an explanatory information output program, an explanatory information output method, and an explanatory information output device. [Background technology]

[0002] Complex machine learning predictive models, primarily those based on deep learning (DL), have a high degree of black-box nature, which is a major obstacle to their practical use in business.

[0003] In recent years, XAI (Explainable Artificial Intelligence) has become known as a technology that makes the process leading to predictions and inferences in machine learning models explainable to humans. An explainable state can be said to be a state that is convincing to users.

[0004] In XAI, tools such as LIME and SHAP are known to interpret the basis on which a machine learning model made a prediction about a given sample. In addition, a method of calculating the contribution rate per factor is known as a conventional technique for clarifying what models such as LIME and SHAP emphasize. In the contribution calculation method, the contribution of each of a number of factors to the output result of the model is calculated, and factors with high contributions are presented. FIG. 9 is a diagram showing an example of output by SHAP.

[0005] In the example shown in Figure 9, the contribution (importance) of multiple factors to a certain output of the model is shown in a graph. In the example shown in Figure 9, we can see that the factor "LSTAT" contributed the most to the prediction task. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2010-165166 A [Patent Document 2] JP 2019-82883 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in practical use of XAI, it is rare for a single explanation to be able to explain the entire user base. FIG. 10 is a diagram for explaining the conditions for an explainable state in XAI.

[0008] FIG. 10 shows how multiple users interpret the explanatory information "the factor 'LSTAT' has the highest contribution to a certain model output" in the example shown in FIG.

[0009] Let us assume that "the factor 'LSTAT' has a high contribution to a certain model output" is highly subjective and is taken for granted (relatively universal) for Mr. A. In such a case, the explanatory information is not useful to Mr. A. Explanatory information in XAI is required to go beyond the barrier of the obvious.

[0010] On the other hand, the above explanatory information, "The factor "LSTAT" has the highest contribution to a certain model output," is different from B's knowledge and insight. In such a case, B cannot understand the above explanatory information. XAI explanatory information is also required to overcome the barriers of knowledge and insight.

[0011] Also, the above explanatory information, "The factor "LSTAT" has the highest contribution to a certain model output," is assumed to match the knowledge and insight of Mr. C. In such a case, the above explanatory information is accepted by Mr. C as being similar to his own sense and convincing.

[0012] In this way, the explainable state may be highly personal and different for each user. Therefore, in the conventional information presentation method in XAI, if the explanatory information does not include explanatory factors that touch on each user's knowledge and insight, the explanatory information may not be in a state that can be explained to the user.

[0013] In the conventional information presentation method for XAI, the contribution of each data (variable) containing multiple attributes to the model output is calculated, and the data with the highest contribution is presented. However, since the attributes that are considered important when interpreting the data vary depending on the time and situation (including person-dependence), the factors presented may not be appropriate. In one aspect, the present invention aims to enable output of appropriate explanatory information for the prediction results of a machine learning model. [Means for solving the problem]

[0014] Therefore, this explanatory information output program uses multiple variables. Have Multiple Data Consists of input Data Machine learning models By entering Prediction results and for each of the plurality of data Regarding the predicted results Ruyomi Calculate the degree of the givenness, and among the multiple variables in Specific First choice by users Variables identification death, One or more pieces of data that are explanatory information for the prediction result, The plurality of data based on the contribution and the value of only the first variable among the plurality of variables, Determined, 1 or more Data Leave The processing is then input to the computer. Effect of the Invention

[0015] According to one embodiment, appropriate explanatory information can be output for the prediction results of a machine learning model. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing apparatus as an example of an embodiment. [Diagram 2]FIG. 1 is a diagram illustrating a hardware configuration of an information processing apparatus according to an embodiment; [Diagram 3] 1 is a diagram illustrating an example of input data in an information processing device as an example of an embodiment; [Figure 4] FIG. 2 is a diagram illustrating a user-oriented data input screen in an information processing apparatus as an example of an embodiment; [Diagram 5] 1 is a diagram for explaining a contribution degree generated by an interpretable model generation unit of an information processing device as an example of an embodiment. FIG. [Figure 6] 11 is a diagram illustrating a method for selecting an edge to be presented when categorical data is used in an information processing device as an example of an embodiment. FIG. [Figure 7] 11A and 11B are diagrams for explaining a method of selecting an edge to be presented when numerical data is displayed in an information processing device as an example of an embodiment. [Figure 8] 1 is a flowchart illustrating a process in an information processing device as an example of an embodiment. [Figure 9] FIG. 13 is a diagram showing an example of output by SHAP. [Figure 10] FIG. 13 is a diagram for explaining the conditions for an explainable state in XAI. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, the embodiments of the explanatory information output program, explanatory information output method, and explanatory information output device will be described with reference to the drawings. However, the embodiments shown below are merely examples, and there is no intention to exclude the application of various modified examples and techniques not explicitly stated in the embodiments. In other words, this embodiment can be implemented with various modifications within the scope of its purpose. Furthermore, each figure is not intended to include only the components shown in the figure, but may include other functions, etc.

[0018] (A) Configuration FIG. 1 is a diagram illustrating a schematic configuration of an information processing device 1 as an example of an embodiment. The information processing device 1 is an explanatory information output device (computer) that realizes XAI, which performs inference on input data using a machine learning model and presents explanatory information showing the grounds for the inference to a user. FIG. 2 is a diagram illustrating a hardware configuration of an information processing device 1 as an example of an embodiment.

[0019] The information processing device 1 includes, as components, a processor 11, a memory 12, a storage device 13, a graphics processing device 14, an input interface 15, an optical drive device 16, a device connection interface 17, and a network interface 18. These components 11 to 18 are configured to be able to communicate with each other via a bus 19.

[0020] The processor (processing unit) 11 controls the entire information processing device 1. The processor 11 may be a multiprocessor. The processor 11 may be, for example, any one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). The processor 11 may also be a combination of two or more types of elements among the CPU, MPU, DSP, ASIC, PLD, and FPGA.

[0021] The processor 11 executes a control program (explanation information output program: not shown) for the information processing device 1 to realize the functions of the processing unit 101 and the explanatory information generating unit 106 illustrated in Fig. 1. As a result, the information processing device 1 functions as an explanatory information output device.

[0022] In addition, the information processing device 1 realizes the functions of the processing unit 101 and the explanatory information generation unit 106 by executing programs {explanatory information output program, OS (Operating System) program} recorded, for example, on a computer-readable non-transitory recording medium.

[0023] The programs describing the processing contents to be executed by the information processing device 1 can be recorded on various recording media. For example, the programs to be executed by the information processing device 1 can be stored in the storage device 13. The processor 11 loads at least a part of the programs in the storage device 13 into the memory 12 and executes the loaded programs.

[0024] The programs to be executed by the information processing device 1 (processor 11) may also be recorded in a non-transitory portable recording medium such as the optical disk 16a, the memory device 17a, or the memory card 17c. The programs stored in the portable recording medium are executable after being installed in the storage device 13, for example, under the control of the processor 11. The processor 11 may also read and execute the programs directly from the portable recording medium.

[0025] The memory 12 is a storage memory including a ROM (Read Only Memory) and a RAM (Random Access Memory). The RAM of the memory 12 is used as a main storage device of the information processing device 1. The RAM temporarily stores at least a part of the program to be executed by the processor 11. The memory 12 also stores various data necessary for processing by the processor 11.

[0026] The storage device 13 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a storage class memory (SCM), and stores various data. The storage device 13 is used as an auxiliary storage device for the information processing device 1. The storage device 13 stores an OS program, a control program, and various data. The control program includes an explanatory information output program.

[0027] As the auxiliary storage device, a semiconductor storage device such as an SCM or a flash memory can also be used. Furthermore, a plurality of storage devices 13 may be used to configure a RAID (Redundant Arrays of Inexpensive Disks). The storage device 13 may also store various data generated when the above-mentioned processing unit 101 and explanation information generating unit 106 execute each process.

[0028] A monitor 14a is connected to the graphics processing device 14. The graphics processing device 14 displays an image on the screen of the monitor 14a in accordance with an instruction from the processor 11. Examples of the monitor 14a include a display device using a CRT (Cathode Ray Tube) and a liquid crystal display device.

[0029] A keyboard 15a and a mouse 15b are connected to the input interface 15. The input interface 15 transmits signals sent from the keyboard 15a and the mouse 15b to the processor 11. The mouse 15b is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

[0030] The optical drive device 16 uses a laser beam or the like to read data recorded on an optical disk 16a. The optical disk 16a is a portable, non-transient recording medium on which data is recorded so that it can be read by the reflection of light. Examples of the optical disk 16a include a DVD (Digital Versatile Disk), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), and a CD-R (Recordable) / RW (ReWritable).

[0031] The device connection interface 17 is a communication interface for connecting peripheral devices to the information processing device 1. For example, a memory device 17a or a memory reader / writer 17b can be connected to the device connection interface 17. The memory device 17a is a non-transitory recording medium equipped with a communication function with the device connection interface 17, such as a Universal Serial Bus (USB) memory. The memory reader / writer 17b writes data to a memory card 17c or reads data from the memory card 17c. The memory card 17c is a card-type non-transitory recording medium.

[0032] The network interface 18 is connected to a network. The network interface 18 transmits and receives data via the network. Other information processing devices, communication devices, etc. may be connected to the network. The information processing device 1 illustrated in FIG. 1 includes a processing unit 101 and an explanation information generating unit 106. The processing unit 101 receives input data and processes the input data using a machine learning model. FIG. 3 is a diagram illustrating input data in the information processing device 1 as an example of the embodiment.

[0033] In this embodiment, the information processing device 1 is used in the field of health management, and a task is described as an example in which a machine learning model is used to perform binary classification of future absence from work (whether or not the person will take leave) from attendance data. That is, the attendance data of a person to be predicted is input to the machine learning model, and the model predicts whether the person to be predicted will take leave in the future. The input data illustrated in FIG. 3 is tensor data indicating employee attendance data, and includes date, attendance, absence, whether or not a business trip is made (business trip), and overtime hours (overtime).

[0034] In the input data shown in Figure 3, the combination of date, attendance, business trip, and overtime is sometimes called an edge. Also, the variables that make up an edge (business trip, overtime) are sometimes called factors. An edge can be said to be a complex factor relationship. The input data corresponds to a plurality of data (attendance data, edges) including a plurality of variables (factors: business trips, overtime).

[0035] User-oriented data is also input to the processing unit 101. The user-oriented data is information indicating factors that the user considers (focuses on) to contribute to prediction by the machine learning model, among multiple types of factors that configure an edge. The user-oriented data may be input in advance by the user using, for example, an input screen such as that shown in FIG. FIG. 4 is a diagram illustrating a user-oriented data input screen in the information processing device 1 as one example of the embodiment.

[0036] The user-oriented data input screen is displayed on the monitor 14a or the like provided in the information processing device 1. The user inputs his / her own user-oriented data via this user-oriented data input screen.

[0037] The user-oriented data entry screen shown in Figure 4 displays two options, "Whether or not to travel" and "Overtime hours", as toggle switches, along with the message, "From the attendance management items shown below, please select one that you think will have the greatest impact on whether or not you will take a leave of absence." These "Whether or not to travel" and "Overtime hours" correspond to the factors of business trips and overtime work.

[0038] For example, multiple users (Mr. A, Mr. B) belonging to a company's human resources or health promotion department each selects on this user-oriented data input screen the item that they believe will have the greatest impact on whether or not they will take a leave of absence, and operates the keyboard 15a and mouse 15b to input their selection.

[0039] The contents selected and input by the user are stored as user-oriented data in a predetermined storage area such as the storage device 13 or memory 12 for each user. Hereinafter, the user-oriented data may be called a priority factor. The user-oriented data corresponds to a priority variable according to the user.

[0040] In the example shown in FIG. 4, one factor is selected as the user-oriented data, but the present invention is not limited to this. The user may select two or more factors as the user-oriented data, and the number of factors may be changed as appropriate. The number of user-oriented data selected by the user may be called the number of priority factors (N). The number of priority factors (N) may be arbitrarily set in advance by a system administrator or the like. In this embodiment, for convenience, an example in which the number of priority factors N=1 is shown. As shown in FIG. 1, the processing unit 101 includes a machine learning unit 102, an inference unit 103, and an interpretable model generation unit 104. The machine learning unit 102 creates a machine learning model using a training data set. The training data set includes input data and correct answer data (teacher data). As the machine learning model, for example, a support vector machine, a neural network, a gradient boosting tree, or the like may be used. In the example shown below, the input data is mainly tables and graphs expressed as tensors, and the machine learning unit 102 creates a machine learning model using the Deep Tensor (registered trademark) algorithm, which is a known machine learning technology.

[0041] The machine learning unit 102 handles tables and graphs in the tensor format. Tables and graphs may be called tensor data. The machine learning unit 102 converts input data into core tensors by tensor decomposition, inputs the core tensors to a neural network, and creates a machine learning model by machine learning that outputs an inference result for the input data.

[0042] The machine learning unit 102 generates a machine learning model by optimizing parameters used in tensor decomposition and parameters of a neural network based on training data. The machine learning unit 102 optimizes the parameters by updating the parameters of the tensor decomposition and the parameters of the neural network using, for example, a gradient descent method in a direction that reduces a loss function that defines an error between an inference result of the machine learning model for the training data and correct data.

[0043] The neural network may be a hardware circuit, or may be a virtual network using software that connects layers virtually constructed by the processor 11 on a computer program.

[0044] The inference unit 103 performs inference using the machine learning model created by the machine learning unit 102. The inference unit 103 inputs tensor data to be inferred to the machine learning model, and obtains an inference result output by the machine learning model.

[0045] In order to explain the inference results of the machine learning model created by the machine learning unit 102, the interpretable model generation unit 104 generates an interpretable model that locally approximates the machine learning model that can easily explain the contribution of each variable included in the input data.

[0046] The interpretable model generating unit 104 generates an interpretable model using a known method for interpreting the basis on which prediction (inference) of a machine learning model was made, such as LIME or SHAP. The interpretable model generating unit 104 may generate a linear regression model that locally approximates the machine learning model, based on a multiple regression analysis of the inference result of the data to be explained inferred by the machine learning model and the inference result of the neighborhood data of the data to be explained. Here, the neighborhood data is data that is similar to the data to be explained, and the difference with the data to be explained is equal to or less than a threshold. The interpretable model generating unit 104 may determine that the data with a core tensor similarity equal to or greater than a threshold is the neighborhood data. The similarity can be defined by the distance between tensors (for example, square error). By generating a linear regression model based on the core tensor of the data to be explained and the inference result thereof, it can be said that the interpretable model generating unit 104 generates an interpretable model based on the core tensor similarity.

[0047] The interpretable model specifies the contribution of each variable included in the data to be explained to the inference result based on the parameters of the generated linear regression model. The interpretable model generation unit 104 may generate the interpretable model based on the prediction result output by the machine learning model in response to the input of multiple data (attendance data, edges), each of which includes multiple variables (factors), and multiple data. In this case, the interpretable model generation unit 104 may calculate the contribution degree for each edge based on the interpretable model. The partial regression coefficient of the linear regression model may be used as the contribution degree. FIG. 5 is a diagram for explaining the contribution degree generated by the interpretable model generation unit 104 of the information processing device 1 as one example of the embodiment.

[0048] 5, for example, a contribution rate of "0.6" is set for the edge identified by the date 5 / 27. Note that the calculation of the contribution rate can be realized by a known method, and the description thereof will be omitted.

[0049] The explanatory information generating unit 106 generates information indicating the grounds for the estimation performed on the input data by the processing unit 101. The explanatory information generating unit 106 generates explanatory information (explanatory information for each user) that allows the user to understand and accept the estimation made by the processing unit 101.

[0050] In the present information processing device 1, in consideration of the fact that user-oriented data differs for each user due to the knowledge and insight of each user, information (explanation information) explaining the basis of prediction (inference) by the machine learning model in XAI is customized for each user and presented. In the present information processing device 1, the explanation information generation unit 106 uses edges selected from the input data as explanation information for the user. That is, the explanation information generation unit 106 selects information (edges) that contributed to the estimation and presents it as explanation information. As shown in FIG. 1, the explanation information generating unit 106 includes a priority factor determining unit 107 and an information generating unit 108.

[0051] The priority factor determination unit 107 checks the priority factors (user-oriented data) of each user. The priority factor determination unit 107 reads the user-oriented data input by the user via the user-oriented data input screen from a predetermined storage area such as the storage device 13 or memory 12, and checks the contents. The information generation unit 108 generates, for each user, information (explanation information) that explains the basis for predictions (inferences) made by the machine learning model in XAI.

[0052] The information generating unit 108 uses, for example, an edge selected for each user from among multiple input data as explanatory information. Hereinafter, an edge presented to a user as explanatory information may be referred to as a presented edge. The presented edge corresponds to explanatory information of a prediction result output by a machine learning model.

[0053] For example, the information generating unit 108 may select some edges as presentation edges from among the multiple edges constituting the input data, and present only these presentation edges as explanatory information to the user. Also, the information generating unit 108 may select multiple edges as presentation edges, set priorities for these multiple presentation edges, and present the multiple presentation edges, arranged in descending order of priority, to the user as explanatory information. The information generating unit 108 selects the presentation edge based on the user preference data of each user.

[0054] Furthermore, when selecting an edge to be presented, the information generating unit 108 compares the contribution degree of each edge set by the interpretable model generating unit 104 with a preset contribution degree threshold value. The information generating unit 108 selects an edge whose contribution degree is greater than the contribution degree threshold value as a candidate for the edge to be presented (a candidate edge to be presented). The contribution degree threshold may be set in advance by a system administrator, etc. Also, the contribution degree threshold may be determined statistically based on past contribution degrees associated with operation.

[0055] The information generating unit 108 generates explanatory information using a method according to the data format of the priority factor determined by the priority factor determining unit 107. The information generating unit 108 generates presentation data using different methods depending on whether the data format of the priority factor is numerical data or categorical data.

[0056] Here, the priority factor, which is numerical data, is a factor expressed by a numerical value, and in this embodiment, it corresponds to "overtime" expressed by the numerical value of "overtime hours." On the other hand, the priority factor, which is categorical data, is a factor expressed by being classified into one of a set of categories, and in this embodiment, it corresponds to "business trip" classified into either "all day (yes)" or "no." FIG. 6 is a diagram for explaining a method of selecting an edge to be presented when categorical data is used in the information processing device 1 as an example of an embodiment.

[0057] When the data format of the priority factor is categorical data, the information generating unit 108 extracts one or more edges (candidate presentation edges) whose contribution degree is greater than a contribution degree threshold from the input data (edge ​​group). Then, the information generating unit 108 specifies one or more edges (categorical data generating edges) in which a value occurs (is active) in the priority factor from among these extracted candidate presentation edges. The candidate presentation edges correspond to candidate presentation data. The information generating unit 108 selects the edge with the highest contribution degree as the presentation edge from among these extracted candidate presentation edges.

[0058] The information generation unit 108 selects a specific variable (the same as the priority factor) from among the multiple variables (factors), and determines a specific data item (presentation edge) from among the multiple edges based on the value of the specific factor of each of the multiple data items (edges) and the contribution degree. That is, the information generating unit 108 determines, from among a plurality of data (edge ​​group), a plurality of candidate presentation data (candidate presentation edges) each of which indicates a specific category in which a value of a specific variable is the same as the priority factor and whose contribution degree is greater than a contribution degree threshold value.Then, the information generating unit 108 determines, from among the plurality of candidate presentation edges, the data with the highest contribution degree as the presentation edge (specific data).

[0059] In the following example, the contribution threshold is 0.45. It is assumed that Mr. B's priority factor (user-oriented data) is the categorical data "business trip." In the example shown in FIG. 6, two edges with dates of 5 / 24 and 5 / 26 correspond to presentation candidate edges whose contribution degree is greater than the contribution degree threshold value (0.45). The information generating unit 108 selects, from among these two presentation candidate edges, the edge with the date of 5 / 26 (see symbol P1), which corresponds to the categorical data occurrence edge, as the presentation edge.

[0060] The information generating unit 108 may select a plurality of presentation edges whose contribution degrees are greater than a contribution degree threshold, and present these presentation edges to the user in order of decreasing contribution degree. FIG. 7 is a diagram for explaining a method of selecting an edge to be presented when numerical data is used in the information processing device 1 as an example of an embodiment.

[0061] When the data format of the priority factor is numerical data, the information generating unit 108 extracts one or more candidate presentation edges (candidate presentation data) whose contribution degree is greater than a contribution degree threshold from the input data (edge ​​group).Then, the information generating unit 108 specifies one or more edges (numerical data generating edges) in which a numerical value is generated (active) in the priority factor from among the extracted candidate presentation edges.The information generating unit 108 selects the presentation edge using two parameters, namely, each data of the priority factor in each numerical data generating edge and each contribution degree. In the example shown below, it is assumed that Mr. A's priority factor (user-oriented data) is "overtime", which is numerical data.

[0062] In the example shown in FIG. 7, the three edges with dates of 5 / 24, 5 / 25, and 5 / 26 all correspond to presentation candidate edges whose contribution degree values ​​are greater than the contribution degree threshold value (0.45).

[0063] The information generation unit 108 obtains each overtime value of the priority factor and each contribution value from these three candidate presentation edges, normalizes the overtime value and contribution value of each edge, and calculates the sum (total value) of these normalized overtime values ​​and contribution values ​​for each edge.

[0064] In the example shown in FIG. 7, the information generation unit 108 standardizes each overtime value of the presentation candidate edge with a mean of 0 and a variance of 1, thereby converting the overtime values ​​of 3, 1, and 1 on dates 5 / 24, 5 / 25, and 5 / 26 to 1.4, -0.7, and -0.7, respectively.

[0065] In addition, the information generating unit 108 standardizes the contribution degree values ​​of the presentation candidate edges with an average of 0 and a variance of 1, thereby converting the contribution degree values ​​of 0.6, 0.5, and 0.7 for the dates 5 / 24, 5 / 25, and 5 / 26 to 0, -1.2, and 1.2, respectively. The information generating unit 108 may use, for example, affine transformation to standardize (normalize) each overtime value and each contribution degree value.

[0066] Then, the information generating unit 108 calculates the sum by adding up the standardized overtime and contribution values ​​for each presentation candidate edge. In the example shown in Fig. 7, for example, the standardized overtime and contribution values ​​for the date 5 / 24 are 1.4 and 0, respectively, and the sum of these is 1.4.

[0067] The information generating unit 108 selects the presentation candidate edge (see symbol P2) with the highest total value and the date of 5 / 24 from among the three presentation candidate edges. The information generating unit 108 presents the selected presentation candidate edge to the user as a presentation edge (see symbol P3).

[0068] The information generating unit 108 may select a plurality of candidate presentation edges, and present the selected plurality of candidate presentation edges to the user in descending order of the total value.

[0069] In this way, when the data format of the priority factor is numerical data, the information generating unit 108 selects the edge to be presented taking into consideration not only the magnitude of the contribution degree but also the magnitude of the numerical data of the priority factor.

[0070] (B) Operation The process of the information processing device 1 as one example of the embodiment configured as described above will be described with reference to the flowchart (steps S1 to S13) shown in FIG. In step S1, the processing unit 101 acquires input data to be inferred. In step S2, the inference unit 103 inputs the input data to the machine learning model created by the machine learning unit 102, and obtains an inference result output by the machine learning model. In step S3, the interpretable model generation unit 104 generates an interpretable model based on the core tensor (feature amount) generated by the machine learning unit 102.

[0071] The interpretable model generation unit 104 determines the neighborhood data based on the similarity of the core tensors. The interpretable model generation unit 104 creates a linear regression model using the neighborhood data and the inference result. The interpretable model specifies the contribution of each variable included in the data to be explained to the inference result based on the parameters of the generated linear regression model. The interpretable model generation unit 104 generates an interpretable model based on the prediction result output by the machine learning model in response to the input of multiple data (attendance data, edges), each of which includes multiple variables (factors), and multiple data. In step S4, the processing unit 101 sets a contribution degree threshold value that is input in advance by the user, for example.

[0072] Furthermore, each user of the information processing device 1 inputs information using the user-oriented data input screen. In step S5, the processing unit 101 acquires user-oriented data (priority factors) for each of a plurality of users (target users) who use the information processing device 1. In step S6, the information generating unit 108 checks whether explanation information has been generated for all users.

[0073] If it is determined that explanation information has not been generated for all users (see the NO route from step S6), the process proceeds to step S7. The following processes in steps S7 to S12 are performed for each user.

[0074] In step S7, the priority factor determination unit 107 confirms the priority factors of the user. The explanation information generation unit 106 acquires, from the input data, one or more edges (edge ​​group) in which the priority factors of the user are active.

[0075] In step S8, the information generation unit 108 checks whether there is an edge (candidate presentation edge) whose contribution exceeds the contribution threshold in the edge group acquired in step S7. If there is no edge whose contribution is greater than the contribution threshold (see NO route in step S8), the process proceeds to step S9. In step S9, the information generation unit 108 sets the output result of the interpretable model generated by the interpretable model generation unit 104 using a known tool such as LIME or SHAP as explanatory information without making any changes. Then, the process returns to step S6.

[0076] Also, if the result of the check in step S8 is that there is an edge whose contribution degree is greater than the contribution degree threshold (see the YES route in step S8), the process proceeds to step S10. In step S10, the information generating unit 108 checks whether the data format of the priority factor is numerical data. As a result of the check, if the data format of the priority factor is numeric data (see the YES route of step S10), the process proceeds to step S12.

[0077] In step S12, the information generating unit 108 selects an edge to be presented or changes the display order of the multiple edges to be presented based on two parameters, that is, the numerical values ​​of the priority factors and the contribution degrees of the multiple edges to be presented. After that, the process returns to step S6.

[0078] Furthermore, if the result of the check in step S10 is that the data format of the priority factor is not numerical data, that is, is categorical data (see the NO route in step S10), the process proceeds to step S11.

[0079] In step S11, the information generating unit 108 selects an edge having the highest contribution degree from among the multiple presentation candidate edges as a presentation edge. Alternatively, the information generating unit 108 changes the display order of the multiple presentation candidate edges so that the edges are displayed in descending order of the contribution degree. After that, the process returns to step S6.

[0080] Also, if the result of the check in step S6 indicates that explanation information has been generated for all users (see the YES route in step S6), the process proceeds to step S13. In step S13, the explanation information generating unit 106 outputs, as explanation information, the presentation edge selected for each user and a plurality of presentation edges arranged in the order according to the priority factors of the users. Then, the process ends.

[0081] (C) Effects In this way, according to the information processing device 1 as one embodiment of the present invention, the information generating unit 108 generates explanatory information (presentation edges) selected or arranged for each user according to the priority factors of each user. This makes it possible to increase the understanding and acceptability of predictions made by the machine learning model for each user, thereby improving the reliability of the entire XAI system. In addition, improving the reliability of the entire XAI system will enable users to expand the range of demand for XAI output.

[0082] When the user's priority factor (user-oriented data) is categorical data, the information generating unit 108 extracts categorical data generating edges as candidate edges for presentation, and selects the edge with the highest contribution from among the extracted candidate edges for presentation as the edge for presentation. As a result, the explanatory information presented to the user includes edges reflecting factors that the user believes will contribute to predictions made by the machine learning model, and explanatory information that matches the user's knowledge and insight can be generated. Therefore, it is possible to make the information explainable to the user.

[0083] Furthermore, when the user's priority factor (user-oriented data) is numerical data, the information generating unit 108 extracts numerical data generating edges as presentation candidate edges, and selects a presentation edge from among the extracted presentation candidate edges based on two parameters, the value of the user's priority factor and the contribution degree. As a result, the explanatory information presented to the user includes edges reflecting factors that the user believes will contribute to the prediction by the machine learning model, and it is possible to generate explanatory information that matches the user's knowledge and insight. Therefore, it is possible to make it possible to provide an explanation to the user.

[0084] The information generating unit 108 obtains each value of the priority factor overtime and each value of the contribution degree from among a plurality of presentation candidate edges, normalizes the overtime value and the contribution degree value of each edge, and calculates the sum of these normalized overtime values ​​and contribution degree values ​​for each edge. This makes it possible to easily select a presentation edge based on two parameters, the value of the priority factor of the user and the contribution degree.

[0085] (D) Other The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

[0086] For example, in the above-described embodiment, an example was given of a task used in the field of health management, in which a machine learning model is used to perform binary classification of future leave of absence (whether or not leave of absence) based on attendance data, but this is not limited to this and can be implemented in various modified forms. Furthermore, in the above-described embodiment, an example in which the number of priority factors N=1 has been shown, but the number of priority factors is not limited to this, and may be multiple.

[0087] In the above embodiment, an example of attendance data in which input data is represented as a table is shown, but the present invention is not limited to this. The input data may have a format other than a table, such as a graph. In addition, the input data may be data other than a table or graph that can be handled as tensor data, and may be implemented in various modifications. Furthermore, the above disclosure enables a person skilled in the art to implement and manufacture the present embodiment.

[0088] (E) Notes (Appendix 1) Calculating the contribution of each of the plurality of data to the prediction result by using an explanatory model generated based on a prediction result output by a machine learning model in response to input of a plurality of data each including a plurality of variables and the plurality of data; Selecting a specific variable from the plurality of variables, and determining the specific data from the plurality of data based on the value of the specific variable of each of the plurality of data and the contribution degree; outputting the specific data as explanatory information of the prediction result; An explanatory information output program that causes a computer to execute a process.

[0089] (Appendix 2) The process of selecting the specific variable includes a process of selecting the specific variable from the plurality of variables based on a priority variable according to a user. 2. An explanatory information output program according to claim 1.

[0090] (Appendix 3) The process of determining the specific data includes, when the specific variable is numerical data, a process of determining the specific data from among a plurality of presentation candidate data, the contribution degree of which is greater than a contribution degree threshold value, by using a value of the specific variable and the contribution degree. 2. An explanatory information output program according to claim 1.

[0091] (Appendix 4) the process of determining the specific data includes a process of determining, as the specific data, data having a largest sum of a normalized value of the specific variable value extracted from the plurality of presentation candidate data and a normalized value of the contribution degree of each of the plurality of presentation candidate data. 4. An explanatory information output program according to claim 3.

[0092] (Appendix 5) In the process of determining the specific data, when the specific variable is categorical data, among the plurality of pieces of data, a plurality of pieces of presentation candidate data in which a value of the specific variable indicates a specific category and the contribution degree is greater than a contribution degree threshold value is selected as the specific data, and the data having the highest contribution degree is determined as the specific data. 2. The explanatory information output program according to claim 1, characterized in that the program causes the computer to execute processing.

[0093] (Appendix 6) Calculating the contribution of each of the plurality of data to the prediction result by using an explanatory model generated based on a prediction result output by a machine learning model in response to input of a plurality of data each including a plurality of variables and the plurality of data; Selecting a specific variable from the plurality of variables, and determining the specific data from the plurality of data based on the value of the specific variable of each of the plurality of data and the contribution degree; outputting the specific data as explanatory information of the prediction result; A method for outputting explanatory information, the processing of which is executed by a computer.

[0094] (Appendix 7) The process of selecting the specific variable includes a process of selecting the specific variable from the plurality of variables based on a priority variable according to a user. 7. The explanatory information output method according to claim 6,

[0095] (Appendix 8) The process of determining the specific data includes, when the specific variable is numerical data, a process of determining the specific data from among a plurality of presentation candidate data, the contribution degree of which is greater than a contribution degree threshold, by using a value of the specific variable and the contribution degree. 7. An explanatory information output method according to claim 6,

[0096] (Appendix 9) the process of determining the specific data includes a process of determining, as the specific data, data having a largest sum of a normalized value of the specific variable value extracted from the plurality of presentation candidate data and a normalized value of the contribution degree of each of the plurality of presentation candidate data. 9. An explanatory information output method according to claim 8,

[0097] (Appendix 10) In the process of determining the specific data, when the specific variable is categorical data, among the plurality of pieces of data, a plurality of pieces of presentation candidate data in which a value of the specific variable indicates a specific category and the contribution degree is greater than a contribution degree threshold value is selected as the specific data, and the data having the highest contribution degree is determined as the specific data. 7. The explanatory information output method according to claim 6, characterized in that the processing is executed by the computer.

[0098] (Appendix 11) Calculating the contribution of each of the plurality of data to the prediction result by using an explanatory model generated based on a prediction result output by a machine learning model in response to input of a plurality of data each including a plurality of variables and the plurality of data; Selecting a specific variable from the plurality of variables, and determining the specific data from the plurality of data based on the value of the specific variable of each of the plurality of data and the contribution degree; outputting the specific data as explanatory information of the prediction result; An explanatory information output device comprising a processing unit.

[0099] (Appendix 12) The processing unit, The specific variable is selected from among the plurality of variables based on a priority variable according to a user as a selection process for the specific variable. 12. The explanatory information output device according to claim 11.

[0100] (Appendix 13) The processing unit, In the process of determining the specific data, when the specific variable is numerical data, the specific data is determined from among a plurality of presentation candidate data, among the plurality of data, whose contribution degree is greater than a contribution degree threshold, by using the value of the specific variable and the contribution degree. 12. An explanatory information output device according to claim 11,

[0101] (Appendix 14) The processing unit, In the process of determining the specific data, data having the largest sum of a normalized value of the specific variable extracted from the plurality of presentation candidate data and a normalized value of the contribution degree of each of the plurality of presentation candidate data is determined as the specific data. 14. An explanatory information output device according to claim 13.

[0102] (Appendix 15) The processing unit, In the process of determining the specific data, when the specific variable is categorical data, among the plurality of pieces of data, a plurality of pieces of presentation candidate data in which a value of the specific variable indicates a specific category and the contribution degree is greater than a contribution degree threshold value, the data having the highest contribution degree is determined as the specific data. 12. An explanatory information output device according to claim 11, [Explanation of symbols]

[0103] 1. Information processing device 11 Processors 12. Memory 13 Storage device 14 Graphics Processing Unit 14a Monitor 15 Input Interface 15a Keyboard 15b Mouse 16 Optical drive device 16a Optical disc 17 Device connection interface 17a Memory Device 17b Memory Reader / Writer 17c Memory Card 18 Network Interface 18a Network 19 Bus 101 Processing section 102 Machine Learning Department 103 Reasoning part 104 Interpretable Model Generation Unit 106 Explanation information generation unit 107 Priority factor judgment unit 108 Information generation section

Claims

1. Input data consisting of multiple data, each of which has multiple variables, is input into a machine learning model to obtain a prediction result; Calculating a contribution of each of the plurality of data to the prediction result; identifying a first variable selected by a particular user among the plurality of variables; determining one or more pieces of data that are explanatory information for the prediction result from the plurality of pieces of data based on the contribution degree and a value of only the first variable among the plurality of variables; outputting the one or more pieces of data; An explanatory information output program that causes a computer to execute a process.

2. The process of determining the one or more pieces of data includes, when the first variable is numerical data, a process of determining the one or more pieces of data from among a plurality of pieces of candidate presentation data, the contribution degree of which is greater than a contribution degree threshold, by using a value of the first variable and the contribution degree.

2. The explanatory information output program according to claim 1,

3. the process of determining the one or more pieces of data includes a process of determining, as the one or more pieces of data, data having a largest sum of a normalized value of the first variable value extracted from the plurality of presentation candidate data and a normalized value of the contribution degree of each of the plurality of presentation candidate data.

3. The explanatory information output program according to claim 2,

4. In the process of determining the one or more pieces of data, when the first variable is categorical data, among the plurality of pieces of data, a piece of candidate presentation data in which a value of the first variable indicates a specific category and the contribution degree is greater than a contribution degree threshold is selected as the one or more pieces of data, and the piece of candidate presentation data having the highest contribution degree is determined as the one or more pieces of data.

2. The explanatory information output program according to claim 1, which causes the computer to execute a process.

5. Input data consisting of multiple data, each of which has multiple variables, is input into a machine learning model to obtain a prediction result; Calculating a contribution of each of the plurality of data to the prediction result; identifying a first variable selected by a particular user among the plurality of variables; determining one or more pieces of data that are explanatory information for the prediction result from the plurality of pieces of data based on the contribution degree and a value of only the first variable among the plurality of variables; outputting the one or more pieces of data; A method for outputting explanatory information, the processing of which is executed by a computer.

6. Input data consisting of multiple data, each of which has multiple variables, is input into a machine learning model to obtain a prediction result; Calculating a contribution of each of the plurality of data to the prediction result; identifying a first variable selected by a particular user among the plurality of variables; determining one or more pieces of data that are explanatory information for the prediction result from the plurality of pieces of data based on the contribution degree and a value of only the first variable among the plurality of variables; outputting the one or more pieces of data; An explanatory information output device comprising a processing unit.

Images