Information processing method, program, and information processing device

Abstract

The present invention provides an information processing method and the like that support a user to interpret the behavior of a generated machine learning model. The information processing method executes, by a computer, a plurality of sets of description data vectors xn inputted to an existing machine learning model (21) and target data vectors yn outputted from the machine learning model (21) in association with each other; calculating an explanatory matrix which is a vector product of a generalized inverse matrix of an explanatory matrix X in which a plurality of sets of explanatory data vectors xn are arranged and a target matrix Y in which target data vectors yn are arranged in an order corresponding to the explanatory data vectors X; and outputting a graph (41, 42, 43) about the interpretation matrix.

Description

Technical Field

[0001] The present invention relates to an information processing method, a program, and an information processing apparatus. Background Art

[0002] A system has been proposed that classifies images using a machine learning model and displays regions that contribute to the classification using a heat map (Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2023-83555 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] The machine learning model generated by machine learning is a black box, and it is difficult for users to decipher its behavior.

[0008] One object of the present invention is to provide an information processing method, etc., that supports a user in interpreting a generated machine learning model.

[0009] Means for solving problems

[0010] The information processing method is performed by a computer to perform the following processing: recording multiple groups of explanatory data vectors (explanatory data vectors) input into an existing machine learning model and target data vectors output from the machine learning model in association with each other; calculating an interpretation matrix (interpretation matrix), which is the vector product of the explanatory matrix (explanatory matrix) and the generalized inverse matrix of the target matrix, the explanatory matrix arranges multiple groups of the explanatory data vectors, and the target matrix arranges the target data vectors in an order corresponding to the explanatory data vectors; and outputting a chart about the interpretation matrix.

[0011] Effects of the Invention

[0012] On the one hand, the present invention can provide information processing methods, etc. that support users in interpreting the behavior of generated machine learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is an explanatory diagram illustrating the outline of AIME.

[0014] Figure 2 is the explanation matrix This figure illustrates the calculation method of .

[0015] Figure 3 This is an explanatory diagram illustrating the configuration of an information processing device.

[0016] Figure 4 This is to explain how to calculate the explanation matrix Flowchart of the processing flow of the program.

[0017] Figure 5 This is an explanatory diagram for explaining the outline of a similarity distribution map creation method.

[0018] Figure 6 This is a flowchart illustrating the processing flow of a program for generating a similarity distribution map.

[0019] Figure 7 This is an explanatory diagram that explains the record structure of the description-target database.

[0020] Figure 8 is an illustration of the global feature importance map.

[0021] Figure 9A is an example of a local feature importance map.

[0022] Figure 9B is an example of a local feature importance map.

[0023] Figure 10 is an example of a similarity distribution graph.

[0024] Figure 11 is an example of a similarity distribution graph.

[0025] Figure 12 is an example of a similarity distribution graph.

[0026] Figure 13 This is an explanatory diagram explaining the importance of local features in the existing model.

[0027] Figure 14 Here is an example of a similarity distribution graph for existing models. DETAILED DESCRIPTION

[0028] [Implementation Method 1]

[0029] Machine learning models that accept explanatory data as input and output target data can be generated using various machine learning algorithms. These models are often black boxes, meaning the process by which they determine the target data from the explanatory data input is difficult for humans to interpret.

[0030] However, when using machine learning models to make decisions in the real world, it is crucial for humans to be able to explain the model's judgment process. For example, if the output target data appears to deviate significantly from common sense, if humans can correctly explain the machine learning model's judgment process, they can also correctly judge how to treat the target data and the machine learning model.

[0031] Explainable AI (XAI) is a technology that explains the rationale behind the target data output by machine learning models. For example, well-known XAI techniques include LIME (Local Interpretable Model Agnostic Explanations), which is suitable for explaining the local behavior of machine learning models, and SHAP (Shapley Additive Explanation), which is suitable for evaluating the importance of explanatory variables.

[0032] In this embodiment, an information processing method that supports users in interpreting the behavior of machine learning models from various perspectives is described. The information processing method described below is called AIME (Approximate Inverse Model Explanations).

[0033] Figure 1 This is an explanatory diagram that explains the outline of AIME. Figure 1 The following describes the procedure for explaining the behavior of the existing model 21, which is a generated machine learning model, using AIME. The existing model 21 is a machine learning model that accepts an explanation data vector xn as input and outputs a target data vector yn.

[0034] The existing model 21 is a machine learning model generated using any supervised machine learning algorithm, such as a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or a random forest. The existing model 21 may also be a machine learning model generated using any unsupervised machine learning algorithm, such as reinforcement learning or a GAN (Generative Adversarial Network).

[0035] The process of inputting the explanatory data vector xn into the existing model 21 and obtaining the target data vector yn is repeated multiple times. If the existing model 21 is generated through supervised machine learning, the explanatory data vector xn may use part or all of the training data used for the machine learning. The explanatory data vector xn may also be generated randomly or based on a predetermined rule.

[0036] By arranging multiple description data vectors xn in the row direction, a description matrix X, which is a two-dimensional matrix, can be generated. Similarly, by arranging multiple target data vectors yn in the row direction, a target matrix Y, which is a two-dimensional matrix, can be generated. Here, the arrangement order of the description data vectors xn is the same as the arrangement order of the corresponding target data vectors yn.

[0037] In order to perform the subsequent process, the target data vector yn needs to be linearly independent. That is, the vector product of the target matrix Y and the transposed matrix of the target matrix Y needs to be a regular matrix.

[0038] Calculate the explanation matrix based on the description matrix X and the target matrix Y As shown in formula (1), the interpretation matrix The vector product with the target matrix Y is equal to the explanation matrix X. Calculate the explanation matrix The method will be described in detail later.

[0039]

[0040] By converting the interpretation matrix Graphs can be created, such as a global feature importance graph 41 showing the overall behavior of the existing model 21, a local feature importance graph 42 showing the solution of each target data vector yn, and a similarity distribution graph 43 showing the similarity distribution between explanatory data vectors xn. These graphs allow users to interpret the behavior of the existing model 21. The global feature importance graph 41, local feature importance graph 42, and similarity distribution graph 43 will be described in detail later.

[0041] Figure 2 is the explanation matrix In the following description, N is a natural number representing the number of repetitions of the process of inputting the explanatory data vector xn into the existing model 21 and obtaining the target data vector yn. n is a natural number indicating the number of times the vector has been input into or output from the existing model 21.

[0042] It is shown that the data vector xn has L elements from Ex1n to ExLn. The target data vector yn has M elements from Ob1n to ObMn. Here, L and M are natural numbers. Figure 2 In FIG. 1 , the explanatory data vector xn and the target data vector yn for n=2 are shown surrounded by dotted lines.

[0043] As described above, by arranging the N description data vectors xn obtained by N processing steps in the row direction, a description matrix X, which is a two-dimensional matrix, can be created. Description matrix X is a two-dimensional matrix with L rows and N columns. Similarly, by arranging the N target data vectors yn in the row direction, a target matrix Y, which is a two-dimensional matrix, can be generated. The target matrix It is a two-dimensional matrix with M rows and N columns.

[0044] For the target matrix Y, calculate the Moore-Penrose generalized inverse In the following descriptions, sometimes Recorded as the target inverse matrix Target inverse matrix Calculated by formula (2). The target inverse matrix is a matrix with N rows and M columns.

[0045]

[0046] Explanation Matrix It is to explain the relationship between matrix X and target inverse matrix Calculate the explanation matrix The formula is shown in formula (3).

[0047]

[0048] Explanation Matrix is a two-dimensional matrix with L rows and M columns, that is, it has the same number of rows as the number of elements in the explanatory data vector xn and the same number of columns as the number of elements in the target data vector yn. The elements in row a and column b of represent the influence of the a-th element of the data vector xn on the b-th element of the target data vector yn.

[0049] As described above, even if there is no information such as the algorithm and training data for generating the existing model 21, as long as there is an environment that can utilize the existing model 21, an explanation matrix can be generated. That is, even if the existing model 21 is a black box state generated by a third party, the explanation matrix

[0050] For reference, the following is a summary of the transformation from equations (1) and (2) to equation (3). First, multiply both sides of equation (1) by the transposed matrix of the target matrix Y to obtain equation (4).

[0051]

[0052] As mentioned above, the cross product of the target matrix Y and the transposed matrix of the target matrix Y is a regular matrix, so the inverse matrix can be calculated. Multiplying the inverse matrix on both sides of equation (4) yields equation (5).

[0053]

[0054] By swapping the left and right sides of equation (5), and substituting the right side into equation (2), we can obtain equation (6). Equation (3) can be derived from both ends of equation (6).

[0055]

[0056] Figure 3 1 is an explanatory diagram illustrating the configuration of the information processing device 10. The information processing device 10 includes a control unit 11, a main storage device 12, an auxiliary storage device 13, a communication unit 14, a display unit 15, an input unit 16, a reading unit 19, and a bus.

[0057] The control unit 11 is a computational control device that executes the program of this embodiment. One or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), or multi-core CPUs can be used as the control unit 11. The control unit 11 is connected to the various hardware components that make up the information processing device 10 via a bus.

[0058] The main storage device 12 is a storage device such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores information required for processing by the control unit 11 and programs currently running in the control unit 11 .

[0059] The auxiliary storage device 13 is a storage device such as SRAM, flash memory, hard disk, or magnetic tape. The auxiliary storage device 13 stores the existing model 21, the program that runs the control unit 11, and various data required for program execution. The existing model 21 can also be stored in an external storage device connected via a network.

[0060] The communication unit 14 is an interface for communication between the information processing device 10 and a network. The display unit 15 is, for example, a liquid crystal display device or an organic EL (Electro Luminescence) display device. The input unit 16 is, for example, an input device such as a keyboard, a mouse, a trackball, or a microphone.

[0061] The portable recording medium 96 is, for example, a USB (Universal Serial Bus) memory, a CD-ROM (Compact Disc Read Only Memory), a magneto-optical medium, other optical disc media, or an SD memory card, etc. The portable recording medium 96 stores a program 97 for implementing AIME.

[0062] The reading unit 19 is an interface to which the portable recording medium 96 can be connected, and is, for example, a USB connector, a CD-ROM drive, or an SD memory card reader. The semiconductor memory 98 stores the program 97 and is a memory that can be installed in the information processing device 10 .

[0063] The information processing device 10 is a general-purpose personal computer, tablet computer, mainframe computer, virtual machine running on a mainframe computer, or quantum computer. The information processing device 10 can be composed of hardware such as multiple personal computers or mainframe computers that perform distributed processing. The information processing device 10 can also be composed of a cloud computing system. The information processing device 10 can also be composed of hardware such as multiple personal computers or mainframe computers that operate in coordination with each other.

[0064] The program 97 is recorded in a portable recording medium 96. The control unit 11 reads the program 97 via the reading unit 19 and stores it in the auxiliary storage device 13. The control unit 11 can also read the program 97 stored in the semiconductor memory 98. In addition, the control unit 11 can also download the program 97 from another server computer (not shown) connected via the communication unit 14 and a network (not shown) and store it in the auxiliary storage device 13.

[0065] The program 97 is installed as a control program for the information processing device 10, loaded into the main storage device 12, and executed. The program 97 of this embodiment is an example of a program product.

[0066] Figure 4 This is to explain how to calculate the explanation matrix Flowchart of the processing flow of the program. The control unit 11 determines the explanatory data vector xn to be input to the existing model 21 for the nth time (step S501). If the training data used for machine learning of the existing model 21 is available, the control unit 11 may extract the explanatory data vector xn from the training data.

[0067] The control unit 11 may generate the explanatory data vector xn randomly or according to a predetermined rule. When the control unit 11 generates the explanatory data vector xn, it is preferable to generate the explanatory data vector xn within the range assuming the use of the existing model 21. For example, if an explanatory data vector xn containing an unexpected element, such as "the age of a human is 200 years old," is generated, the behavior of the existing model 21 cannot be correctly interpreted.

[0068] Likewise, the distribution of the plurality of explanatory data vectors xn preferably matches the range assumed for the use of the existing model 21. For example, in the existing model 21 for predicting the behavior of an average adult, the element "human age" preferably has a distribution close to data such as population dynamics.

[0069] The control unit 11 inputs the explanatory data vector xn obtained in step S501 into the existing model 21 to obtain the target data vector yn (step S502). The control unit 11 records the explanatory data vector xn and the target data vector yn in association with each other in the main storage device 12 or the auxiliary storage device 13 (step S503).

[0070] The control unit 11 determines whether to complete the generation of pairs of the explanatory data vector xn and the target data vector yn (step S504). For example, if the processing of the explanatory data vector xn recorded in the training data has been completed, the control unit 11 determines in step S504 that the processing is complete. Alternatively, the control unit 11 may terminate the processing after a predetermined number of pairs have been generated.

[0071] If it is determined that the process is not finished ("No" in step S504), the control unit 11 returns to step S501. If it is determined that the process is finished ("Yes" in step S504), the control unit 11 generates the description matrix X based on the data recorded in step S503 (step S505). The control unit 11 generates the target matrix Y based on the data recorded in step S503 (step S506). The control unit 11 calculates the target inverse matrix as the Moore-Penrose generalized inverse matrix based on the target matrix Y. (Step S507) The control unit 11 calculates the explanatory matrix X and the target inverse matrix Interpretation matrix of the vector product (Step S508 ) The control unit 11 ends the processing.

[0072] The following explains the generated explanation matrix An overview of the method used to explain the behavior of the existing model 21 graphically.

[0073] [Global Feature Importance]

[0074] Global feature importance refers to the degree to which each element constituting the explanatory data vector xn contributes to a specific prediction result to be output by the existing model 21. The user can understand the overall behavior of the existing model 21 based on the visualized global feature importance.

[0075] As mentioned before, the interpretation matrix The elements in row a and column b of the explanatory data vector xn represent the influence of the ath element of the explanatory data vector xn on the bth element of the target data vector yn. For example, the degree of influence of each element of the explanatory data vector xn on a specific item in the target data vector yn can be visualized using a bar chart or line chart, where the first axis of the bar chart or line chart represents the item name corresponding to each element of the explanatory data vector xn, and the second axis represents the explanation matrix. The specific example of a chart that visualizes the global feature importance will be described later.

[0076] [Representative presumption examples]

[0077] The representative estimated example x* refers to an ideal or typical explanatory data vector xn in a specific state of the target data vector yn to be output from the existing model 21. The representative estimated example x* is a vector having the same number of elements as the explanatory data vector xn.

[0078] The representative inferred instance x* is given by the explanation matrix The vector product of the target data vector yn is calculated. The formula for calculating the representative estimation example x* is shown in formula (7).

[0079]

[0080] In the following description, the case where the target data vector yn is a unit target vector yuk, whose kth element is 1 and all other elements are 0, is used as an example. K is a natural number equal to or less than the number of elements in the target data vector yn. When the target data vector yn is the unit target vector yuk, the formula for calculating the representative estimation example x* is shown in Equation (8), and the definition of the i-th element yuk(i) of the unit target vector yuk is shown in Equation (9).

[0081]

Number 1

[0082]

[0083] The representative inference example x* allows the user to understand the importance of each item in the explanatory data vector xn when a specific target data vector yn is output. Furthermore, the representative inference example x* allows the user to obtain clues for explaining the behavior of the existing model 21. A specific example of the representative inference example x* will be described later.

[0084] [Local feature importance]

[0085] The local feature importance vector L indicates the degree of contribution of each element constituting a specific explanatory data vector xn to the target data vector yn output from the existing model 21. The local feature importance vector L is a vector having the same number of elements as the explanatory data vector xn.

[0086] In the following description, a specific description data vector xn is referred to as an object description vector xo. The object description vector xo can be included in the training data or created separately from the training data. This description uses the example of a target data vector yn being a unit target vector yuk. The local feature importance vector L is calculated using Equation (10).

[0087]

Number 2

[0088]

[0089] ° represents the Hadamard product.

[0090] For example, the local feature importance vector L can be visualized using a bar graph or a line graph, wherein the first axis of the bar graph or line graph represents the item name corresponding to each element constituting the local feature importance vector L, that is, the item name corresponding to each element constituting the explanatory data vector xn, and the second axis represents the value of each element constituting the local feature importance vector L. A specific example of a graph for visualizing the local feature importance vector L will be described later.

[0091] The user can use the local feature importance vector L to explain why the existing model 21, which received the explanatory data vector xn as input, outputs the target data vector. The user can also use the local feature importance vector L to understand the elements of the explanatory data vector xn that influence the target data vector yn. A specific example of the local feature importance vector L will be described later.

[0092] [Similarity distribution graph]

[0093] Figure 5 This is an explanatory diagram for explaining the outline of the method for creating the similarity distribution map 43. First, two representative estimated instances x*, a first representative example vector and a second representative example vector, are calculated according to equation (8). Different k is used when calculating the first representative example vector and the second representative example vector.

[0094] An explanation data vector xn that satisfies a specific condition is selected from the explanation matrix X. For example, when training data used for machine learning is used for the explanation data vector xn, the explanation data vector xn corresponding to the unit target vector yuk used to calculate the first representative example vector is preferably selected as the correct answer data. Alternatively, an explanation data vector xn whose kth element in the target data vector yn to be output from the existing model 21 is greater than a predetermined threshold value may be selected.

[0095] For the selected explanatory data vector xn, a first similarity is calculated, that is, the similarity with the first representative example vector. Similarity can be calculated using, for example, the reciprocal of an arbitrarily defined inter-vector distance, such as the Euclidean distance, Manhattan distance, or Hamming distance between the explanatory data vector xn and the first representative example vector. Cosine similarity can also be used, for example.

[0096] Similarly, the second similarity is calculated, that is, the similarity between the selected explanation data vector xn and the second representative example vector. Each explanation data vector xn is plotted on a two-dimensional graph with the first similarity value on the x-axis and the second similarity value on the y-axis.

[0097] Kernel density estimation, for example using an RBF kernel, is performed on the plotted data to generate a smoothed similarity distribution graph 43. The user can intuitively understand the behavior of the existing model 21 associated with the selected explanatory data vector xn from the similarity distribution graph 43. Alternatively, each point can be displayed directly without performing kernel density estimation.

[0098] Kernel density estimation can also be performed when plotting the individual explanatory data vectors xn. A smoothed similarity distribution map 43 can be generated by superimposing the individual points.

[0099] The similarity distribution map 43 created using the explanation data vectors xn selected under a plurality of conditions may be displayed in a superimposed manner. Alternatively, a three-dimensional similarity distribution map 43 may be created using three representative estimation examples x*. A specific example of the similarity distribution map 43 will be described later.

[0100] Figure 6 This is a flowchart illustrating the processing flow of the program for generating the similarity distribution graph 43. The control unit 11 receives a designation of a first explanatory data vector xn (step S521). For example, the control unit 11 receives a designation of k and uses a unit target vector yuk as the first explanatory data vector xn. The unit target vector yuk is an example of a first unit vector. The control unit 11 calculates a representative estimated instance x* according to equation (7). The control unit 11 uses the calculated representative estimated instance x* as a first typical example vector (step S522).

[0101] The control unit 11 receives a designation for the second explanatory data vector xn (step S523). For example, the control unit 11 receives a designation for k and uses the unit target vector yuk as the second explanatory data vector xn. This unit target vector yuk is an example of the second unit vector. The control unit 11 calculates the representative estimated instance x* according to equation (7). The control unit 11 uses the calculated representative estimated instance x* as the second typical example vector (step S524).

[0102] The control unit 11 selects a description data vector xn that meets the specified conditions from the description matrix X (step S525). The control unit 11 calculates the similarity between the description data vector xn selected in step S525 and the first representative example vector calculated in step S522 (step S526). The control unit 11 calculates the similarity between the description data vector xn selected in step S525 and the second representative example vector calculated in step S524 (step S527). The control unit 11 records the similarity calculated in step S526 in association with the similarity calculated in step S527 in the main storage device 12 or the auxiliary storage device 13 (step S528).

[0103] The control unit 11 determines whether to terminate the similarity calculation (step S529). Specifically, for example, the control unit 11 determines to terminate the similarity calculation when processing of the description data vector xn that satisfies a predetermined condition in the description matrix X has been completed. Alternatively, the control unit 11 may determine to terminate the similarity calculation when the similarity calculation has been repeated a predetermined number of times.

[0104] If the process is determined not to be completed ("No" in step S529), the control unit 11 returns to step S525. If the process is determined to be completed ("Yes" in step S529), the control unit 11 creates a scatter plot for each data recorded in step S528, where the first axis of the scatter plot represents the similarity with the first representative example vector and the second axis represents the similarity with the second representative example vector (step S530).

[0105] The control unit 11 performs kernel density estimation on the scatter plot generated in step S530 to smooth the distribution of each point in the scatter plot (step S531). Through the above steps, the similarity distribution graph 43 is completed. The control unit 11 displays the completed similarity distribution graph 43 (step S532). Then, the control unit 11 ends the processing.

[0106] [Specific example 1: Titanic dataset]

[0107] An example of explaining an existing model 21 is described, which is generated using the Titanic dataset as training data, which summarizes the survival status of passengers on the Titanic, which sank due to an iceberg in the North Atlantic in 1912.

[0108] Conventional model 21 is a learning model trained to output the survival and non-survival probabilities of each passenger, given a 12-element descriptive data vector xn. The technique for generating a machine learning model using supervised machine learning is well known, so the generation method is omitted here. Table 1 shows the elements of the descriptive data vector xn.

[0109]

Table 1

[0110]

[0111] In Table 1 , the eight items sent from first-class to Port S are represented by binary values: 1 for yes and 0 for no.

[0112] Figure 7This diagram illustrates the record structure of the specification-target database 31. The specification-target database 31 includes a number field, a specification data vector field, and a target data vector field. The specification data vector field includes subfields corresponding to the elements of the specification data vector xn shown in Table 1, such as the Age field, the SibSp field, and the Parch field. The target data vector field includes a survival probability field and a non-survivor probability field.

[0113] The Number field records consecutive numbers from 1 to N. Each subfield of the Description Data field records the elements of the Description Data vector xn. The Survival Probability field records the probability that a passenger with the attributes of the Description Data vector xn survived. The Non-Survival Probability field records the probability that a passenger with the attributes of the Description Data vector xn did not survive. For each record, the sum of the values ​​recorded in the Survival Probability field and the Non-Survival Probability field is 1.

[0114] In use Figure 4 In step S503 of the description procedure, the control unit 11 adds one record to the description-target database 31 . Figure 7 The part surrounded by dotted lines is Figure 2 The shown illustration is the transposed matrix of matrix X. Figure 7 The part surrounded by solid lines is Figure 2 The transposed matrix of the target matrix Y is shown.

[0115] Figure 8 is an illustration of the global feature importance diagram 41. Figure 8 The upper side schematically shows the use of Figure 4 The interpretation matrix calculated in step S508 of the described procedure Explanation Matrix Is a matrix with 12 rows and 2 columns. Explain the matrix The vertical number of the elements, that is, the third word is expressed in hexadecimal. That is, A represents 10, B represents 11, and C represents 12.

[0116] As mentioned before, the interpretation matrix The elements in row a and column b of the data vector xn represent the influence of the ath element of the data vector xn on the bth element of the target data vector yn. Therefore, AD11 represents the influence of the passenger's age on the passenger's survival, and AD12 represents the influence of the passenger's age on the passenger's failure to survive.

[0117] Figure 8 In FIG, the global feature importance graph 41 is shown at the bottom. The vertical axis represents the item name of the explanation data vector xn. The horizontal axis represents the explanation matrix The hatched line that slopes downward to the left represents the explanation matrix The values ​​of the column on the left of , that is, the influence of each element on the passenger's survival. The hatched line sloping downward to the right represents the explanation matrix The values ​​in the column to the right of , that is, the influence of each element on the passenger not surviving.

[0118] according to Figure 8 , the number of passengers A has little effect on the passenger's survival. Therefore, it can be seen that data on the number of passengers A is not necessary to predict passenger survival. For example, the user can reduce the computational complexity of the existing model 21 by removing the number of passengers A from the explanatory variables without affecting prediction accuracy.

[0119] Similarly, according to Figure 8 , the factor that has the greatest impact on whether a passenger can survive is that the passenger is female, and the factor that has the greatest impact on whether a passenger can survive is that the passenger is male. It can be seen that the passenger's gender has a greater impact on whether or not to survive than other factors.

[0120] Table 2 shows representative estimation examples involving survivors and non-survivors calculated according to formula (8).

[0121]

Table 2

[0122]

[0123]

[0124] As can be seen from Table 2 , the typical survivor is a female with a higher passenger fare, and the typical non-survivor is a male with a lower passenger fare.

[0125] Next, we'll describe an example of local feature importance. To calculate local feature importance, we created two object description vectors xo: Rose, a survivor, and Jack, a non-survivor, based on the characters from the 1997 film Titanic. The generated object description vectors xo are shown in Table 3.

[0126]

Table 3

[0127] Name (Chinese) Ross Jack age 17 19 Number of passengers for Type A 0 0 Number of passengers for Type B 1 0 Passenger fares 33.3 0.0 First Class 1 0 Second Class 0 0 Third Class 0 1 female 1 0 male 0 1 Departure from Port C 0 0 Departure from Port Q 0 0 Departure from Port S 0 0

[0128] Figure 9A and Figure 9B is an example of the local feature importance diagram 42. The local feature importance vector L for Ross's survival calculated according to formula (10) is shown in Figure 9A Figure 42 shows the importance of local features. The unit target vector yuk uses the transposed matrix of (1, 0). The vertical axis represents the item name of the object description vector xo. The horizontal axis represents the value of each element of the local feature importance vector L. Figure 9A , the first reason Rose survived was that she was a woman, and the second reason was that she was a first-class passenger.

[0129] The local feature importance vector L calculated according to formula (10) about Jack not surviving is shown in Figure 9B Figure 42 shows the importance of local features. The unit target vector yuk uses the transposed matrix of (0, 1). The vertical axis represents the item name of the object description vector xo. The horizontal axis represents the value of each element of the local feature importance vector L. Figure 9B The first reason why Jack could not survive was that he was a man, and the second reason was that he was a third-class passenger.

[0130] like Figure 9A and Figure 9B As shown, the explanation matrix can be used A local feature importance map 42 is plotted for any object description vector xo. The user can also use the local feature importance map 42 to resolve description data vectors xn that are not included in the description matrix X.

[0131] Figures 10 to 12 This is an example of a similarity distribution map 43 . Figure 10 A similarity distribution diagram 43 is shown for the survivors. Figure 10 The control unit 11 uses Figure 6 The first representative example vector calculated in step S522 of the described procedure is equal to the representative estimated examples of non-survivors shown in Table 2. The second representative example vector calculated by the control unit 11 in step S524 is equal to the representative estimated examples of survivors shown in Table 2. In step S525, the control unit 11 selects the explanation data vector xn for the survivors from the explanation matrix X.

[0132] Figure 10 The horizontal axis of represents the non-survivor score corresponding to the similarity with the representative presumed instance of the non-survivor. The higher the non-survivor score, the more similar it is to the representative presumed instance of the non-survivor. Figure 10 The vertical axis represents the survival score, which corresponds to the similarity with the representative presumed survivor instance. A higher survival score indicates a greater similarity with the representative presumed survivor instance. The denser the hatching, which slopes downward to the left, the denser the distribution of the explanatory data vector xn about the survivor. Figure 10 is an example of a first distribution graph.

[0133] Figure 11 A similarity distribution diagram 43 is shown for non-survivors. Figure 11 Overview of the processing. Figure 6 The processing and production of steps S522 and S524 of the described procedure Figure 10 In step S525, the control unit 11 selects the explanation data vector xn about the non-survivors from the explanation matrix X.

[0134] Figure 11 The vertical and horizontal axes are Figure 10 The denser the hatching lines that slope downward to the right, the denser the distribution of the explanation data vector xn for non-survivors. Figure 11 is an example of the second distribution graph.

[0135] Figure 12 Shows the Figure 10 and Figure 11 Overlaid similarity distribution graph 43. Specifically, the distributions for survivors and non-survivors are displayed in a single graph overlaid with each other. The white dots represent the non-survivor and survivor scores for Ross, as shown in Table 3. The black dots represent the non-survivor and survivor scores for Jack, as shown in Table 3.

[0136] according to Figure 12 , there is an area where the distribution of survivors and non-survivors overlap. Users can understand that in this area, it is difficult to predict whether a passenger will survive. Figure 12 This is an example of the similarity distribution graph 43 in which the first distribution graph and the second distribution graph are displayed superimposed on each other.

[0137] When the values ​​of the intermediate layers of the existing model 21 can be obtained, the explanation matrices can be generated from the input layer that receives the explanation data vector xn to the intermediate layer and from the intermediate layer to the output layer that outputs the target data vector yn. By generating an explanation matrix in a hierarchical manner It is possible to improve the interpretability of existing models21.

[0138] [Specific example 2: MNIST (Mixed National Institute of Standards and Technology database)

[0139] This example explains existing model 21, which was generated using the MNIST dataset as training data for classifying handwritten digits. The MNIST dataset associates handwritten digit images, consisting of 784 8-bit grayscale pixels arranged vertically and horizontally, with the correct answer.

[0140] Existing model 21 is a learning model trained to input a grayscale image of 28 pixels vertically and 28 pixels horizontally and output the probabilities of each digit from 0 to 9. The data vector xn is a 784-element vector that arranges the brightness of 784 pixels in a specified order. The target data vector yn is a 10-element vector that arranges the probabilities of each digit from 0 to 9.

[0141] Figure 13 2 is an explanatory diagram illustrating the importance of local features of the existing model 21. Figure 13 As shown in the upper part of the figure, the process of inputting the description data vector xn into the existing model 21 and obtaining the target data vector yn is repeated, and the process of using Figure 2 Explanation Matrix Explanation Matrix It is a matrix with 784 rows and 10 columns.

[0142] exist Figure 13 The lower side of the table shows three types of local feature importance vectors L. The local feature importance vectors L are shown when the digital image of the object description vector xo is input into the existing model 21 and is judged to be 0, 3, and 8. In addition, Figure 13 In the figure, each handwritten character and local feature importance vector L are converted into black and white binary values ​​and displayed. When it is determined to be a corresponding number, the pixel with importance is displayed in black.

[0143] Through the local feature importance vector L, the user can realize that when the object description vector xo is judged to be 0, the upper and lower parts of the handwritten text contribute; when the object description vector xo is judged to be 3, the right part of the handwritten text contributes; when the object description vector xo is judged to be 8, almost the entire handwritten text contributes.

[0144] Figure 14 This is an example of a similarity distribution diagram 43 for the existing model 21. The horizontal axis is the similarity score with the representative inferred instance x* judged as "3". The vertical axis is the similarity score with the representative inferred instance x* judged as "8". The white dots indicate Figure 13 The object shown in the lower right of the figure illustrates where the vector xo is drawn.

[0145] In the MNIST dataset, the distribution of the explanation data vector xn for the correct answer data "8" is shown by hatching that slopes downward to the left, while the distribution of the explanation data vector xn for the correct answer data "3" is shown by hatching that slopes downward to the right. Areas with denser hatching indicate denser distribution.

[0146] The areas for "3" and "8" overlap significantly, so the user can recognize that 3 and 8 are easily misclassified. For example, the areas for "0" and "1" do not overlap, and although this is not shown in the figure, the user can also recognize that "0" and "1" are less likely to be misclassified.

[0147] The existing model 21 may be a learning model that accepts a sentence input and outputs a keyword. The existing model 21 may also be a learning model that accepts an image input and outputs the name of the photographed object. In addition, for any existing model 21, the interpretation matrix can be calculated. Used for behavioral unraveling.

[0148] [Variation 1]

[0149] This variation involves rapidly generating an interpretation matrix The description of the parts common to the first embodiment is omitted. In this variation, the existing model 21 is generated by supervised machine learning.

[0150] The control unit 11 does not repeat the operation of inputting the description data vector xn into the existing model 21 to obtain the target data vector yn, but extracts the pair of the description data vector xn and the target data vector yn from the training data to generate the description matrix X and the target matrix Y.

[0151] According to this modification, the interpretation matrix can be calculated before the existing model 21 is completely generated. For example, users can use the explanation matrix before generating an existing model21 The behavior of the existing model 21 is predicted, and measures such as increasing training data are taken when the existing model 21 does not have the desired characteristics.

[0152] [Variation 2]

[0153] This modification relates to a method of regularizing the target matrix Y using Ridge regression, ie, L2 norm regularization. The description of the parts common to the first embodiment will be omitted.

[0154] In this modification, the control unit 11 uses Figure 4 In step S507 of the flowchart described above, the generalized inverse matrix of Y is calculated using equation (11) instead of equation (2).

[0155]

[0156] λ is a positive real number.

[0157] E is the identity matrix with M rows and M columns.

[0158] According to this modification, an appropriate explanation matrix can be calculated even for a complex existing model 21 having a large number of parameters or interactions between parameters.

[0159] [Variation 3]

[0160] This modification relates to a method of regularizing the target matrix Y using Lasso regression, that is, L1 norm regularization. Detailed descriptions of parts common to the first embodiment are omitted.

[0161] In this modification, the control unit 11 uses Figure 4 In step S507 of the flowchart described above, the generalized inverse matrix of Y is calculated. Then, the generalized inverse matrix of Y is transformed into Regularization.

[0162] According to this modification, it is also possible to calculate an appropriate interpretation matrix for a complex existing model 21 having a large number of parameters or interactions between parameters. Furthermore, the regularization method is not limited to the Ridge regression of Modification 2 and the Lasso regression of this Modification. For example, a known method such as ElasticNet can also be used for regularization.

[0163] A program is an example of a program product. A computer program can be executed on a single computer, on multiple computers deployed at one site, or on multiple computers distributed across multiple sites and interconnected by a communication network.

[0164] The technical features (constituent elements) described in each embodiment can be combined with each other, and new technical features can be formed by the combination.

[0165] The embodiments disclosed herein are illustrative in all respects and should not be construed as restrictive. The scope of the present invention is defined by the claims rather than the above, and is intended to encompass all modifications within the meaning and scope of the claims and equivalents.

[0166] Independent and dependent claims in the claims may be combined in all possible combinations, regardless of the format of citation. Furthermore, the claims employ a format whereby a claim references two or more other claims (multiple citation claim format), but are not limited thereto. Multiple citation claims may also reference at least one multiple citation claim (multiple citation multiple claims).

[0167] Description of Reference Numerals

[0168] 10Information processing device

[0169] 11Control Unit

[0170] 12 Main storage device

[0171] 13 Auxiliary storage device

[0172] 14Ministry of Communications

[0173] 15 Display unit

[0174] 16 Input unit

[0175] 19 reading part

[0176] 21 Existing Models (Existing Machine Learning Models)

[0177] 31 Description-Target Database

[0178] 41 Global Feature Importance Graph (Graph)

[0179] 42 Local Feature Importance Map (Graph)

[0180] 43 Similarity distribution diagram (chart)

[0181] 96 Portable Recording Media

[0182] 97 Program

[0183] 98 semiconductor memory

Claims

1. An information processing method, wherein: The following processing is performed by a computer: recording a plurality of sets of description data vectors input to an existing machine learning model and target data vectors output from the machine learning model in association with each other; Calculating an interpretation matrix, the interpretation matrix being a vector product of a description matrix and a generalized inverse matrix of a target matrix, wherein the description matrix arranges a plurality of the description data vectors, and the target matrix arranges the target data vectors in an order corresponding to the description data vectors; and Output a graph of the explanatory matrix.

2. The information processing method according to claim 1, wherein: The generalized inverse matrix of the target matrix is ​​the Moore-Penrose generalized inverse matrix of the target matrix.

3. The information processing method according to claim 1, wherein: The graph is a graph using the item name corresponding to each element of the explanation data vector as a first axis and the value of the element constituting each column of the explanation matrix as a second axis.

4. The information processing method according to claim 1, wherein: Get the description data vector; generating a unit target vector in the target data vector, in which one element is 1 and the other elements are 0, as an object showing the importance of the local feature; The graph is a graph represented by using the item name corresponding to each element of the explanatory data vector as the first axis and the value of the element constituting the local feature importance vector calculated by equation (1) as the second axis. 【Number 1】 Among them, L is the local feature importance vector; A is the explanation matrix; yuk is the unit target vector whose k-th element is 1; k is a natural number representing an element as an object showing the importance of local features; xo is a vector of explanatory data that serves as an object to show the importance of local features; represents the Hadamard product.

5. The information processing method according to claim 1, wherein: Calculating a first typical example vector, where the first typical example vector is a vector product of the interpretation matrix and a first unit vector of the target data vector, one element of which is 1 and the other elements of which are 0; Calculate a second typical example vector, where the second typical example vector is a vector product of the interpretation matrix and a second unit vector of the target data vector whose other element is 1 and other elements are 0; The graph is a distribution graph obtained by plotting the explanation data vectors selected from the explanation matrix with the similarity to the first typical example vector as the horizontal axis and the similarity to the second typical example vector as the vertical axis, and performing kernel density estimation. The information processing method according to claim 5 , wherein: The description data vector is the same as the description data in the training data used in the machine learning of the machine learning model, The chart is a graph that overlaps a first distribution graph and a second distribution graph, wherein the first distribution graph is made using the correct answer data in the training data and the explanatory data corresponding to an element when calculating the first typical example vector, and the second distribution graph is made using the correct answer data in the training data and the explanatory data corresponding to an element when calculating the second typical example vector.

7. The information processing method according to any one of claims 1 to 5, wherein: The description data vector is the same as the description data in the training data used in the machine learning of the machine learning model.

8. A program, wherein Make the computer execute the following processing: recording a plurality of sets of description data vectors input to an existing machine learning model and target data vectors output from the machine learning model in association with each other; Calculating an interpretation matrix, the interpretation matrix being a vector product of a description matrix and a generalized inverse matrix of a target matrix, wherein the description matrix arranges a plurality of the description data vectors, and the target matrix arranges the target data vectors in an order corresponding to the description data vectors; and Output a graph of the explanatory matrix.

9. An information processing device, wherein: Including the control unit, The control unit, recording a plurality of sets of description data vectors input to an existing machine learning model and target data vectors output from the machine learning model in association with each other; Calculating an interpretation matrix, the interpretation matrix being a vector product of a description matrix and a generalized inverse matrix of a target matrix, wherein the description matrix arranges a plurality of the description data vectors, and the target matrix arranges the target data vectors in an order corresponding to the description data vectors; and Output a graph of the explanatory matrix.

Images