Information processing method, program, and information processing device

Abstract

To provide an information processing method and the like that assist a user in interpreting the behavior of a generated machine learning model.SOLUTION: In this information processing method, a computer performs a process for: associating and recording explanatory data vectors xn input to an existing machine learning model 21 and objective data vectors yn output from the machine learning model 21, as a plurality of pairs; calculating an interpretation matrix A+, which is the vector product of an explanatory matrix X obtained by arranging a plurality of sets of explanatory data vectors xn, and a generalized inverse matrix of an objective matrix Y obtained by arranging objective data vectors yn in an order corresponding to the order of the explanatory data vectors X; and outputting charts 41, 42, 43 for the interpretation matrix A+.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing method, a program, and an information processing device. [Background technology]

[0002] A system has been proposed that uses a machine learning model to classify images and display the areas that contributed to the classification using a heat map (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2023-83555 Summary of the Invention [Problem to be solved by the invention]

[0004] Machine learning models generated by machine learning are black boxes, making it difficult for users to interpret their behavior.

[0005] In one aspect, an object of the present invention is to provide an information processing method, etc. that assists a user in interpreting the behavior of a generated machine learning model. [Means for solving the problem]

[0006] The information processing method involves a computer executing a process in which explanatory data vectors input to an existing machine learning model are associated with target data vectors output from the machine learning model and recorded in multiple sets, an interpretation matrix which is the vector product of an explanatory matrix in which multiple sets of the explanatory data vectors are arranged and a generalized inverse matrix of the target matrix in which the target data vectors are arranged in an order corresponding to the explanatory data vectors, and a chart related to the interpretation matrix. Effect of the Invention

[0007] In one aspect, an information processing method can be provided that assists a user in interpreting the behavior of a generated machine learning model. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of AIME. [Diagram 2] FIG. 11 is an explanatory diagram for explaining a method of calculating an interpretation matrix A†. [Diagram 3] FIG. 2 is an explanatory diagram illustrating a configuration of an information processing device. [Figure 4] 11 is a flowchart illustrating the process flow of a program for calculating an interpretation matrix A†. [Diagram 5] FIG. 1 is an explanatory diagram for explaining an outline of a method for creating a similarity distribution map. [Figure 6] 11 is a flowchart illustrating the process flow of a program for generating a similarity distribution map. [Figure 7] FIG. 11 is an explanatory diagram for explaining the record layout of a description-objective DB. [Figure 8] FIG. 13 is an explanatory diagram illustrating a global feature importance graph. [Figure 9] 1 is an example of a local feature importance graph. [Figure 10] 1 is an example of a similarity distribution map. [Figure 11] 1 is an example of a similarity distribution map. [Figure 12] 1 is an example of a similarity distribution map. [Figure 13] FIG. 11 is an explanatory diagram illustrating local feature importance regarding an existing model. [Figure 14] 1 is an example of a similarity distribution diagram regarding existing models. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Embodiment 1] Using various machine learning algorithms, machine learning models that accept explanatory data as input and output target data are generated. The generated machine learning models are usually black boxes, and it is difficult for humans to interpret the decision-making process from input of explanatory data to output of target data.

[0010] However, when using machine learning models for decision-making in the real world, it is important that humans can interpret the decision-making process of the machine learning model. For example, if the output target data seems to deviate significantly from human common sense, if humans can properly interpret the decision-making process of the machine learning model, they can also properly determine how to handle the target data and the machine learning model.

[0011] The technology that explains the reason for the output of target data from a machine learning model is called XAI (Explainable AI). For example, XAI technologies such as LIME (Local Interpretable Model Agnostic Explanations), which is suitable for explaining the local behavior of machine learning models, and SHAP (Shapley Additive Explanation), which is suitable for evaluating the importance of explanatory variables, are well known.

[0012] In this embodiment, an information processing method for supporting a user to interpret the behavior of a machine learning model from various viewpoints will be described. The information processing method described below is referred to as AIME (Approximate Inverse Model Explanations).

[0013] Fig. 1 is an explanatory diagram for explaining an overview of AIME. Using Fig. 1, a procedure for interpreting the behavior of an existing model 21, which is a generated machine learning model, using AIME will be explained. The existing model 21 is a machine learning model that accepts an input of an explanatory data vector xn and outputs a target data vector yn.

[0014] The existing model 21 is a machine learning model generated using any supervised machine learning algorithm, such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a Random Forest. The existing model 21 may be a machine learning model generated using any unsupervised machine learning algorithm, such as reinforcement learning or a Generative Adversarial Network (GAN).

[0015] The procedure of inputting the explanation data vector xn into the existing model 21 to obtain the target data vector yn is repeated multiple times. If the existing model 21 is generated by supervised machine learning, the explanation data vector xn may be a part or all of the training data used in the machine learning. The explanation data vector xn may be generated randomly or based on a predetermined rule.

[0016] A two-dimensional matrix, that is, an explanation matrix X, is generated by arranging a plurality of explanation data vectors xn in the row direction. Similarly, a two-dimensional matrix, that is, a target matrix Y, is generated by arranging a plurality of target data vectors yn in the row direction. Here, the arrangement order of the explanation data vectors xn is the same as the arrangement order of the corresponding target data vectors yn.

[0017] In order to execute the subsequent processes, the target data vector yn must be linearly independent, i.e., the vector product of the target matrix Y and the transposed matrix of the target matrix Y must be a regular matrix.

[0018] Based on the explanation matrix X and the objective matrix Y, an interpretation matrix A† is calculated, as shown in equation (1), which is a matrix whose vector product with the objective matrix Y is equal to the explanation matrix X. The method of calculating the interpretation matrix A† will be described later in detail. X = A†Y (1)

[0019] By charting the interpretation matrix A†, charts such as a global feature importance graph 41 showing the overall behavior of the existing model 21, a local feature importance graph 42 showing the interpretation of each target data vector yn, and a similarity distribution diagram 43 showing the distribution of similarities between explanatory data vectors xn are created. By using these charts, the user can interpret the behavior of the existing model 21. Details of the global feature importance graph 41, the local feature importance graph 42, and the similarity distribution diagram 43 will be described later.

[0020] 2 is an explanatory diagram for explaining a method for calculating the interpretation matrix A†. In the following explanation, N is a natural number indicating the number of times the process of inputting the explanation data vector xn to the existing model 21 and obtaining the target data vector yn is repeated. n is a natural number indicating the number of times the vector is input to the existing model 21 or output from the existing model 21.

[0021] The explanation data vector xn has L elements from Ex1n to ExLn. The target data vector yn has M elements from Ob1n to ObMn, where L and M are natural numbers. In FIG. 2, the explanation data vector xn and the target data vector yn, where n=2, are surrounded by dashed lines.

[0022] As described above, the explanation matrix X, which is a two-dimensional matrix, is created by arranging the N explanation data vectors xn obtained by N processing steps in the row direction. The explanation matrix X is a two-dimensional matrix with L rows and N columns. Similarly, the objective matrix Y, which is a two-dimensional matrix, is generated by arranging the N objective data vectors yn in the row direction. The objective matrix Y† is a two-dimensional matrix with M rows and N columns.

[0023] For the objective matrix Y, the Moore-Penrose generalized inverse matrix Y† is calculated. In the following explanation, Y† may be referred to as the objective inverse matrix Y†. The objective inverse matrix Y† is calculated using equation (2). The objective inverse matrix Y† is a matrix with N rows and M columns. Y†=Y T (YY T ) -1... (2)

[0024] The interpretation matrix A† is the vector product of the explanation matrix X and the objective inverse matrix Y†. The formula for calculating the interpretation matrix A† is shown in formula (3). A† = XY† ‥‥‥ (3)

[0025] The interpretation matrix A† is a two-dimensional matrix with L rows and M columns, i.e., the number of rows is the same as the number of elements in the explanatory data vector xn, and the number of columns is the same as the number of elements in the target data vector yn. The element in the a-th row and b-th column of the interpretation matrix A† indicates the influence of the a-th element of the explanatory data vector xn on the b-th element of the target data vector yn.

[0026] As a result, even if there is no information such as the algorithm and training data used to generate the existing model 21, the interpretation matrix A† can be generated as long as there is an environment in which the existing model 21 can be used. In other words, even if the existing model 21 is in a black box state generated by a third party, the interpretation matrix A† can be generated.

[0027] For reference, the outline of the transformation to derive equation (3) from equations (1) and (2) is shown below. First, both sides of equation (1) are multiplied by the transpose of the objective matrix Y from the right to obtain equation (4). XY T = A† YY T ... (4)

[0028] As described above, the vector product of the objective matrix Y and the transposed matrix of the objective matrix Y is a regular matrix, so the inverse matrix can be calculated. This inverse matrix is ​​multiplied from the right to both sides of equation (4) to obtain equation (5). XY T (YY T ) -1 = A†(YY T )(YY T ) -1 = A† ... (5)

[0029] After swapping the left and right sides of equation (5), we can substitute equation (2) into the right side to obtain equation (6). Equation (2) is derived from both ends of equation (6). A† = XY T (YY T ) -1 = XY† ... (6)

[0030] 3 is an explanatory diagram illustrating the configuration of the information processing device 10. The information processing device 10 includes a control unit 11, a main storage device 12, an auxiliary storage device 13, a communication unit 14, a display unit 15, an input unit 16, a reading unit 19, and a bus.

[0031] The control unit 11 is an arithmetic and control device that executes the program of this embodiment. The control unit 11 uses one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), multi-core CPUs, etc. The control unit 11 is connected to each hardware unit constituting the information processing device 10 via a bus.

[0032] The main memory device 12 is a storage device such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc. The main memory device 12 temporarily stores information required during processing performed by the control unit 11 and programs being executed by the control unit 11.

[0033] The auxiliary storage device 13 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 13 stores the existing model 21, a program to be executed by the control unit 11, and various data required for executing the program. The existing model 21 may be stored in an external storage device connected via a network.

[0034] The communication unit 14 is an interface for communicating between the information processing device 10 and a network. The display unit 15 is, for example, a liquid crystal display device or an organic EL (Electro Luminescence) display device. The input unit 16 is, for example, an input device such as a keyboard, a mouse, a trackball, or a microphone.

[0035] The portable recording medium 96 is, for example, a Universal Serial Bus (USB) memory, a Compact Disc Read only memory (CD-ROM), a magneto-optical disk medium, other optical disk medium, an SD memory card, etc. The portable recording medium 96 stores a program 97 that realizes AIME.

[0036] The reading unit 19 is an interface to which a portable recording medium 96, such as a USB connector, a CD-ROM drive, or an SD memory reader, can be connected. The semiconductor memory 98 stores a program 97 and is a memory that can be attached inside the information processing device 10.

[0037] The information processing device 10 is a general-purpose personal computer, a tablet, a mainframe computer, a virtual machine running on a mainframe computer, or a quantum computer. The information processing device 10 may be configured with hardware such as a plurality of personal computers or a mainframe computer that perform distributed processing. The information processing device 10 may be configured with a cloud computing system. The information processing device 10 may be configured with hardware such as a plurality of personal computers or a mainframe computer that operate in cooperation with each other.

[0038] The program 97 is recorded in a portable recording medium 96. The control unit 11 reads the program 97 via the reading unit 19 and stores it in the auxiliary storage device 13. The control unit 11 may also read out the program 97 stored in a semiconductor memory 98. Furthermore, the control unit 11 may download the program 97 from another server computer (not shown) connected via the communication unit 14 and a network (not shown) and store it in the auxiliary storage device 13.

[0039] The program 97 is installed as a control program for the information processing device 10, and is executed by being loaded into the main storage device 12. The program 97 of the present embodiment is an example of a program product.

[0040] 4 is a flowchart explaining the processing flow of a program for calculating the interpretation matrix A†. The control unit 11 determines the explanatory data vector xn to be inputted to the existing model 21 as the nth (step S501). When the training data used for the machine learning of the existing model 21 is available, the existing model 21 may extract the explanatory data vector xn from the training data.

[0041] The control unit 11 may generate the explanatory data vector xn randomly or based on a predetermined rule. When the control unit 11 generates the explanatory data vector xn, it is desirable to generate the explanatory data vector xn within the range in which the use of the existing model 21 is assumed. For example, when the explanatory data vector xn including an unexpected element such as "the age of a human being is 200 years" is generated, the behavior of the existing model 21 cannot be correctly interpreted.

[0042] Similarly, it is desirable that the distribution of the multiple explanatory data vectors xn coincide with the range in which the existing model 21 is expected to be used. For example, in the case of an existing model 21 that predicts the behavior of a general adult, it is desirable that the element of "human age" be close to the distribution of data such as demographics.

[0043] The control unit 11 inputs the explanatory data vector xn acquired in step S501 to the existing model 21 to acquire the target data vector yn (step S502). The control unit 11 associates the explanatory data vector xn with the target data vector yn and records them in the main storage device 12 or the auxiliary storage device 13 (step S503).

[0044] The control unit 11 determines whether or not to end the generation of pairs of the explanatory data vector xn and the target data vector yn (step S504). For example, the control unit 11 determines to end the process in step S504 when the processing of the explanatory data vector xn recorded in the training data is ended. The control unit 11 may determine to end the process when a predetermined number of pairs are generated.

[0045] If it is determined not to end (NO in step S504), the control unit 11 returns to step S501. If it is determined to end (YES in step S504), the control unit 11 generates an explanation matrix X based on the data recorded in step S503 (step S505). The control unit 11 generates a target matrix Y based on the data recorded in step S503 (step S506). The control unit 11 calculates a target inverse matrix Y†, which is a Moore-Penrose generalized inverse matrix, based on the target matrix Y (step S507). The control unit 11 calculates an interpretation matrix A†, which is a vector product of the explanation matrix X and the target inverse matrix Y† (step S508). The control unit 11 ends the process.

[0046] The following provides an overview of how to interpret the behavior of the existing model 21 by plotting the generated interpretation matrix A†.

[0047] [Global feature importance] The global feature importance means the degree to which each element constituting the explanatory data vector xn contributes to a specific prediction result output from the existing model 21. The user can understand the overall behavior of the existing model 21 based on the visualized global feature importance.

[0048] As mentioned above, the element in row a and column b of the interpretation matrix A† indicates the influence of the a-th element of the explanatory data vector xn on the b-th element of the target data vector yn. For example, the degree of influence that each element of the explanatory data vector xn has on a specific item of the target data vector yn can be visualized by a bar graph or line graph using the item names corresponding to each element of the explanatory data vector xn on the first axis and the value of an arbitrary column of the interpretation matrix A† on the second axis. Specific examples of charts that visualize global feature importance will be described later.

[0049] [Representative Estimated Instance] The representative estimation instance x* means an ideal or typical explanatory data vector xn in which the target data vector yn output from the existing model 21 is in a specific state. The representative estimation instance x* is a vector having the same number of elements as the explanatory data vector xn.

[0050] The representative estimated instance x* is calculated by the vector product of the interpretation matrix A† and the target data vector yn. The equation for calculating the representative estimated instance x* is shown in equation (7). x* = A†yn... (7)

[0051] In the following explanation, an example will be described in which the target data vector yn is a unit target vector yuk whose k-th element is 1 and the other elements are 0. k is a natural number equal to or less than the number of elements of the target data vector yn. The formula for calculating the representative estimated instance x* when the target data vector yn is the unit target vector yuk is shown in formula (8), and the definition of the i-th element yuk(i) of the unit target vector yuk is shown in formula (9).

[0052]

number

[0053] Based on the representative estimated instance x*, the user can grasp the importance of each item of the explanatory data vector xn when a specific target data vector yn is output. Furthermore, based on the representative estimated instance x*, the user can obtain clues for interpreting the behavior of the existing model 21. Specific examples of the representative estimated instance x* will be described later.

[0054] [Local feature importance] The local feature importance vector L means the degree to which each element constituting a specific explanatory data vector xn contributes to the target data vector yn output from the existing model 21. The local feature importance vector L is a vector having the same number of elements as the explanatory data vector xn.

[0055] In the following description, a specific explanation data vector xn is described as a target explanation vector xo. The target explanation vector xo may be data included in the training data or data created separately from the training data. An example will be described in which the target data vector yn is the unit target vector yuk. The local feature importance vector L is calculated by equation (10).

[0056]

number

[0057] For example, the local feature importance vector L can be visualized by a bar graph or line graph in which the first axis represents the item names corresponding to each element constituting the local feature importance vector L, i.e., the item names corresponding to each element constituting the explanatory data vector xn, and the second axis represents the values ​​of each element constituting the local feature importance vector L. Specific examples of charts for visualizing the local feature importance vector L will be described later.

[0058] Based on the local feature importance vector L, the user can interpret why the target data vector yn was output from the existing model 21 that accepted the input of the explanatory data vector xn. The user can also understand the elements of the explanatory data vector xn that affect the target data vector yn based on the local feature importance vector L. Specific examples of the local feature importance vector L will be described later.

[0059] [Similarity distribution chart] 5 is an explanatory diagram outlining a method for creating the similarity distribution diagram 43. First, two representative estimated instances x*, a first typical example vector and a second typical example vector, are calculated based on equation (8). Different values ​​of k are used when calculating the first typical example vector and when calculating the second typical example vector.

[0060] An explanatory data vector xn that satisfies a specific condition is selected from the explanatory matrix X. For example, when training data used in machine learning is used for the explanatory data vector xn, it is desirable to select an explanatory data vector xn that corresponds to the unit objective vector yuk when the correct answer data calculates the first typical example vector. Of the objective data vectors yn output from the existing model 21, an explanatory data vector xn whose k-th element is greater than a predetermined threshold may be selected.

[0061] A first similarity is calculated for the selected explanatory data vector xn, which is the similarity between the explanatory data vector xn and the first typical example vector. The similarity may be, for example, the inverse of an inter-vector distance between the explanatory data vector xn and the first typical example vector, which is defined by an arbitrary factor such as the Euclidean distance, the Manhattan distance, or the Hamming distance. The similarity may be, for example, the cosine similarity.

[0062] Similarly, a second similarity between the selected explanatory data vector xn and the second typical example vector is calculated. Each explanatory data vector xn is plotted on a two-dimensional graph using the first similarity value on the x-axis and the second similarity value on the y-axis.

[0063] Kernel density estimation is performed on the plotted data by applying, for example, an RBF kernel, to generate a smooth similarity distribution diagram 43. The similarity distribution diagram 43 allows the user to intuitively recognize the behavior of the existing model 21 with respect to the selected explanatory data vector xn. Note that kernel density estimation may not be performed, and individual plots may be displayed directly.

[0064] Kernel density estimation may be performed on plotting each explanatory data vector xn, and the plots are then overlaid to produce a smooth similarity plot 43.

[0065] A similarity distribution diagram 43 created using explanatory data vectors xn selected under a plurality of conditions may be superimposed and displayed. A three-dimensional similarity distribution diagram 43 may be generated using three representative estimated instances x*. A specific example of the similarity distribution diagram 43 will be described later.

[0066] 6 is a flowchart for explaining the process flow of the program for generating the similarity distribution diagram 43. The control unit 11 receives a designation for the first explanatory data vector xn (step S521). For example, the control unit 11 receives a designation of k and uses the unit objective vector yuk for the first explanatory data vector xn. This unit objective vector yuk is an example of the first unit vector. The control unit 11 calculates the representative estimated instance x* based on the formula (7). The control unit 11 uses the calculated representative estimated instance x* as the first typical example vector (step S522).

[0067] The control unit 11 receives a designation for the second explanatory data vector xn (step S523). For example, the control unit 11 receives a designation of k and uses the unit objective vector yuk for the second explanatory data vector xn. This unit objective vector yuk is an example of the second unit vector. The control unit 11 calculates the representative estimated instance x* based on the formula (7). The control unit 11 uses the calculated representative estimated instance x* for the second typical example vector (step S524).

[0068] The control unit 11 selects one explanatory data vector xn that meets a predetermined condition from the explanatory matrix X (step S525). The control unit 11 calculates the similarity between the explanatory data vector xn selected in step S525 and the first typical example vector calculated in step S522 (step S526). The control unit 11 calculates the similarity between the explanatory data vector xn selected in step S525 and the second typical example vector calculated in step S524 (step S527). The control unit 11 associates the similarity calculated in step S526 with the similarity calculated in step S527 and records them in the main storage device 12 or the auxiliary storage device 13 (step S528).

[0069] The control unit 11 determines whether to end the calculation of the similarity (step S529). Specifically, for example, the control unit 11 determines to end the calculation of the similarity when the processing of the explanation data vector xn that satisfies a predetermined condition in the explanation matrix X is ended. The control unit 11 may determine to end the calculation of the similarity when the calculation of the similarity is repeated a predetermined number of times.

[0070] If it is determined not to end (NO in step S529), the control unit 11 returns to step S525. If it is determined to end (YES in step S529), the control unit 11 creates a scatter plot for each data recorded in step S528, with the similarity to the first typical example vector on the first axis and the similarity to the second typical example vector on the second axis (step S530).

[0071] The control unit 11 performs kernel density estimation on the scatter diagram created in step S530 to smooth the distribution of each plot in the scatter diagram (step S531). Through the above, the similarity distribution diagram 43 is completed. The control unit 11 displays the completed similarity distribution diagram 43 (step S532). After that, the control unit 11 ends the process.

[0072] [Example 1: Titanic Dataset] We explain an example of interpreting an existing model21 that was generated using the Titanic dataset as training data, which summarizes the survival status of passengers on the Titanic, which sank in the North Atlantic Ocean after being struck by an iceberg in 1912.

[0073] The existing model 21 is a learning model trained to output the survival probability and non-survival probability of a passenger when a 12-element explanatory data vector xn related to the passenger is input. Since the method of generating a machine learning model using supervised machine learning is publicly known, the generation method will not be described here. The elements of the explanatory data vector xn are shown in Table 1.

[0074] [Table 1]

[0075] In Table 1, the eight items from first-class cabin to departure from Port S are expressed as binary values, with 1 for YES and 0 for NO.

[0076] 7 is an explanatory diagram for explaining the record layout of the explanation-purpose DB 31. The explanation-purpose DB 31 has a No. field, an explanation data vector field, and a purpose data vector field. The explanation data vector field has subfields corresponding to each element of the explanation data vector xn shown in Table 1, such as an Age field, a SibSp field, and a Parch field. The purpose data vector field has a survival probability field and a non-survival probability field.

[0077] The No. field records consecutive numbers from 1 to N. Each subfield of the explanatory data field records an element of the explanatory data vector xn. The survival probability field records the probability that a passenger having attributes of the explanatory data vector xn will survive. The non-survival probability field records the probability that a passenger having attributes of the explanatory data vector xn will not survive. For each record, the sum of the values ​​recorded in the survival probability field and the non-survival probability field is 1.

[0078] In step S503 of the program described using Fig. 4, the control unit 11 adds one record to the explanation-objective DB 31. The part surrounded by the dashed line in Fig. 7 is the transposed matrix of the explanation matrix X shown in Fig. 2. The part surrounded by the solid line in Fig. 7 is the transposed matrix of the objective matrix Y shown in Fig. 2.

[0079] Fig. 8 is an explanatory diagram for explaining the global feature importance graph 41. The upper part of Fig. 8 shows a schematic diagram of the interpretation matrix A† calculated in step S508 of the program explained using Fig. 4. The interpretation matrix A† is a matrix with 12 rows and 2 columns. The vertical numbers of the elements of the interpretation matrix A†, i.e., the third character, are expressed in hexadecimal. That is, A indicates 10, B indicates 11, and C indicates 12.

[0080] As mentioned above, the element in row a and column b of the interpretation matrix A† indicates the influence of the a-th element of the explanatory data vector xn on the b-th element of the objective data vector yn. Therefore, AD11 indicates the influence of the passenger's age on the passenger's survival, and AD12 indicates the influence of the passenger's age on the passenger's non-survival.

[0081] The lower part of Fig. 8 shows a global feature importance graph 41. The vertical axis shows the item names of the explanatory data vector xn. The horizontal axis shows the values ​​of each element of the interpretation matrix A†. The left-leaning hatching indicates the values ​​of the left column of the interpretation matrix A†, i.e., the influence of each element on the passenger's survival. The right-leaning hatching indicates the values ​​of the right column of the interpretation matrix A†, i.e., the influence of each element on the passenger's non-survival.

[0082] From FIG. 8, it can be seen that the number of passengers A has almost no effect on whether the passengers survive or not. Therefore, it can be seen that data on the number of passengers A is not necessary to predict whether the passengers will survive or not. For example, by deleting the number of passengers A from the explanatory variables, the user can reduce the amount of calculations of the existing model 21 without affecting the prediction accuracy.

[0083] Similarly, Figure 8 shows that the factor that has a large influence on whether a passenger survives is whether the passenger is female, and the factor that has a large influence on whether a passenger does not survive is whether the passenger is male. It can be seen that the gender of a passenger has a larger influence on whether or not they survive than other factors.

[0084] Table 2 shows representative estimation instances for survivors and non-survivors calculated based on equation (8).

[0085] [Table 2]

[0086] Table 2 shows that the typical survivor is a woman with a relatively high passenger fare, while the typical non-survivor is a man with a relatively low passenger fare.

[0087] Next, an example of local feature importance is explained. In calculating the local feature importance, we created object explanation vectors xo for two people, Rose, a survivor, and Jack, a non-survivor, named after the characters in the 1997 movie Titanic. The created object explanation vectors xo are shown in Table 3.

[0088] [Table 3]

[0089] FIG. 9 is an example of a local feature importance graph 42. The local feature importance vector L for Rose's survival, calculated based on equation (10), is shown in the local feature importance graph 42 of FIG. 9A. The transposed matrix of (1,0) is used for the unit objective vector yuk. The vertical axis indicates the item names of the object explanation vector xo. The horizontal axis indicates the values ​​of each element of the local feature importance vector L. According to FIG. 9A, the first reason for Rose's survival is that she is a woman, and the second reason is that she is a passenger in a first-class cabin.

[0090] The local feature importance vector L for Jack not surviving, calculated based on equation (10), is shown in the local feature importance graph 42 in FIG. 9B. For the unit objective vector yuk, a transposed matrix of (0,1) is used. The vertical axis indicates the item names of the object explanation vector xo. The horizontal axis indicates the values ​​of each element of the local feature importance vector L. According to FIG. 9B, the first reason for Jack not surviving is that he is a male, and the second reason is that he is a passenger in a third-class cabin.

[0091] As shown in Figures 9A and 9B, the interpretation matrix A can be used to create a local feature importance graph 42 for any target explanation vector xo. The user can also use the local feature importance graph 42 to make interpretations for explanation data vectors xn that are not included in the explanation matrix X.

[0092] 10 to 12 are examples of similarity distribution diagrams 43. FIG. 10 shows a similarity distribution diagram 43 for survivors. An overview of the process for creating FIG. 10 will be described. The first typical example vector calculated by the control unit 11 in step S522 of the program described using FIG. 6 is equal to the representative inferred instance of a non-survivor shown in Table 2. The second typical example vector calculated by the control unit 11 in step S524 is equal to the representative inferred instance of a survivor shown in Table 2. In step S525, the control unit 11 selects an explanatory data vector xn for a survivor from the explanatory matrix X.

[0093] The horizontal axis of FIG. 10 indicates a non-survival score corresponding to the similarity to the representative estimated instance of a non-survivor. A larger non-survival score means a greater similarity to the representative estimated instance of a non-survivor. The vertical axis of FIG. 10 indicates a survival score corresponding to the similarity to the representative estimated instance of a survivor. A larger survival score means a greater similarity to the representative estimated instance of a survivor. The finer the hatching downward to the left, the denser the distribution of the explanatory data vector xn related to the survivor. FIG. 10 is an example of a first distribution diagram.

[0094] Fig. 11 shows a similarity distribution diagram 43 for non-survivors. An overview of the process for creating Fig. 11 will be described. The processes in steps S522 and S524 of the program described using Fig. 6 are the same as those in creating Fig. 10. In step S525, the control unit 11 selects an explanatory data vector xn for non-survivors from the explanatory matrix X.

[0095] The vertical and horizontal axes in Fig. 11 are the same as those in Fig. 10, and therefore will not be described. The finer the hatching from the lower right to the lower right, the denser the distribution of the explanatory data vector xn relating to non-survivors. Fig. 11 is an example of the second distribution diagram.

[0096] FIG. 12 shows a similarity distribution diagram 43 in which FIG. 10 and FIG. 11 are superimposed. That is, the distribution for survivors and the distribution for non-survivors are superimposed on one diagram. The open circles indicate the non-survival scores and survival scores for Rose described in Table 3. The closed circles indicate the non-survival scores and survival scores for Jack described in Table 3.

[0097] From Fig. 12, there is an area where the distribution of survivors and the distribution of non-survivors overlap. The user can understand that in this area, it is difficult to predict whether the passengers will survive or not. Fig. 12 is an example of a similarity distribution diagram 43 in which the first distribution diagram and the second distribution diagram are superimposed.

[0098] When values ​​of the intermediate layer of the existing model 21 can be obtained, an interpretation matrix A† may be generated for each of the portion from the input layer that receives the explanatory data vector xn to the intermediate layer and the portion from the intermediate layer to the output layer that outputs the target data vector yn. By generating the interpretation matrix A† in a layered manner, it is possible to improve the interpretability of the existing model 21.

[0099] [Example 2: MNIST (Mixed National Institute of Standards and Technology database) dataset] Here we explain an example of interpreting an existing model21 for classifying handwritten digits, which was generated using the "MNIST dataset" as training data. The MNIST dataset is a dataset that records handwritten digit images, which are composed of 784 8-bit grayscale pixels arranged in 28 rows and 28 columns, in association with the correct answer data, which is the digit.

[0100] The existing model 21 is a learning model trained to output the probability that each number from 0 to 9 is a digit when a grayscale image of 28 pixels in each direction is input. The explanatory data vector xn is a 784-element vector in which the intensities of 784 pixels are arranged in a predetermined order. The target data vector yn is a 10-element vector in which the probability that each number from 0 to 9 is an digit is arranged.

[0101] Fig. 13 is an explanatory diagram for explaining local feature importance regarding the existing model 21. As shown in the upper part of Fig. 13, the process of inputting the explanation data vector xn to the existing model 21 and obtaining the target data vector yn is repeated, and an interpretation matrix A† is calculated using the procedure described using Fig. 2. The interpretation matrix A† is a matrix with 784 rows and 10 columns.

[0102] The lower part of Fig. 13 shows three examples of local feature importance vectors L in table format. When the number image of the object explanation vector xo is input to the existing model 21, the local feature importance vectors L are shown for the cases where the numbers are determined to be 0, 3, and 8. Note that Fig. 13 shows each handwritten character and local feature importance vector L converted into black and white binary. When each number is determined to be a number, the pixels that are given importance are displayed in black.

[0103] The local feature importance vector L enables the user to recognize that if the object explanation vector xo is determined to be 0, then the top and bottom portions of the handwritten character contribute; if it is determined to be 3, then the right side of the handwritten character contributes; and if it is determined to be 8, then almost the entire handwritten character contributes.

[0104] FIG. 14 is an example of a similarity distribution diagram 43 for the existing model 21. The horizontal axis is the similarity score with respect to the representative inferred instance x* determined to be "3". The vertical axis is the similarity score with respect to the representative inferred instance x* determined to be "8". The white circle indicates the position where the target explanation vector xo shown in the lower right of FIG. 13 is plotted.

[0105] In the MNIST dataset, the distribution of the explanation data vector xn for which the correct data is "8" is shown with hatching slanting downwards to the left, and the distribution of the explanation data vector xn for which the correct data is "3" is shown with hatching slanting downwards to the right. The darker the hatching, the denser the distribution.

[0106] Since there is a large overlap between the "3" area and the "8" area, the user can recognize that 3 and 8 are easily mistakenly determined. Although not shown in the figure, for example, the "0" area and the "1" area do not overlap, so the user can recognize that "0" and "1" are not easily mistakenly determined.

[0107] The existing model 21 may be, for example, a learning model that receives input of a sentence and outputs a keyword. The existing model 21 may be, for example, a learning model that receives input of an image and outputs the name of the subject. For any other existing model 21, the interpretation matrix A† can be calculated and used to interpret the behavior.

[0108] [Variation 1] This modification relates to a method for quickly generating an interpretation matrix A†, and a description will be omitted for parts common to the first embodiment. In this modification, the existing model 21 is generated by supervised machine learning.

[0109] Instead of repeatedly inputting the explanatory data vector xn into the existing model 21 to obtain the target data vector yn, the control unit 11 extracts pairs of the explanatory data vector xn and the target data vector yn from the training data and generates the explanatory matrix X and the target matrix Y.

[0110] According to this modified example, the interpretation matrix A† can be calculated before completing the generation of the existing model 21. For example, a user can predict the behavior of the existing model 21 using the interpretation matrix A† before generating the existing model 21, and take measures such as increasing the training data if the existing model 21 does not have the desired characteristics.

[0111] [Variation 2] This modification relates to a method of regularizing the target matrix Y using Ridge regression, that is, L2 norm regularization. Explanation of parts common to the first embodiment will be omitted.

[0112] In this modification, the control unit 11 calculates the generalized inverse matrix Y† of Y by using equation (11) instead of equation (2) in step S507 of the flowchart described with reference to FIG. Y†=Y T (YY T -λE) -1 ... (11) λ is a positive real number. E is an identity matrix with M rows and M columns.

[0113] According to this modification, an appropriate interpretation matrix A† can be calculated even for a complex existing model 21 having a large number of parameters or interactions between parameters.

[0114] [Variation 3] This modification relates to a method of regularizing the target matrix Y using Lasso regression, that is, L1 norm regularization. Explanation of parts common to the first embodiment will be omitted.

[0115] In this modified example, in step S507 of the flowchart described using FIG. 4, the control unit 11 calculates the generalized inverse matrix Y† of Y, and then regularizes the generalized inverse matrix Y† of Y using a known iterative algorithm.

[0116] This modification also makes it possible to calculate an appropriate interpretation matrix A† for a complex existing model 21 having a large number of parameters or interactions between parameters. Note that the regularization method is not limited to the Ridge regression of modification 2 and the Lasso regression of this modification. For regularization, a known method such as ElasticNet can also be used.

[0117] A program is an example of a program product. A computer program can be deployed to be executed on a single computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

[0118] The technical features (constitutive elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims.

[0119] Independent claims and dependent claims described in the claims may be combined with each other in any combination, regardless of the reference format. Furthermore, the claims may be written in a format in which a claim references two or more other claims (multiple claim format), but this is not limited to this format. The claims may also be written in a format in which a multiple claim references at least one other claim (multiple multiple claim format). [Explanation of symbols]

[0120] 10. Information processing device 11 Control section 12 Main storage 13 Auxiliary storage device 14 Communications Department 15 Display section 16 Input section 19 Reading section 21 Existing models (existing machine learning models) 31 Description-Purpose DB 41 Global feature importance graph (chart) 42 Local feature importance graph (chart) 43 Similarity distribution chart (chart) 96 Portable recording media 97 Program 98 Semiconductor Memory

Claims

1. A plurality of sets of explanatory data vectors input to an existing machine learning model and target data vectors output from the machine learning model are recorded in association with each other; Calculate an interpretation matrix which is a vector product of an explanation matrix in which a plurality of sets of the explanation data vectors are arranged and a generalized inverse matrix of a target matrix in which the target data vectors are arranged in an order corresponding to the explanation data vectors; Output a chart related to the interpretation matrix An information processing method in which processing is performed by a computer.

2. The generalized inverse of the objective matrix is ​​the Moore-Penrose generalized inverse of the objective matrix The information processing method according to claim 1 .

3. The chart is a graph in which the first axis represents the item names corresponding to the individual elements of the explanatory data vector, and the second axis represents the element values ​​for each column constituting the interpretation matrix. The information processing method according to claim 1 .

4. Take the description data vector, Generate a unit objective vector having one element of the objective data vector that is an object to represent a local feature importance and other elements of the objective data vector being 1 and 0, The chart is a graph in which the first axis represents the item names corresponding to the individual elements of the explanatory data vector, and the second axis represents the values ​​of the elements constituting the local feature importance vector calculated by equation (1). The information processing method according to claim 1 . [0010]

5. Calculating a first typical example vector which is a vector product of the interpretation matrix and a first unit vector having one element of the target data vector being 1 and the other elements being 0; A second typical example vector is calculated as a vector product of the interpretation matrix and a second unit vector whose other elements of the target data vector are 1 and whose other elements are 0; The chart is a distribution diagram obtained by plotting the similarity between each of the explanatory data vectors selected from the explanatory matrix and the first typical example vector on the horizontal axis and the similarity between each of the explanatory data vectors and the second typical example vector on the vertical axis, and then performing kernel density estimation. The information processing method according to claim 1 .

6. The explanation data vector is identical to the explanation data in the training data used for machine learning of the machine learning model, The diagram is: FIG. 1 is a diagram showing a first distribution map created using explanatory data corresponding to one element when the first typical example vector is calculated from correct answer data in the training data, and a second distribution map created using explanatory data corresponding to one element when the second typical example vector is calculated from correct answer data in the training data, superimposed on each other. The information processing method according to claim 5.

7. The explanation data vector is the same as the explanation data in the training data used for machine learning of the machine learning model.

6. The information processing method according to claim 1.

8. A plurality of sets of explanatory data vectors input to an existing machine learning model and target data vectors output from the machine learning model are recorded in association with each other; Calculate an interpretation matrix which is a vector product of an explanation matrix in which a plurality of sets of the explanation data vectors are arranged and a generalized inverse matrix of a target matrix in which the target data vectors are arranged in an order corresponding to the explanation data vectors; Output a chart related to the interpretation matrix A program that causes a computer to carry out processing.

9. An information processing device including a control unit, The control unit is A plurality of sets of explanatory data vectors input to an existing machine learning model and target data vectors output from the machine learning model are recorded in association with each other; Calculate an interpretation matrix which is a vector product of an explanation matrix in which a plurality of sets of the explanation data vectors are arranged and a generalized inverse matrix of a target matrix in which the target data vectors are arranged in an order corresponding to the explanation data vectors; Output a chart related to the interpretation matrix Information processing device.

Images