Information processing method, program, and information processing device

Abstract

The purpose of the present invention is to provide an information processing method and the like that assist a user in interpreting the behavior of a generated machine learning model.　In this information processing method, a computer performs a process for: associating and recording explanatory data vectors xn input to an existing machine learning model (21) and objective data vectors yn output from the machine learning model (21), as a plurality of pairs; calculating an interpretation matrix A†, which is the vector product of an explanatory matrix X obtained by arranging a plurality of sets of explanatory data vectors xn, and a generalized inverse matrix of an objective matrix Y obtained by arranging objective data vectors yn in an order corresponding to the order of the explanatory data vectors X; and outputting charts (41, 42, 43) for the interpretation matrix A†.

Description

Information processing method, program, and information processing device

[0001] The present invention relates to an information processing method, a program, and an information processing device.

[0002] A system has been proposed that uses a machine learning model to classify images and displays the areas that contributed to the classification using a heat map (Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2023-83555

[0004] Machine learning models generated by machine learning are black boxes, making it difficult for users to interpret their behavior.

[0005] In one aspect, the present invention aims to provide an information processing method or the like that assists a user in interpreting the behavior of a generated machine learning model.

[0006] The information processing method involves a computer performing a process in which multiple sets of explanatory data vectors input into an existing machine learning model and target data vectors output from the machine learning model are associated and recorded, an interpretation matrix is ​​calculated which is the vector product of an explanatory matrix in which multiple sets of explanatory data vectors are arranged, and a generalized inverse matrix of the target matrix in which the target data vectors are arranged in an order corresponding to the explanatory data vectors, and a chart related to the interpretation matrix.

[0007] In one aspect, an information processing method or the like can be provided that assists a user in interpreting the behavior of a generated machine learning model.

[0008] 1 is an explanatory diagram illustrating an overview of AIME. An explanatory diagram illustrating a method for calculating an interpretation matrix A†. An explanatory diagram illustrating the configuration of an information processing device. A flowchart illustrating the processing flow of a program for calculating an interpretation matrix A†. An explanatory diagram illustrating an overview of a method for creating a similarity distribution diagram. A flowchart illustrating the processing flow of a program for generating a similarity distribution diagram. An explanatory diagram illustrating the record layout of an explanation-objective DB. An explanatory diagram illustrating a global feature importance graph. An example of a local feature importance graph. An example of a local feature importance graph. An example of a similarity distribution diagram. An example of a similarity distribution diagram. An example of a similarity distribution diagram. An explanatory diagram illustrating local feature importance for an existing model. An example of a similarity distribution diagram for an existing model.

[0009] [First Embodiment] A machine learning model that receives input of explanatory data and outputs target data is generated using various machine learning algorithms. The generated machine learning model is usually a black box, and it is difficult for humans to interpret the decision-making process from input of explanatory data to output of target data.

[0010] However, when using machine learning models for real-world decision-making, it is important that humans can interpret the decision-making process of the machine learning model. For example, if the output target data appears to deviate significantly from human common sense, if humans can properly interpret the decision-making process of the machine learning model, they can also properly determine how to handle the target data and the machine learning model.

[0011] A technology that explains the reason for the output of target data from a machine learning model is called XAI (Explainable AI). For example, known XAI technologies include LIME (Local Interpretable Model Agnostic Explanations), which is suitable for explaining the local behavior of machine learning models, and SHAP (Shapley Additive Explanation), which is suitable for evaluating the importance of explanatory variables.

[0012] In this embodiment, an information processing method that supports a user in interpreting the behavior of a machine learning model from various perspectives will be described. The information processing method described below will be referred to as AIME (Approximate Inverse Model Explanations).

[0013] Fig. 1 is an explanatory diagram outlining AIME. Using Fig. 1, we will explain the procedure for interpreting the behavior of an existing model 21, which is a generated machine learning model, using AIME. The existing model 21 is a machine learning model that accepts an explanation data vector xn as input and outputs a target data vector yn.

[0014] The existing model 21 is a machine learning model generated using any supervised machine learning algorithm, such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a random forest. The existing model 21 may also be a machine learning model generated using any unsupervised machine learning algorithm, such as reinforcement learning or a generative adversarial network (GAN).

[0015] The procedure of inputting the explanation data vector xn into the existing model 21 and obtaining the target data vector yn is repeated multiple times. If the existing model 21 is generated by supervised machine learning, some or all of the training data used in the machine learning may be used as the explanation data vector xn. The explanation data vector xn may be generated randomly or based on a predetermined rule.

[0016] A two-dimensional matrix, explanation matrix X, is generated by arranging multiple explanation data vectors xn in the row direction. Similarly, a two-dimensional matrix, objective matrix Y, is generated by arranging multiple objective data vectors yn in the row direction. Here, the arrangement order of the explanation data vectors xn and the arrangement order of the corresponding objective data vectors yn are the same.

[0017] In order to execute the subsequent processes, the target data vectors yn must be linearly independent, i.e., the vector product of the target matrix Y and its transpose must be a regular matrix.

[0018] Based on the explanation matrix X and the objective matrix Y, an interpretation matrix A† is calculated, which is a matrix whose vector product with the objective matrix Y is equal to the explanation matrix X, as shown in equation (1). The method for calculating the interpretation matrix A† will be described in detail later. X = A† Y (1)

[0019] By graphically representing the interpretation matrix A†, various charts are created, including a global feature importance graph 41 showing the overall behavior of the existing model 21, a local feature importance graph 42 showing the interpretation of each target data vector yn, and a similarity distribution diagram 43 showing the distribution of similarities between explanatory data vectors xn. By using these charts, the user can interpret the behavior of the existing model 21. Details of the global feature importance graph 41, the local feature importance graph 42, and the similarity distribution diagram 43 will be described later.

[0020] 2 is an explanatory diagram illustrating a method for calculating the interpretation matrix A†. In the following description, N is a natural number indicating the number of times the process of inputting the explanation data vector xn into the existing model 21 and obtaining the target data vector yn has been repeated. n is a natural number indicating the number of times the vector has been input into or output from the existing model 21.

[0021] The explanation data vector xn has L elements from Ex1n to ExLn. The target data vector yn has M elements from Ob1n to ObMn, where L and M are natural numbers. In Figure 2, the explanation data vector xn and the target data vector yn, where n=2, are surrounded by dashed lines.

[0022] As described above, the explanation matrix X, which is a two-dimensional matrix, is created by arranging the N explanation data vectors xn obtained through N processing steps in the row direction. The explanation matrix X is a two-dimensional matrix with L rows and N columns. Similarly, the target matrix Y, which is a two-dimensional matrix, is generated by arranging the N target data vectors yn in the row direction. The target matrix Y† is a two-dimensional matrix with M rows and N columns.

[0023] For the target matrix Y, Y†, which is the Moore-Penrose generalized inverse matrix, is calculated. In the following explanation, Y† may be referred to as the target inverse matrix Y†. The target inverse matrix Y† is calculated using equation (2). The target inverse matrix Y† is a matrix with N rows and M columns. Y† = YT (Y YT) -1 ... (2)

[0024] The interpretation matrix A† is the vector product of the explanation matrix X and the inverse objective matrix Y†. The formula for calculating the interpretation matrix A† is shown in formula (3). A† = X Y† (3)

[0025] The interpretation matrix A† is a two-dimensional matrix with L rows and M columns, i.e., the number of rows is the same as the number of elements in the explanation data vector xn, and the number of columns is the same as the number of elements in the target data vector yn. The element in the a-th row and b-th column of the interpretation matrix A† indicates the influence of the a-th element of the explanation data vector xn on the b-th element of the target data vector yn.

[0026] As described above, even if there is no information such as the algorithm and training data that generated the existing model 21, the interpretation matrix A† can be generated as long as there is an environment in which the existing model 21 can be used. In other words, even if the existing model 21 is in a black box state generated by a third party, the interpretation matrix A† can be generated.

[0027] For reference, the outline of the transformation to derive equation (3) from equations (1) and (2) is shown below. First, both sides of equation (1) are multiplied by the transpose of the target matrix Y from the right to obtain equation (4). X YT = A† Y YT (4)

[0028] As mentioned above, the vector product of the objective matrix Y and its transposed matrix is ​​a regular matrix, so the inverse matrix can be calculated. This inverse matrix is ​​multiplied from the right of both sides of equation (4) to obtain equation (5). X YT (Y YT) -1 = A† (Y YT) (Y YT) -1 = A† (5)

[0029] After swapping the left and right sides of equation (5), substituting equation (2) into the right side, we obtain equation (6). Equation (3) is derived from both ends of equation (6). A† = X YT (Y YT)-1 = X Y† ... (6)

[0030] 3 is an explanatory diagram illustrating the configuration of the information processing device 10. The information processing device 10 includes a control unit 11, a main memory device 12, an auxiliary memory device 13, a communication unit 14, a display unit 15, an input unit 16, a reading unit 19, and a bus.

[0031] The control unit 11 is an arithmetic and control device that executes the program of this embodiment. The control unit 11 uses one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), multi-core CPUs, etc. The control unit 11 is connected to each hardware unit that constitutes the information processing device 10 via a bus.

[0032] The main memory device 12 is a storage device such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc. The main memory device 12 temporarily stores information required during processing performed by the control unit 11 and programs currently being executed by the control unit 11.

[0033] The auxiliary storage device 13 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 13 stores the existing model 21, the program to be executed by the control unit 11, and various data required for executing the program. The existing model 21 may be stored in an external storage device connected via a network.

[0034] The communication unit 14 is an interface for communication between the information processing device 10 and a network. The display unit 15 is, for example, a liquid crystal display device or an organic EL (Electro Luminescence) display device. The input unit 16 is, for example, an input device such as a keyboard, a mouse, a trackball, or a microphone.

[0035] The portable recording medium 96 is, for example, a USB (Universal Serial Bus) memory, a CD-ROM (Compact Disc Read Only Memory), a magneto-optical disk medium, other optical disk medium, an SD memory card, etc. The portable recording medium 96 stores a program 97 that realizes AIME.

[0036] The reading unit 19 is an interface that can connect to a portable recording medium 96, such as a USB connector, a CD-ROM drive, or an SD memory reader. The semiconductor memory 98 stores a program 97 and is a memory that can be attached inside the information processing device 10.

[0037] The information processing device 10 is a general-purpose personal computer, a tablet, a mainframe computer, a virtual machine running on a mainframe computer, or a quantum computer. The information processing device 10 may be configured with hardware such as multiple personal computers or mainframe computers that perform distributed processing. The information processing device 10 may be configured with a cloud computing system. The information processing device 10 may be configured with hardware such as multiple personal computers or mainframe computers that operate in cooperation with each other.

[0038] The program 97 is recorded on a portable recording medium 96. The control unit 11 reads the program 97 via the reading unit 19 and stores it in the auxiliary storage device 13. The control unit 11 may also read out the program 97 stored in a semiconductor memory 98. Furthermore, the control unit 11 may download the program 97 from another server computer (not shown) connected via the communication unit 14 and a network (not shown) and store it in the auxiliary storage device 13.

[0039] The program 97 is installed as a control program for the information processing device 10, and is loaded into and executed by the main storage device 12. The program 97 of this embodiment is an example of a program product.

[0040] 4 is a flowchart illustrating the processing flow of a program for calculating the interpretation matrix A†. The control unit 11 determines the explanatory data vector xn to be input to the existing model 21 as the nth vector (step S501). If training data used in the machine learning of the existing model 21 is available, the existing model 21 may extract the explanatory data vector xn from the training data.

[0041] The control unit 11 may generate the explanatory data vector xn randomly or based on a predetermined rule. When the control unit 11 generates the explanatory data vector xn, it is desirable to generate the explanatory data vector xn within the range in which the use of the existing model 21 is expected. For example, if the control unit 11 generates an explanatory data vector xn that includes an unexpected element, such as "the age of a human being is 200 years," it will be impossible to correctly interpret the behavior of the existing model 21.

[0042] Similarly, it is desirable that the distribution of the multiple explanatory data vectors xn coincides with the range in which the existing model 21 is expected to be used. For example, in the case of an existing model 21 that predicts the behavior of a typical adult, it is desirable that the element of "human age" be close to the distribution of data such as demographics.

[0043] The control unit 11 inputs the explanatory data vector x acquired in step S501 into the existing model 21 to acquire the target data vector yn (step S502). The control unit 11 associates the explanatory data vector xn with the target data vector yn and records them in the main storage device 12 or the auxiliary storage device 13 (step S503).

[0044] The control unit 11 determines whether to end the generation of pairs of the explanation data vector x and the target data vector yn (step S504). For example, the control unit 11 determines to end the process in step S504 when the control unit 11 has finished processing the explanation data vector x recorded in the training data. The control unit 11 may also determine to end the process when a predetermined number of pairs have been generated.

[0045] If it is determined not to end (NO in step S504), the control unit 11 returns to step S501. If it is determined to end (YES in step S504), the control unit 11 generates an explanation matrix X based on the data recorded in step S503 (step S505). The control unit 11 generates a target matrix Y based on the data recorded in step S503 (step S506). The control unit 11 calculates a target inverse matrix Y†, which is a Moore-Penrose generalized inverse matrix, based on the target matrix Y (step S507). The control unit 11 calculates an interpretation matrix A†, which is the vector product of the explanation matrix X and the target inverse matrix Y† (step S508). The control unit 11 ends the processing.

[0046] The following provides an overview of a method for interpreting the behavior of the existing model 21 by graphically representing the generated interpretation matrix A†.

[0047] [Global Feature Importance] Global feature importance refers to the degree to which each element constituting the explanatory data vector x contributes to a specific prediction result output from the existing model 21. The user can understand the overall behavior of the existing model 21 based on the visualized global feature importance.

[0048] As mentioned above, the element in row a, column b of the interpretation matrix A† indicates the influence of the a-th element of the explanation data vector xn on the b-th element of the target data vector yn. For example, the degree of influence that each element of the explanation data vector xn has on a specific item in the target data vector yn can be visualized using a bar graph or line graph in which the item names corresponding to each element of the explanation data vector xn are used on the first axis and the values ​​of any column of the interpretation matrix A† are used on the second axis. Specific examples of charts that visualize global feature importance will be described later.

[0049] [Representative Estimation Instance] The representative estimation instance x* means an ideal or typical explanatory data vector xn in which the target data vector yn output from the existing model 21 is in a specific state. The representative estimation instance x* is a vector having the same number of elements as the explanatory data vector xn.

[0050] The representative inferred instance x* is calculated by the vector product of the interpretation matrix A† and the target data vector yn. The formula for calculating the representative inferred instance x* is shown in formula (7). x* = A† yn (7)

[0051] In the following explanation, an example will be described in which the target data vector yn is a unit target vector yuk, where the k-th element is 1 and the other elements are 0. k is a natural number equal to or less than the number of elements in the target data vector yn. When the target data vector yn is a unit target vector yuk, the formula for calculating the representative estimated instance x* is shown in formula (8), and the definition of the i-th element yuk(i) of the unit target vector yuk is shown in formula (9).

[0052]

[0053] Based on the representative inferred instance x*, the user can understand the importance of each item of the explanatory data vector xn when a specific target data vector yn is output. Furthermore, based on the representative inferred instance x*, the user can obtain clues for interpreting the behavior of the existing model 21. Specific examples of the representative inferred instance x* will be described later.

[0054] [Local Feature Importance] The local feature importance vector L represents the degree to which each element constituting a specific explanatory data vector x contributes to the target data vector yn output from the existing model 21. The local feature importance vector L is a vector having the same number of elements as the explanatory data vector xn.

[0055] In the following description, a specific explanation data vector xn will be referred to as a target explanation vector xo. The target explanation vector xo may be data included in the training data or data created separately from the training data. An example will be described in which the target data vector yn is a unit target vector yuk. The local feature importance vector L is calculated using equation (10).

[0056]

[0057] For example, the local feature importance vector L can be visualized using a bar graph or line graph in which the first axis represents the item names corresponding to the elements constituting the local feature importance vector L, i.e., the item names corresponding to the elements constituting the explanation data vector x, and the second axis represents the values ​​of the elements constituting the local feature importance vector L. Specific examples of diagrams for visualizing the local feature importance vector L will be described later.

[0058] Based on the local feature importance vector L, the user can interpret why the target data vector yn was output from the existing model 21 that accepted the input of the explanatory data vector xn. The user can also understand the elements of the explanatory data vector xn that affect the target data vector yn based on the local feature importance vector L. Specific examples of the local feature importance vector L will be described later.

[0059] [Similarity Distribution Map] Figure 5 is an explanatory diagram outlining a method for creating the similarity distribution map 43. First, two representative estimated instances x*, a first typical example vector and a second typical example vector, are calculated based on equation (8). Different values ​​of k are used when calculating the first typical example vector and when calculating the second typical example vector.

[0060] An explanatory data vector xn that satisfies a specific condition is selected from the explanation matrix X. For example, if training data used in machine learning is used for the explanatory data vector xn, it is desirable to select an explanatory data vector xn that corresponds to the unit objective vector yuk when the first typical example vector is calculated using the correct answer data. Of the objective data vectors yn output from the existing model 21, an explanatory data vector xn whose k-th element is greater than a predetermined threshold may be selected.

[0061] A first similarity, which is the similarity between the selected explanatory data vector x and the first typical example vector, is calculated. The similarity may be, for example, the inverse of the inter-vector distance between the explanatory data vector x and the first typical example vector, defined by an arbitrary definition such as the Euclidean distance, Manhattan distance, or Hamming distance. The similarity may also be, for example, cosine similarity.

[0062] Similarly, a second similarity, which is the similarity between the selected explanatory data vector xn and the second typical example vector, is calculated. Each explanatory data vector xn is plotted on a two-dimensional graph, with the first similarity value on the x-axis and the second similarity value on the y-axis.

[0063] Kernel density estimation is performed on the plotted data by applying, for example, an RBF kernel, to generate a smooth similarity distribution diagram 43. The similarity distribution diagram 43 allows the user to intuitively recognize the behavior of the existing model 21 with respect to the selected explanatory data vector xn. Note that individual plots may be displayed directly without performing kernel density estimation.

[0064] Kernel density estimation may be performed on plotting the individual explanatory data vectors xn, and the plots are overlaid to produce a smooth affinity map 43.

[0065] Similarity distribution diagrams 43 created using explanatory data vectors x selected under multiple conditions may be superimposed and displayed. Three representative estimated instances x* may be used to generate a three-dimensional similarity distribution diagram 43. Specific examples of similarity distribution diagrams 43 will be described later.

[0066] 6 is a flowchart illustrating the processing flow of a program for generating a similarity distribution diagram 43. The control unit 11 receives a specification for the first explanatory data vector xn (step S521). For example, the control unit 11 receives a specification for k and uses the unit objective vector yuk as the first explanatory data vector xn. This unit objective vector yuk is an example of a first unit vector. The control unit 11 calculates a representative estimated instance x* based on equation (7). The control unit 11 uses the calculated representative estimated instance x* as the first typical example vector (step S522).

[0067] The control unit 11 accepts a specification regarding the second explanation data vector xn (step S523). For example, the control unit 11 accepts a specification of k and uses the unit objective vector yuk as the second explanation data vector xn. This unit objective vector yuk is an example of a second unit vector. The control unit 11 calculates a representative estimated instance x* based on equation (7). The control unit 11 uses the calculated representative estimated instance x* as the second typical example vector (step S524).

[0068] The control unit 11 selects one explanatory data vector xn that meets predetermined conditions from the explanatory matrix X (step S525). The control unit 11 calculates the similarity between the explanatory data vector xn selected in step S525 and the first typical example vector calculated in step S522 (step S526). The control unit 11 calculates the similarity between the explanatory data vector xn selected in step S525 and the second typical example vector calculated in step S524 (step S527). The control unit 11 associates the similarity calculated in step S526 with the similarity calculated in step S527, and records them in the main storage device 12 or the auxiliary storage device 13 (step S528).

[0069] The control unit 11 determines whether to end the calculation of the similarity (step S529). Specifically, for example, the control unit 11 determines to end the calculation of the similarity when the processing of the explanation data vector xn that satisfies a predetermined condition in the explanation matrix X is completed. The control unit 11 may also determine to end the calculation of the similarity when the calculation of the similarity is repeated a predetermined number of times.

[0070] If it is determined not to end (NO in step S529), the control unit 11 returns to step S525. If it is determined to end (YES in step S529), the control unit 11 creates a scatter plot for each of the data recorded in step S528, with the similarity to the first typical example vector on the first axis and the similarity to the second typical example vector on the second axis (step S530).

[0071] The control unit 11 performs kernel density estimation on the scatter diagram created in step S530 to smooth the distribution of each plot in the scatter diagram (step S531). This completes the similarity distribution diagram 43. The control unit 11 displays the completed similarity distribution diagram 43 (step S532). The control unit 11 then terminates the process.

[0072] [Specific Example 1: Titanic Dataset] An example will be described in which an existing model 21 is interpreted, which has been generated using the Titanic dataset, which summarizes the survival status of passengers on the Titanic, which sank in the North Atlantic Ocean after being hit by an iceberg in 1912, as training data.

[0073] The existing model 21 is a learning model trained to output the probability of survival and the probability of non-survival for a passenger when a 12-element explanatory data vector xn related to the passenger is input. Since the method for generating a machine learning model using supervised machine learning is well known, a description of the generation method will be omitted. The elements of the explanatory data vector xn are shown in Table 1.

[0074]

[0075] In Table 1, the eight items from first-class cabin to departure from Port S are expressed as binary values: 1 for YES and 0 for NO.

[0076] 7 is an explanatory diagram illustrating the record layout of the explanation-purpose DB 31. The explanation-purpose DB 31 has a No. field, an explanation data vector field, and a purpose data vector field. The explanation data vector field has subfields corresponding to each element of the explanation data vector xn shown in Table 1, such as an Age field, a SibSp field, and a Parch field. The purpose data vector field has a survival probability field and a non-survival probability field.

[0077] The No. field records consecutive numbers from 1 to N. Each subfield of the explanatory data field records an element of the explanatory data vector xn. The survival probability field records the probability that a passenger with the attributes of the explanatory data vector xn will survive. The non-survival probability field records the probability that a passenger with the attributes of the explanatory data vector xn will not survive. For each record, the sum of the value recorded in the survival probability field and the value recorded in the non-survival probability field is 1.

[0078] In step S503 of the program described using Fig. 4, the control unit 11 adds one record to the explanation-objective DB 31. The part surrounded by the dashed line in Fig. 7 is the transposed matrix of the explanation matrix X shown in Fig. 2. The part surrounded by the solid line in Fig. 7 is the transposed matrix of the objective matrix Y shown in Fig. 2.

[0079] Fig. 8 is an explanatory diagram illustrating the global feature importance graph 41. The upper part of Fig. 8 schematically shows the interpretation matrix A† calculated in step S508 of the program described using Fig. 4. The interpretation matrix A† is a matrix with 12 rows and 2 columns. The vertical numbers of the elements of the interpretation matrix A†, i.e., the third character, are expressed in hexadecimal. That is, A represents 10, B represents 11, and C represents 12.

[0080] As mentioned above, the element in row a, column b of the interpretation matrix A† indicates the influence of the a-th element of the explanation data vector xn on the b-th element of the objective data vector yn. Thus, AD11 indicates the influence of the passenger's age on the passenger's survival, and AD12 indicates the influence of the passenger's age on the passenger's non-survival.

[0081] The lower part of Fig. 8 shows a global feature importance graph 41. The vertical axis shows the item names of the explanation data vector xn. The horizontal axis shows the values ​​of each element of the interpretation matrix A†. The hatching sloping downward to the left indicates the values ​​of the left column of the interpretation matrix A†, i.e., the influence of each element on the passenger's survival. The hatching sloping downward to the right indicates the values ​​of the right column of the interpretation matrix A†, i.e., the influence of each element on the passenger's non-survival.

[0082] 8, the number of fellow passengers A has almost no effect on whether a passenger will survive. Therefore, it can be seen that data on the number of fellow passengers A is not necessary to predict whether a passenger will survive. For example, by deleting the number of fellow passengers A from the explanatory variables, the user can reduce the amount of calculation of the existing model 21 without affecting the prediction accuracy.

[0083] Similarly, Figure 8 shows that the factor that has the greatest influence on whether a passenger survives is whether the passenger is female, and the factor that has the greatest influence on whether a passenger does not survive is whether the passenger is male. It can be seen that the gender of a passenger has a greater influence on whether or not they survive than other factors.

[0084] Table 2 shows representative estimated instances for survivors and non-survivors calculated based on equation (8).

[0085]

[0086] Table 2 shows that the typical survivor is a woman with a relatively high passenger fare, while the typical non-survivor is a man with a relatively low passenger fare.

[0087] Next, an example of local feature importance will be described. To calculate local feature importance, object explanation vectors x were created for two characters, Rose, a survivor, and Jack, a non-survivor, named after characters from the 1997 film Titanic. The created object explanation vectors x are shown in Table 3.

[0088]

[0089] 9A and 9B are examples of a local feature importance graph 42. The local feature importance vector L for Rose's survival, calculated based on equation (10), is shown in the local feature importance graph 42 of FIG. 9A. The transposed matrix of (1, 0) is used for the unit objective vector yuk. The vertical axis indicates the item names of the object explanation vector xo. The horizontal axis indicates the values ​​of each element of the local feature importance vector L. According to FIG. 9A, the first reason for Rose's survival is that she is a woman, and the second reason is that she is a first-class passenger.

[0090] The local feature importance vector L for Jack's non-survival calculated based on equation (10) is shown in the local feature importance graph 42 of FIG. 9B. The unit objective vector yuk uses the transposed matrix of (0, 1). The vertical axis indicates the item names of the object explanation vector xo. The horizontal axis indicates the values ​​of each element of the local feature importance vector L. According to FIG. 9B, the first reason why Jack will not survive is that he is a male, and the second reason is that he is a passenger in a third-class cabin.

[0091] 9A and 9B, the interpretation matrix A can be used to create a local feature importance graph 42 for any target explanation vector x. The user can also use the local feature importance graph 42 to provide interpretations for explanation data vectors x that are not included in the explanation matrix X.

[0092] 10 to 12 are examples of similarity distribution diagrams 43. FIG. 10 shows a similarity distribution diagram 43 for survivors. An overview of the processing for creating FIG. 10 will be described. The first typical example vector calculated by the control unit 11 in step S522 of the program described using FIG. 6 is equal to the representative inferred instance of a non-survivor shown in Table 2. The second typical example vector calculated by the control unit 11 in step S524 is equal to the representative inferred instance of a survivor shown in Table 2. In step S525, the control unit 11 selected an explanatory data vector xn for a survivor from the explanatory matrix X.

[0093] The horizontal axis of FIG. 10 indicates the non-survival score, which corresponds to the similarity to the representative inferred instance of the non-survivor. A larger non-survival score means a greater similarity to the representative inferred instance of the non-survivor. The vertical axis of FIG. 10 indicates the survival score, which corresponds to the similarity to the representative inferred instance of the survivor. A larger survival score means a greater similarity to the representative inferred instance of the survivor. The finer the hatching downward to the left, the denser the distribution of the explanatory data vector xn related to the survivor. FIG. 10 is an example of a first distribution diagram.

[0094] Figure 11 shows a similarity distribution diagram 43 for non-survivors. An overview of the process for creating Figure 11 will be described. The processes in steps S522 and S524 of the program described using Figure 6 are the same as those used to create Figure 10. In step S525, the control unit 11 selected an explanatory data vector xn for non-survivors from the explanatory matrix X.

[0095] The vertical and horizontal axes in Figure 11 are the same as those in Figure 10, and therefore will not be described further. The finer the hatching that slopes downward to the right, the denser the distribution of explanatory data vectors xn related to non-survivors. Figure 11 is an example of a second distribution diagram.

[0096] Figure 12 shows a similarity distribution diagram 43 in which Figures 10 and 11 are superimposed. That is, the distribution for survivors and the distribution for non-survivors are superimposed on one diagram. Open circles indicate the non-survival scores and survival scores for Rose described in Table 3. Closed circles indicate the non-survival scores and survival scores for Jack described in Table 3.

[0097] 12, there is an area where the distribution of survivors and the distribution of non-survivors overlap. The user can understand that in this area it is difficult to predict whether a passenger will survive or not. FIG. 12 is an example of a similarity distribution map 43 in which the first distribution map and the second distribution map are superimposed.

[0098] If it is possible to obtain values ​​for the intermediate layers of the existing model 21, interpretation matrices A† may be generated for the part from the input layer that receives the explanation data vector xn to the intermediate layer, and for the part from the intermediate layer to the output layer that outputs the target data vector yn. Generating the interpretation matrices A† in a layered manner may increase the interpretability of the existing model 21.

[0099] [Specific Example 2: MNIST (Mixed National Institute of Standards and Technology database) Dataset] An example of interpreting an existing model 21 for classifying handwritten digits, which was generated using the "MNIST dataset" as training data, will be described. The MNIST dataset is a dataset in which handwritten digit images, each composed of 28 8-bit grayscale pixels arranged vertically and horizontally, for a total of 784 pixels, are recorded in association with the numbers that are the correct answer data.

[0100] The existing model 21 is a learning model trained to output the probability that a grayscale image of 28 pixels in each direction is a digit between 0 and 9 when input. The explanatory data vector xn is a 784-element vector in which the brightness of 784 pixels is arranged in a predetermined order. The target data vector yn is a 10-element vector in which the probability of each digit between 0 and 9 is arranged.

[0101] Fig. 13 is an explanatory diagram illustrating local feature importance for the existing model 21. As shown in the upper part of Fig. 13, the explanation data vector xn is input to the existing model 21, and the process of obtaining the target data vector yn is repeated to calculate the interpretation matrix A† using the procedure described using Fig. 2. The interpretation matrix A† is a matrix with 784 rows and 10 columns.

[0102] The lower part of Fig. 13 shows three examples of local feature importance vectors L in table format. When a number image of the target explanation vector x is input to the existing model 21, the local feature importance vectors L are shown for cases where the number is determined to be 0, 3, and 8. Note that Fig. 13 shows each handwritten character and local feature importance vector L converted into binary black and white. When the number is determined to be a number, the pixel that is given importance is displayed in black.

[0103] The local feature importance vector L allows the user to recognize that if the target explanation vector x is determined to be 0, then the top and bottom portions of the handwritten character contribute; if it is determined to be 3, then the right portion of the handwritten character contributes; and if it is determined to be 8, then almost the entire handwritten character contributes.

[0104] 14 is an example of a similarity distribution diagram 43 for the existing model 21. The horizontal axis represents the similarity score with respect to the representative inferred instance x* determined to be "3." The vertical axis represents the similarity score with respect to the representative inferred instance x* determined to be "8." The open circles indicate the positions where the target explanation vector xo shown in the lower right of FIG. 13 is plotted.

[0105] In the MNIST dataset, the distribution of the explanation data vector xn for which the correct answer data is "8" is shown by hatching sloping downward to the left, and the distribution of the explanation data vector xn for which the correct answer data is "3" is shown by hatching sloping downward to the right. The darker the hatching, the denser the distribution.

[0106] Since there is a large overlap between the "3" area and the "8" area, the user can recognize that it is easy to mistakenly determine that 3 is 8. Although not shown in the figure, for example, the "0" area and the "1" area do not overlap, so the user can also recognize that it is difficult to mistakenly determine that "0" is 1.

[0107] The existing model 21 may be, for example, a learning model that accepts input of a sentence and outputs a keyword. The existing model 21 may be, for example, a learning model that accepts input of an image and outputs the name of the subject. For any other existing model 21, the interpretation matrix A† can be calculated and used to interpret behavior.

[0108] [Modification 1] This modification relates to a method for quickly generating an interpretation matrix A†. Explanation of parts common to embodiment 1 will be omitted. In this modification, the existing model 21 is generated by supervised machine learning.

[0109] Instead of repeatedly inputting the explanatory data vector xn into the existing model 21 and obtaining the target data vector yn, the control unit 11 extracts pairs of the explanatory data vector xn and the target data vector yn from the training data and generates the explanatory matrix X and the target matrix Y.

[0110] According to this modification, the interpretation matrix A† can be calculated before completing the generation of the existing model 21. For example, a user can predict the behavior of the existing model 21 using the interpretation matrix A† before generating the existing model 21, and take measures such as increasing the amount of training data if the existing model 21 does not have the desired characteristics.

[0111] [Modification 2] This modification relates to a method of regularizing the target matrix Y using Ridge regression, that is, L2 norm regularization. Explanation of parts common to the first embodiment will be omitted.

[0112] In this modification, the control unit 11 calculates the generalized inverse matrix Y† of Y using equation (11) instead of equation (2) in step S507 of the flowchart described with reference to Figure 4: Y† = YT (YYT - λE) -1 (11) where λ is a positive real number, and E is an M-row, M-column unit matrix.

[0113] According to this modification, an appropriate interpretation matrix A† can be calculated even for a complex existing model 21 having a large number of parameters or interactions between parameters.

[0114] [Modification 3] This modification relates to a method of regularizing the target matrix Y using Lasso regression, that is, L1 norm regularization. Explanation of parts common to the first embodiment will be omitted.

[0115] In this modified example, in step S507 of the flowchart described using Figure 4, the control unit 11 calculates the generalized inverse matrix Y† of Y, and then regularizes the generalized inverse matrix Y† of Y using a known iterative algorithm.

[0116] This modification also makes it possible to calculate an appropriate interpretation matrix A† for a complex existing model 21 having a large number of parameters or interactions between parameters. Note that the regularization method is not limited to the Ridge regression of modification 2 and the Lasso regression of this modification. For example, a known method such as ElasticNet can also be used for regularization.

[0117] A program is an example of a program product. A computer program can be deployed to be executed on a single computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

[0118] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should be considered not to be limiting. The scope of the present invention is defined by the claims, not by the meaning described above, and is intended to include all modifications within the meaning and scope of the claims.

[0119] Independent and dependent claims may be combined with each other in any combination, regardless of the reference format. Furthermore, while the claims use a format in which a claim references two or more other claims (multiple claim format), this is not a limitation. Multiple claims that reference at least one other claim (multi-multi claim format) may also be used.

[0120] 10 Information processing device 11 Control unit 12 Main memory device 13 Auxiliary memory device 14 Communication unit 15 Display unit 16 Input unit 19 Reading unit 21 Existing model (existing machine learning model) 31 Explanation-purpose DB 41 Global feature importance graph (diagram) 42 Local feature importance graph (diagram) 43 Similarity distribution map (diagram) 96 Portable recording medium 97 Program 98 Semiconductor memory

Claims

1. An information processing method in which a computer executes the processes of: correlating and recording multiple sets of explanatory data vectors input to an existing machine learning model with target data vectors output from the machine learning model; calculating an interpretation matrix which is the vector product of an explanation matrix in which multiple sets of the explanatory data vectors are arranged and a generalized inverse matrix of a target matrix in which the target data vectors are arranged in an order corresponding to the explanatory data vectors; and outputting a chart related to the interpretation matrix.

2. The information processing method according to claim 1, wherein the generalized inverse matrix of the target matrix is ​​a Moore-Penrose generalized inverse matrix of the target matrix.

3. The information processing method according to claim 1, wherein the chart is a graph in which the first axis represents the item names corresponding to the individual elements of the explanatory data vector, and the second axis represents the element values ​​for each column constituting the interpretation matrix.

4. An information processing method as described in claim 1, further comprising the steps of: obtaining an explanatory data vector; generating a unit objective vector from the objective data vector in which one element that is the target for displaying local feature importance is 1 and the other elements are 0; and displaying the diagram as a graph in which the first axis represents the item names corresponding to the individual elements of the explanatory data vector, and the second axis represents the values ​​of the elements that make up the local feature importance vector calculated by equation (1).

5. The information processing method according to claim 1, wherein a first typical example vector is calculated which is a vector product of the interpretation matrix and a first unit vector of the target data vector, one element of which is 1 and the other elements are 0; a second typical example vector is calculated which is a vector product of the interpretation matrix and a second unit vector of the target data vector, the other elements of which are 1 and the other elements are 0; and the diagram is a distribution chart obtained by plotting the similarity with the first typical example vector on the horizontal axis and the similarity with the second typical example vector on the vertical axis for each of the explanatory data vectors selected from the explanation matrix, and then performing kernel density estimation.

6. The information processing method of claim 5, wherein the explanatory data vector is identical to the explanatory data in the training data used for machine learning of the machine learning model, and the diagram is a superimposed display of a first distribution chart created using explanatory data corresponding to one element when the first typical example vector is calculated from the correct data in the training data, and a second distribution chart created using explanatory data corresponding to one element when the second typical example vector is calculated from the correct data in the training data.

7. An information processing method according to any one of claims 1 to 5, wherein the explanatory data vector is identical to explanatory data in training data used for machine learning of the machine learning model.

8. A program that causes a computer to execute the following processes: associate and record multiple sets of explanatory data vectors input to an existing machine learning model and target data vectors output from the machine learning model; calculate an interpretation matrix which is the vector product of an explanatory matrix in which multiple sets of the explanatory data vectors are arranged and a generalized inverse matrix of a target matrix in which the target data vectors are arranged in an order corresponding to the explanatory data vectors; and output a chart related to the interpretation matrix.

9. An information processing device having a control unit, wherein the control unit: records multiple sets of explanatory data vectors input to an existing machine learning model and target data vectors output from the machine learning model in association with each other; calculates an interpretation matrix which is the vector product of an explanation matrix in which multiple sets of the explanatory data vectors are arranged and a generalized inverse matrix of a target matrix in which the target data vectors are arranged in an order corresponding to the explanatory data vectors; and outputs a chart related to the interpretation matrix.

Images