A method for automatically recommending explainability methods for machine learning models

Abstract

The application discloses a method for automatically recommending explainability for machine learning models, including the following steps: step 1, constructing a target model. Step 2, modeling the explainability requirement analysis framework. Analyze the target to indicate what goals need to be achieved to solve the problem, and the quality target indicates which quality attributes may be involved when achieving this goal. Then model the solution. Step 3, design a prototype tool to implement the framework, design a prototype tool to implement the modeling results in step 2. Finally, the participants are experimented to verify and evaluate the feasibility of the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of explainability requirements, and in particular to automatically recommending suitable explainability methods for the design of explainable parts of machine learning models. BACKGROUND

[0002] Machine learning models are well known and widely used, but as black-box models, the process of making predictions can be difficult for humans to understand. An explanation of the model prediction process can increase user trust in the model, especially in many important fields such as the medical field, where the trend of using machine learning for high-risk predictions is growing, so users need to know how a machine learning system makes a particular decision or how a set of inputs produces a particular output. At the same time, the lack of explainability also hinders the application of the latest machine learning models in traditional industries such as banks, finance, securities, etc.

[0003] At the same time, explainability as a new emerging non-functional requirement is increasingly valued by people, and from the perspective of different stakeholders, different requirements can be generated. Markus Langer et al. divide stakeholders into five categories: users, system developers, affected parties, deployment personnel, and legislators. For developers, explanations can provide insights into the model prediction process and help improve the model, while for users, explanations can increase the transparency and explainability of the model.

[0004] So far, people have developed a variety of explainability methods, and their implementation principles are different, and their respective advantages and disadvantages are also different, such as: some methods can only explain specific models, and some explainability methods can be used to explain any model. In particular, model-agnostic explainability methods, although they all claim to be suitable for any machine learning system, there is no clear conclusion on which method is most suitable in specific situations. In addition, many machine learning model developers have limited knowledge of explainability, and it is very difficult to choose an explainability method suitable for their own model without relevant knowledge base. Even when it comes to the demand for explainability, many people cannot express this demand well. In addition, most of the current work focuses on the development of explainability methods, and there is a lack of a systematic method to guide users to use these explainability methods.

[0005] Recently, many researchers have proposed explainability research for software systems, and Larissa Chazette et al. proposed a framework to help define and implement explainability requirements, promoting the analysis, operation and evaluation of explainability, but the implementation of the framework requires high human involvement and a large amount of explainability-related knowledge, making it difficult to implement.

[0006] Therefore, the present application proposes an adaptive explainability framework that automatically recommends explainability methods through the collection of context information. The framework integrates context knowledge, extracts explainability requirements from users, converts them into user context, and extracts dataset features into data context. Through the verification of the context, the framework automatically recommends explainability methods. The framework weakens the degree of users' requirements for explainability knowledge and simplifies the process of users selecting explainability methods.

[0007] Although there may not be many explainability methods specific to a particular field, users can be directly given these methods to understand and choose them. However, due to the rapid development of explainability, a large number of explainability methods will inevitably appear in the future, so this approach will bring a heavy burden to users. Vilone et al. analyzed relevant data and found that there were 208 explainability methods by 2020, and the growth rate was very rapid. In the face of such a situation, the present application analyzes existing review literature and collects 19 mainstream explainability methods as basic methods. The framework of the present application integrates these methods and provides a more convenient selection method, which can shorten the time spent by users in making choices to the greatest extent. SUMMARY

[0008] The present application proposes a context-based framework for automatically recommending explainability methods for machine learning models, namely an explainability requirement analysis framework. The framework provides a method for explainability novices to automatically recommend the most suitable explainability method for the model at hand. Users only need to verify some contexts to get the recommended method. When designing the explainability part of the machine learning model using the framework, the model developer does not need to have much explainability-related knowledge, and the importance of the framework increases with the increase in the number of explainability methods. It can be expected that there will be hundreds or thousands of explainability methods in the future, and manual selection will undoubtedly waste a lot of time, while using the framework can quickly get the recommended results.

[0009] The present application abstracts the explainability requirement analysis framework into three parts: problem, solution, and context, and models these three concepts respectively. The final modeling is shown in Figure 2 .

[0010] The technical scheme adopted by the present application is a method for automatically recommending explainability methods for machine learning models, which mainly includes the following steps:

[0011] Step 1, building the goal model. In this step, three main concepts are first proposed, which are problem, solution, and context. Specifically, problem represents the goal to be achieved, solution represents the methods that can be used to achieve this goal, and context represents the situation in which different solutions are applicable. In this study, the goal model is used to represent these three concepts, as shown in Figure 1 .

[0012] Step 2, modeling the explainability requirement analysis framework. In this step, the goal model in step one is refined, as shown in Figure 2 . First, the three concepts of problem, solution, and context are modeled respectively. Specifically, problem can be represented as goal, and goal can be represented as analysis goal and quality goal. Analysis goal represents the goals that need to be achieved to solve the problem, and quality goal represents the quality attributes that may be involved in achieving this goal. Then, the solution is modeled. Each analysis goal corresponds to different solutions, and indicators can evaluate the effectiveness of each solution and analysis goal. Some solutions can be further divided into several sub-solutions, and the performance of sub-solutions in quality is different. Finally, the context is modeled, which can be divided into user context, data context, and method context. Specifically, user context represents the user's requirements and other information, data context represents the characteristics of machine learning models or datasets, and method context represents the parameters and attributes that need to be set for explainability methods. Context can be applied to the connection between solution and analysis goal to represent the applicability of solution, and can also be applied to analysis goal to represent the effective scenario of the analysis goal.

[0013] Step 3, designing a prototype tool to implement the framework. In this step, a prototype tool is designed to implement the modeling results in step 2, and the UML class diagram of the prototype tool is shown in Figure 4 . Specifically, requirement extractor and data extractor are used to extract user context and data context respectively, and then the context information is input into context monitor. Context monitor is responsible for verifying the effectiveness of context, and inputting the effectiveness of context into context planner. Context planner converts context information into binary stream information and inputs it into explainability method recommender. Finally, explainability method recommender recommends the most suitable explainability method for the current context according to the binary stream information.

[0014] Finally, an experiment was conducted with 10 participants to evaluate the performance of the framework. Specifically, the feasibility, expressiveness and usability of the framework were evaluated. They are experts in the fields of computer vision, text classification and entity extraction, etc. Specifically, two of them have a good understanding of explainability, four of them have some understanding of explainability, and the remaining four have no understanding of explainability at all.

[0015] First, some relevant knowledge was introduced to users who were not familiar with explainability, so that they could understand the importance of explainability and how to apply it to their own work. Then, an experiment was conducted with all users to let them use the framework to design the explainability part of their own machine learning systems according to the recommended scheme. Second, a questionnaire was used to collect user feedback, and the questionnaire mainly asked questions about the feasibility, expressiveness and usability of the framework. After collecting the feedback from the questionnaire, some less-than-ideal feedback was interviewed, and after further understanding the reasons behind it, the following conclusions were drawn:

[0016] The results show that all users believe that the framework is helpful for designing explainability, but some users believe that it is not very helpful because they do not pay much attention to the explainability of machine learning systems, especially for image classification tasks, they are more concerned about how to improve the accuracy of indicators, rather than whether the system is explainable. However, they believe that the framework is helpful if they need to design explainability for machine learning systems in the future.

[0017] Second, regarding the expressiveness of the framework, the feedback of users with different levels of knowledge is different. For those who fully understand explainability, they believe that the framework can fully express the relevant knowledge of explainability, and these contexts are easy to verify. For those who only have some understanding or no understanding at all, the framework contains insufficient relevant knowledge, and it is difficult to verify the context, but they also indicate that the decomposed context is easier to verify, indicating that the decomposition of the context is necessary for beginners.

[0018] Finally, regarding the usefulness of the framework, user feedback indicates that the framework provides an effective way to automatically recommend explainability methods, especially for beginners, as the framework encapsulates a large amount of explainability knowledge, which can help them quickly design the explainability part of machine learning systems. However, some users still indicate that there will be some difficulties in using the framework, mainly in the lack of understanding of explainability and the difficulty in verifying the context. At the same time, some individual users indicate that the recommended explainability methods do not meet their needs very well, which indicates that the adaptability of the framework for some fields and tasks is not good enough. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 Problem analysis framework for goal model representation.

[0020] Figure 2 Requirement analysis framework for explainability of the invention.

[0021] Figure 3 Decomposition process for user context C1.

[0022] Figure 4 Prototype tool diagram for the method. DETAILED DESCRIPTION

[0023] Step 1. In this step, three main concepts are first proposed, which are problem, solution and context. Specifically, problem represents the goal to be achieved, solution represents the methods that can be used to achieve this goal, and context represents the situation in which different solutions are applicable. In this study, goal model is used to represent the three concepts, as shown in Figure 1 .

[0024] Step 2. In this step, the three concepts of problem, solution and context are modeled respectively.

[0025] First, the problem part is modeled. Specifically, this problem can be represented as the requirement for explainability. Goal modeling is widely used in requirement engineering, which can capture the requirements of stakeholders, represent the goals of users with analysis goals, and capture the quality of different methods for achieving analysis goals with quality goals. Therefore, the explainability requirement is modeled based on the goal modeling method. The main modeling elements are analysis goals and quality goals. The analysis goal is the problem to be solved, i.e. to automatically recommend explainability methods, which expresses the functional requirements of stakeholders. This goal can be further decomposed into three sub-analysis goals, corresponding to the stages of explainability methods, which are: before model, in model and after model. Specifically, the method before model is suitable for the explanation of the dataset before machine learning model. In model means that the model itself is explainable, such as the sparsity, monotonicity and causality of the model itself. After model means to explain after the model is established, which is also called model-independent. The quality goal expresses the quality requirement of each explainability method for solving the problem, which is reflected in the link between quality goal, explainability method and contribution. For example: when feature importance ranking is selected as the explainability method, this method has a negative impact on the soft goal of "applicability to unknown data", Figure 2 indicated by (-) in .

[0026] After the goal modeling is done, the solution for achieving the goal is modeled. Here the solution refers to the explainability method, which describes the method for solving the analysis goal, which is reflected in the link between the analysis goal, the explainability method, and the means-end. Each explainability method has specific assumptions, which limit its applicability in a specific context. This will be explained in the next section. Different explainability methods have different implementation principles, and their evaluation methods are also different, so the evaluation indicators of the explainability method are modeled. The indicators indicate the aspects that should be focused on when evaluating the effectiveness of the explainability method, which is reflected in the link between the indicators, the explainability method, and the evaluation.

[0027] Finally, the context is modeled. The context can be used in the link between the means-end, contribution, and evaluation, and also in the analysis goal. Only when the context is established, the explainability method can achieve the corresponding analysis goal. When the context is applied to the contribution link, it represents the case where the algorithm makes a (positive or negative) contribution to the quality goal. On the other hand, the context applied to the evaluation link indicates when the indicators are applicable to the evaluation of the explainability method. Finally, the context applied to the analysis goal indicates the activation rule of the analysis goal.

[0028] The modeling of the context is divided into the following two parts:

[0029] (1). Define the context. The context is mainly divided into three parts: user context, data context, and method context. Table 1 shows some contexts involved in the method.

[0030] The user context is the context that needs to be verified according to the information provided by the user, which will directly affect the demand for explanation, the granularity of explanation, and the form of explanation, etc. Considering that the cultural background, technical background, and field expertise of the user may have an impact on the context verification process, the user's background is first defined, which can usually be defined as a novice or an expert (or someone between the two). Then the user's explainability needs are defined, which not only include the user's expected explanation form (text or image), the object of explanation (dataset or model), and the stage of applying explanation (before the event, during the event, and after the event) and other functional requirements, but also include the quality requirements that the user expects the explanation to have, such as the required time is short, the applicability to different types of data is good, etc. The data context is the context that is verified based on the information of the dataset (such as: feature distribution, data type), which expresses the features and attributes of the dataset.

[0031] The definition of data context contains two aspects, which are the definition of data set context and the definition of machine learning model context. The context of data set includes the type, size and density of data, and other contexts can be further extracted from the data set (such as sparsity, heterogeneity and correlation, etc.). Although most of the explainability methods are model-independent, there are still a few explainability methods that are only applicable to specific models, so the characteristics of the model are also used as a constraint for selecting explainability methods. The characteristics of the model include the type of the model (decision tree, deep neural network, etc.) and the task of the model (classification, regression or other) and other information.

[0032] The method context refers to the properties and parameters that need to be set for a specific explainability method (such as: the complexity of the proxy model, the number of prototypes and criticisms). The properties of the method include the type of explanation (text, image or audio), the density of explanation (amount of information or granularity of information) and the applicability (the ability of the explanation to adapt to the wishes or characteristics of a specific user). The parameters of the method are the parameter values that may need to be set when using the explainability method, and the accuracy of the explanation can be improved by setting the parameter values.

[0033] (2) Decomposition of context. After the definition of context is completed, each context needs to be verified, but for complex contexts, it may be difficult to directly verify them, so a substitute set is used to replace the verification of the context. These contexts are decomposed into a combination of facts and statements. Facts can be verified, and statements cannot be verified. Statements that cannot be directly verified can be decomposed into a combination of verifiable facts until the statement can be verified. The decomposition process of user context C4 is shown.

[0034] In Figure 3 , the world predicate wp represents a state in the real world, which can be divided into two categories, namely fractions and facts. Statements are represented by rectangles (w), and facts are represented by parallelograms (f). The relationship support is represented by a curved arrow, which indicates that the combination of the linked ends of the arrow provides evidence for the truth value of the statement pointed to by the arrow. The decomposition of the world predicate wp1 can be expressed as , which means that wp1 is true when w1 is true and f3 is true. The decomposition of the statement w1 can be expressed as , which means that w1 is true when f1 or f2 is satisfied. wp1 and wp2 can be expressed by the formulas , and the context C1 can also be expressed by the formula , which means that C4 is valid when wp1 and wp2 are satisfied at the same time.

[0035] Step 3. In this step, a prototype tool is designed to implement the modeling results in step 2, and the UML class diagram of the prototype tool is as followsFigure 4 The pseudo code of the prototype tool is shown in Table 2.

[0036] A requirement extractor and a data extractor are designed to extract user context and data context. In this process, the requirement extractor collects user's explainability requirements and user background, etc. through a form. The data extractor uses popular data analysis libraries such as NumPy, Pandas, SciPy, etc. in Python to analyze the dataset and model, and extracts information such as data type, density of dataset and type of model. Then the collected context information is input into the context monitor for further processing.

[0037] The context monitor is responsible for verifying the effectiveness of the context. For a specific model and dataset, the corresponding user context and data context are verified, and the explainability method parameters and attributes are set as method context. Finally, the context and its effectiveness are input into the context planner.

[0038] The context planner converts the context information into binary stream information, 0 representing that the context is not effective, and 1 representing that the context is effective. Finally, the binary stream information is input into the explainability method recommender.

[0039] The explainability method recommender contains a selection logic, which reads the binary stream information of the context, and verifies the entire binary stream. The result of the verification points to a specific explainability method, and gives the default parameters, which is the most suitable method recommended for the current machine learning model.

[0040] Table 1 Context

[0041] User context 1. User wants to interpret a dataset 2. User wants to interpret a machine learning model 3. User wants to obtain an explanation of the whole model 4. User wants to obtain an explanation of a set of instances 5. User wants to obtain an explanation of the relevance of features 6. User wants to obtain an explanation of the relevance of data points 7. User wants to use an interpretable model instead of a machine learning model 8. User wants to know the influence of a feature on the prediction 9. User wants to know the interaction between features 10. User wants to decompose a high-dimensional function into the sum of individual feature effects and interaction effects 11. User wants to know the importance of a feature 12. User wants to know the distribution of complex data 13. User wants to know the influence of a feature on the prediction for a single instance 14. User wants to know which new examples or changes in feature values can lead to a predefined prediction 15. User wants to find a decision rule that anchors a single prediction 16. User wants to know how the prediction is fairly distributed among the features 17. User wants to explain the predicted value as the sum of the attributed values for each input feature Data context 1. data type is table 2. data type is image 3. data type is text 4. number of features <= 25. NOT C4 6. data has heterogeneous effect 7. NOT C6 8. features are independent of each other 9. NOT C8 10. machine learning model itself is interpretable 11. NOT C10 12. machine learning model is DNN 13. data dimensionality is greater than 3 14. NOT C13 15. linear feature mapping 16. NOT C15 Method context 1. When using prototypes and critiques, need to set the number of prototypes and critiques and the cutoff value 2. When using LIME, need to define the fidelity measure 3. When using LIME, need to define the complexity of the surrogate model 4. When using LIME and the data type is a table, need to define the domain size 5. When using LIME, need to set the perturbation function 6. When using k-means, need to specify the value of K

[0042] Step 4. Finally, a prototype tool is designed to implement the framework, and the pseudo code of the prototype tool is shown as follows:

[0043] Table 2 Adaptive Explainability Method Recommendation Algorithm

[0044] Algorithm: Adaptive interpretability method recommendation n ), explainable method selection tree graph file G, detail partition tree graph group file P = (P1,..., P m ), m is the total number of all context nodes of G. Output: explainable method list L.]] ​ <![CDATA[dictTree = xmind_to_dict(G); / / Parse the xmind file into dictionary formatcreateTree(dictTree, tree); / / Build a tree structure according to the dictionary formattreewhile (C i ϵC) do / / Loop until the input endsif(C i ==0) then / / Select the left child nodetreenode = getLeftChild(treenode);end ifif(C i ==1) then / / Select the right child nodetreenode = getRightChild(treenode);end ifif(C i ==2) then / / Detail division, the left child node corresponds to the contextdictTree j = xmind_to_dict(P j ); / / Parse the xmind file of the j-th node of the tree structure tree P j into dictionary formatcreateTree(dictTree j , tree j ); / / Build a tree structure according to the dictionary format tree j while (getLeftChild(treenode j ) is not None or getRightChild(treenode j ) is not None) do / / Loop until the leaf nodeif(C i ==0) then / / Select the left child nodetreenode j = getLeftChild(treenode j );end ifif(C i ==1) then / / Select the right child nodetreenode j = getRightChild(treenode j );end ifif(getLeftChild(treenode j ) is None and getRightChild(treenode j ) is None) then / / Leaf nodeif(getRootVal(treenode j Then / / Leaf node value is true treenode = getLeftChild(treenode); / / Select the left node of the node before subdivision else / / Leaf node value is false treenode = getRightChild(treenode); / / Select the right node of the node before subdivision end if endif end while end ifif(C i ==3) then / / Detailed division of the right child node's corresponding context dictTree j = xmind_to_dict(P j / / The j-th node of the tree structure (tree) is in the XMind file P. j Parse into dictionary format: createTree(dictTree) j tree j / / Create a tree structure based on dictionary format j while (getLeftChild(treenode j ) is not None or getRightChild(treenode j () is not None) do / / Loop until leaf node if (C i ==0) then / / Select the left child node treenode j = getLeftChild(treenode j );end ifif(C i ==1) then / / Select the right child node treenode j = getRightChild(treenode j );end ifif(getLeftChild(treenode j ) is None and getRightChild(treenode j If () is None) then / / Leaf node if (getRootVal(treenode) j )) then / / leaf node value is true treenode = getRightChild(treenode); / / select right child of node before subdivision else / / leaf node value is false treenode = getLeftChild(treenode); / / select left child of node before subdivision end if end while end if if (getLeftChild(treenode) is None and getRightChild(treenode) is None) then / / leaf node L.append(getRootVal(treenode)); / / add corresponding method to list treenode = root; / / go back to root node for next selection end if end while

[0045] The above is a specific embodiment of the present application.

Claims

1. A method for automatically recommending interpretability methods for machine learning models, characterized in that, This method includes the following steps: Step 1: Construct the goal model. Three concepts are introduced: problem, solution, and context. Specifically, the problem represents the goal to be achieved, the solution represents the methods to be used to achieve the goal, and the context represents the situations in which different solutions are applicable. Step 2: Model the interpretability requirements analysis framework; In this step, the target model from step 1 is refined; firstly, the three concepts of problem, solution, and context are modeled separately. The problem is represented as the objective, and the objective is further represented as the analysis objective and the quality objective; The analysis objectives represent the goals that need to be achieved to solve the problem, while the quality objectives represent the quality attributes involved in achieving these goals. Next, the solution is modeled; the solution is subdivided into several sub-solutions, each with different quality performance. Finally, the context is modeled, divided into user context, data context, and method context. The user context represents user requirements, the data context represents the features of the machine learning model or dataset, and the method context represents the parameters and attributes that need to be set for interpretable methods. The context is applied to the connection between the solution and the analysis objectives to represent the applicability of the solution; when applied to the analysis objectives, it represents the effective scenarios for those objectives. Step 3: Design a prototyping tool to implement the framework; design a prototyping tool to implement the modeling results in Step 2; Use the demand extractor and data extractor to extract the user context and data context respectively, and then input the context information into the context monitor; The context monitor is responsible for verifying the effectiveness of the context and inputting the context effectiveness status into the context planner; The context planner converts contextual information into binary stream information and inputs it into the interpretability method recommender; Finally, the interpretability method recommender recommends the most suitable interpretability method for the current context based on the binary stream information; Context modeling includes decomposing the context, and the specific decomposition process is as follows: The world predicate wp represents a state in the real world, divided into two categories: statements and facts. Statements are represented by rectangles w, and facts by parallelograms f. Relationship support is represented by curved arrows, indicating that the combination linked at the ends of the arrows provides evidence of the truth value of the statement pointed to by the arrow. The decomposition of the world predicate wp1 is expressed as wp1 = w1 ∧ f3, meaning that wp1 is true when w1 is true and f3 is true. The decomposition of the statement w1 is expressed as w1 = f1 ∨ f2, meaning that w1 is true when either f1 or f2 is satisfied. wp1 and wp2 are expressed by formulas α and β, respectively. Similarly, the context C1 is expressed by the formula C1 = wp1 ∧ wp2, meaning that C1 is effective when both wp1 and wp2 are satisfied. w1 is the range that the user can choose to interpret. f3 indicates that the user is aware of the role of explainability; f1 indicates that the user wants a local explanation; f2 indicates that the user wants a global explanation. WP2 is for users who have an interpretability requirement.

2. The method for automatically recommending interpretability methods for machine learning models according to claim 1, characterized in that, Context modeling is further divided into the following two parts: (1) Define the context; the context is divided into three parts: user context, data context and method context; User context is the context that needs to be validated based on the information provided by the user. It directly affects the interpretation requirements, the granularity of the interpretation, and the form of the interpretation. First, define the user's background, and then define the user's interpretability requirements. Data context is the context for verifying information based on the dataset, expressing the characteristics and attributes of the dataset; The definition of data context includes two aspects: the definition of dataset context and the definition of machine learning model context. The dataset context includes the data type, size, and density, and the dataset is processed to extract other contexts. The model features include the model type and the model task. Method context refers to the attributes and parameters that need to be set for a specific interpretable method; method attributes include the type of interpretation, interpretation density, and applicability; method parameters are the parameter values ​​that need to be set when the interpretable method is used, and the accuracy of the interpretation is improved by setting the parameter values; (2) Decompose the context; After defining the contexts, each context needs to be validated. For complex contexts, a substitute set is used to replace the validation of the contexts. These contexts are decomposed into combinations of facts and statements. Facts can be validated, while statements cannot be validated. Statements that cannot be directly validated are further decomposed into combinations of validated facts until the statement is validated.

3. The method for automatically recommending interpretability methods for machine learning models according to claim 1, characterized in that, In step 3, a requirement extractor and a data extractor were designed to extract user context and data context; the requirement extractor collects the user's interpretable requirements and user background context through a form. The data extractor uses popular Python data analysis libraries such as NumPy, Pandas, and SciPy to analyze datasets and models, extracting the data types, densities, and model types of the datasets; then, the collected context information is input into the context monitor for processing.

4. The method for automatically recommending interpretability methods for machine learning models according to claim 1, characterized in that, For a specific model and dataset, validate its corresponding user context and data context, and set interpretability method parameters and properties as method context; finally, input the context and its effectiveness into the context planner.

5. The method for automatically recommending interpretability methods for machine learning models according to claim 1, characterized in that, The context planner converts the context information into binary stream information, where 0 indicates that the context is ineffective and 1 indicates that it is effective; finally, the binary stream information is input into the interpretability method recommender.

6. The method for automatically recommending interpretability methods for machine learning models according to claim 1, characterized in that, The interpretability method recommender includes a selection logic that reads the binary stream information of the context and verifies the entire binary stream. The verification result points to a specific interpretability method and provides default parameters. This interpretability method is the most suitable method recommended for the current machine learning model.

Images