Method and apparatus for interpreting a model

Abstract

This disclosure relates to a method and apparatus for interpreting a model. The method includes: determining a model to be interpreted as a first model based on a first type of parameter features; acquiring input data corresponding to the first model; analyzing the first model based on the input data to obtain a first interpretability scheme for interpreting the first model, wherein the interpretation parameters of the first interpretability scheme are determined based at least on the model attributes of the first model; interpreting the first model using the first interpretability scheme to obtain an interpretation result; and determining the interpretation relevance of the first model based on the interpretation result, wherein the interpretation relevance is used to characterize the interpretation effect of the interpretation parameters on the first model. This disclosure can automatically generate interpretability schemes for models, making complex machine learning models white-boxed, and allows for adjustment and optimization of model parameters based on interpretation results, thereby improving the transparency and credibility of the model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a method and apparatus for interpreting a model. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, more and more enterprises and individuals are applying AI to various aspects of production and life, using AI to analyze various types of data accumulated over a long period of time to solve practical problems encountered in actual production and life. However, many machine learning models are similar to "black boxes," which can output results after inputting data, but cannot understand their internal decision-making mechanisms and cannot meet the needs of business scenarios.

[0003] Existing technologies require manually customizing model interpretability solutions based on the specific business scenarios in which the model is used. This results in low automation, and users need to have a certain understanding of the model's principles and commonly used model interpretability algorithms. Furthermore, models with unknown business scenarios cannot be interpreted. Additionally, while existing technologies can achieve automatic model interpretation, they can only interpret simple models such as linear models, and cannot effectively interpret complex models such as deep neural networks. Summary of the Invention

[0004] This disclosure provides a method and apparatus for interpreting a model, which can solve the problem of poor model interpretability in the prior art.

[0005] According to one of the solutions disclosed herein, a method for interpreting a model is provided, comprising:

[0006] Based on the characteristics of the first type of parameters, the model to be explained is determined to be the first model;

[0007] Obtain the input data corresponding to the first model;

[0008] The first model is analyzed based on the input data to obtain a first interpretability scheme for interpreting the first model, wherein the interpretation parameters of the first interpretability scheme are determined at least based on the model attributes of the first model.

[0009] The first model is explained using the first interpretability scheme to obtain the explanation result;

[0010] Based on the explanation results, the explanatory relevance of the first model is determined, wherein the explanatory relevance is used to characterize the explanatory effect of the explanation parameters on the first model.

[0011] In some embodiments, the model attributes of the first model include the model task of the first model, and the step of analyzing the first model based on the input data to obtain a first interpretability scheme for interpreting the first model includes:

[0012] The input data is processed according to the model task of the first model;

[0013] The processed input data is input into the first model to obtain the first output result of the first model;

[0014] The first explanatory parameter of the first model is determined based on the first output result. The first explanatory parameter includes the importance of the input features in the input data to the first model.

[0015] In some embodiments, the method further comprises:

[0016] The input features in the input data are adjusted according to the model task of the first model, and the adjusted input features are input into the first model to obtain the second output result;

[0017] The second explanatory parameters of the first model are determined based on the second output result, wherein the second explanatory parameters include the way in which the input features affect the output result of the first model.

[0018] In some embodiments, the first interpretability scheme further includes an interpretation method for the first model, and the method further includes:

[0019] Based on the model attributes of the first model and / or the data attributes of the input data, the interpretation method of the first model is determined, wherein the data attributes of the input data include data type and / or data size.

[0020] In some embodiments, determining the explanatory relevance of the first model based on the explanation result includes:

[0021] Determine the contribution of the explanation parameters to the output of the first model;

[0022] The explanatory relevance of the first model is determined based on the contribution and the preset input features of the first model;

[0023] The input features of the first model are adjusted based on the explained relevance.

[0024] In some embodiments, determining the model to be explained as a first model based on the first type of parameter features includes:

[0025] Whether the model to be explained is a deep learning model is determined based on at least one of the model complexity, model structure, or model format of the model to be explained;

[0026] If so, the model to be explained is determined to be the first model.

[0027] In some embodiments, the method further comprises:

[0028] Based on the second type of parameter features, the model to be explained is determined to be the second model;

[0029] The second model is analyzed to obtain a second explanatory scheme for explaining the second model.

[0030] In some embodiments, the analysis of the second model to obtain a second explanatory scheme for explaining the second model includes:

[0031] Determine whether the second model includes feature engineering;

[0032] If included, the second interpretability scheme is configured based on the data cleaning, feature extraction, and feature selection methods in the feature engineering; and / or

[0033] The second interpretability scheme is configured based on the task category processed by the second model, wherein the task category includes at least one of regression, classification, clustering, or dimensionality reduction; and / or

[0034] The second interpretability scheme is configured based on whether there is input data suitable for the second model and whether the input data needs to be sampled.

[0035] In some embodiments, after determining the type of the model to be explained, the method further includes:

[0036] Based on the type of the model to be explained, it is determined whether there is a third interpretability scheme applicable to the model to be explained, wherein the third interpretability scheme is predetermined.

[0037] According to one of the solutions disclosed herein, a model interpretation apparatus is also provided, comprising:

[0038] The first determination module is configured to determine the model to be explained as the first model based on the first type of parameter features;

[0039] The acquisition module is configured to acquire the input data corresponding to the first model;

[0040] The analysis module is configured to analyze the first model based on the input data to obtain a first interpretability scheme for interpreting the first model, wherein the interpretation parameters of the first interpretability scheme are determined at least based on the model attributes of the first model.

[0041] The explanation module is configured to explain the first model using the first interpretability scheme and obtain the explanation result;

[0042] The second determining module is configured to determine the explanatory relevance of the first model based on the explanation results, wherein the explanatory relevance is used to characterize the explanatory effect of the explanation parameters on the first model.

[0043] The various embodiments of this disclosure provide a model interpretation method and apparatus that, after determining the model to be interpreted as a first model based on a first type of parameter features, analyzes the first model based on the input data corresponding to the first model to obtain a first interpretability scheme for interpreting the first model, and uses the first interpretability scheme to automatically and reasonably interpret the first model, thereby making the complex machine learning model white-boxed. After interpreting the first model using the first interpretability scheme, the model parameters of the first model can be adjusted and optimized based on the interpretation results to obtain a more credible and reasonable first model, thereby improving the transparency and credibility of the model. Attached Figure Description

[0044] Figure 1 A flowchart illustrating the method for interpreting the model according to an embodiment of this disclosure;

[0045] Figure 2 Another flowchart illustrating the method for interpreting the model of an embodiment of this disclosure is shown;

[0046] Figure 3 Another flowchart illustrating the method of interpreting the model of an embodiment of this disclosure is shown;

[0047] Figure 4 A schematic diagram illustrating the explanation method of the model in the embodiments of this disclosure is shown.

[0048] Figure 5 A schematic diagram of the structure of the explanatory apparatus for the model of the present disclosure embodiment. Detailed Implementation

[0049] Various embodiments and features of this disclosure are described herein with reference to the accompanying drawings.

[0050] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this disclosure will be apparent to those skilled in the art.

[0051] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

[0052] These and other features of this disclosure will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0053] It should also be understood that although this disclosure has been described with reference to specific examples, many other equivalent forms of this disclosure can be readily implemented by those skilled in the art.

[0054] The above and other aspects, features and advantages of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0055] Specific embodiments of this disclosure are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this disclosure, which may be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure this disclosure. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely to serve as the basis and representative basis for the claims to teach those skilled in the art to use this disclosure in a variety of substantially any suitable detailed structures.

[0056] Figure 1 A flowchart illustrating a method for interpreting a model according to an embodiment of this disclosure is shown. Figure 1 As shown, this disclosure provides a method for interpreting a model, including:

[0057] S101: Based on the first type of parameter features, the model to be explained is determined to be the first model.

[0058] After obtaining the model to be explained, a preliminary analysis is performed on the model to determine whether it contains first-class parameter features, so as to determine whether the model to be explained is the first model.

[0059] The model to be explained is a pre-trained model. The model to be explained can be any machine learning model, such as a simple model like a linear regression model or a decision tree, or a complex model like a neural network model (such as a CNN convolutional neural network model), or a combination of the above simple and complex models.

[0060] In this embodiment, the first model is preferably a deep learning model, which has a relatively complex model structure and computational logic. Deep learning models can include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), feedforward fully connected deep neural networks (DNNs), or generative adversarial networks (GANs). Furthermore, a deep learning model can also be a combination of multiple neural networks; for example, a deep learning model can simultaneously include CNNs and RNNs.

[0061] The first type of parameter features includes parameter features such as model complexity, model structure, or model format, which are used to distinguish the first model from other models.

[0062] Step S101 specifically includes the following steps:

[0063] S1011: Determine whether the model to be explained is a deep learning model based on at least one of the model complexity, model structure, or model format of the model to be explained;

[0064] S1012: If so, determine that the model to be explained is the first model.

[0065] Since deep learning models are generally complex and contain N layers, it is possible to determine whether the model to be explained is a deep learning model based on the model complexity and / or model structure. Deep learning models are generally stored in a specific data format, so it is also possible to determine whether the model to be explained is a deep learning model based on the model format.

[0066] S102: Obtain the input data corresponding to the first model.

[0067] Since the first model is a deep learning model, it cannot be effectively explained based solely on its own model structure features. Therefore, after obtaining the first model, the corresponding input data for the first model is obtained.

[0068] The data types of input data can include, but are not limited to, text, images, tables, charts, audio, and video. For example, when the first model is a model for image recognition, the input data can be images; when the first model is a model for natural language processing (NLP) such as text processing, semantic understanding, and machine translation, the input data can be either text or images containing text.

[0069] S103: Analyze the first model based on the input data to obtain a first interpretability scheme for interpreting the first model, wherein the interpretation parameters of the first interpretability scheme are determined at least based on the model attributes of the first model.

[0070] After obtaining the input data, the input data can be input into the first model. The first model can extract features from the input data, extract the input features used as input to the first model, and use the extracted input features and the corresponding operation logic to calculate the output result of the first model. The output result can be a prediction result, a recognition result, etc.

[0071] Understandably, the input data is the data fed into the model to be explained for prediction. The format of the input data is the same as that of the training data samples used during the training of the model, and the extraction methods for each input feature are also the same as those for the training data samples. For example, in the medical field, the input data can be various patient indicators, and the output can be predictions such as disease types or specific treatment recommendations corresponding to those indicators. For ship fuel consumption prediction, the input data can be ship voyage data related to fuel consumption, and the prediction result can be fuel consumption corresponding to different ship voyage data.

[0072] After obtaining the output results, the first model can be analyzed based on the output results, and a first interpretability scheme can be configured for the first model to explain the decision-making process from input data to output results.

[0073] Furthermore, when configuring a first interpretability scheme for a first model, the interpretation parameters of the corresponding first interpretability scheme are determined based on the model attributes of the first model, so that different first interpretability schemes can be configured for different first models, thereby improving the accuracy of the configuration of the first interpretability scheme.

[0074] In this embodiment, the interpretability parameters of the first interpretability scheme can be relevant parameters used to interpret the first model. For example, the interpretability parameters can be input features (and their specific feature parameters) in the input data that affect the decision results of the first model, that is, the first model is interpreted according to the importance of the features. For example, when it is determined that a patient has breast cancer, the interpretability parameter can be the input feature of age to explain the correlation between breast cancer and age. As another example, in the above-mentioned model for predicting fuel consumption, the interpretability parameters can be input features such as the ship's navigation conditions (direction, speed, distance, etc.) and climate conditions (wind direction, temperature, etc.) to explain how the above-mentioned input features affect fuel consumption.

[0075] S104: The first model is explained using the first interpretability scheme to obtain the explanation result.

[0076] After configuring the first interpretability scheme for interpreting the first model, the first model can be interpreted directly using the first interpretability scheme to obtain the interpretation result.

[0077] S105: Based on the explanation results, determine the explanatory relevance of the first model, wherein the explanatory relevance is used to characterize the explanatory effect of the explanation parameters on the model to be explained.

[0078] The explanatory effect of the explanatory parameters on the model to be explained refers to whether the explanatory parameters can provide a good explanation for the first model, so that the prediction results given by the first model have higher credibility.

[0079] If the explanatory relevance of the first model is high, it can be determined that the above explanatory parameters can effectively explain the first model, and the first model has high credibility and is reasonable and usable; if the explanatory relevance of the first model is low, it can be determined that the above explanatory parameters cannot effectively explain the first model, and the first model has low credibility. It is necessary to continue training the first model, or to adjust the model parameters (features that affect the decision results) of the first model and then train it to obtain a credible first model.

[0080] The explanatory parameters may contain all relevant features that affect the decision outcome (prediction result). For example, ship navigation data affecting fuel consumption includes over 100 influencing features such as navigation conditions (direction, speed, distance, etc.), climate conditions (wind direction, temperature, etc.), geographical information (latitude and longitude, ocean currents, etc.), ship conditions (cargo capacity, etc.), and the operating status of each engine (speed, temperature, pressure). Some features are not closely related to the result and may even become model noise, interfering with the model's prediction results. In particular, when the explanatory parameters contain too many features, there is a possibility of model overfitting. Therefore, after step S104, it is necessary to determine the explanatory relevance of the first model based on the explanatory results, and then adjust the first model to obtain a more reliable first model.

[0081] In practice, the explanatory relevance of the first model can be determined based on prior knowledge. For example, after the user obtains the explanation result through step S104, the business logic of the first model can be determined. For instance, based on the first interpretability scheme, it can be determined that the first model is a bank wealth management recommendation model, and the user's investment preferences are positively correlated with income, with higher income leading to a greater preference for riskier wealth management. Based on experience in bank wealth management recommendations, it is known that when a user's income is low, their investment preferences are conservative, and wealth management products such as government bonds can be recommended. Therefore, it can be determined that the explanatory relevance of the first model is high, and the first interpretability scheme can provide a good explanation for the first model. However, when it is determined based on the first interpretability scheme that the user's investment preferences are not correlated with income or are negatively correlated, based on empirical knowledge, it can be determined that the explanatory relevance of the first model is low, the first interpretability scheme fails to provide a credible explanation for the first model, and the rationality and credibility of the first model are low, requiring readjustment of the model parameters.

[0082] The model interpretation method provided in this disclosure determines the model to be interpreted as a first model based on the first type of parameter features. Then, it analyzes the first model based on the input data corresponding to the first model to obtain a first interpretability scheme for interpreting the first model. The first interpretability scheme is then used to automatically and reasonably interpret the first model, making the complex machine learning model white-boxed, improving the transparency of the model, and enabling users to understand in a timely manner which features in the model are more important and how the features affect the final result. After interpreting the first model using the first interpretability scheme, the model parameters of the first model can be adjusted and optimized based on the interpretation results to obtain a more credible and reasonable first model, thereby improving the credibility of the model.

[0083] In some embodiments, as Figure 2 As shown, the model attributes of the first model include the model task of the first model. Step S103 analyzes the first model based on the input data to obtain a first interpretability scheme for interpreting the first model, including:

[0084] S1031: Process the input data according to the model task of the first model;

[0085] S1032: Input the processed input data into the first model to obtain the first output result of the first model;

[0086] S1033: Determine the first interpretation parameter of the first model based on the first output result, wherein the first interpretation parameter includes the importance of the input features in the input data to the first model.

[0087] In one specific embodiment, the model task of the first model includes a binary classification task, and the input data contains a first target object to be identified, the first target object having a classification category; steps S1031 to S1033 specifically include the following steps:

[0088] S10311: Divide the input data into multiple data units, each of the data units containing at least one data feature;

[0089] S10312: A set of associated data of the input data is obtained by randomly hiding different data units, wherein the hidden data units in each set of associated data are different;

[0090] S10321: Input each associated data into the first model to obtain the classification and recognition result of each associated data, wherein the classification and recognition result represents the probability of recognizing the target object in the input data;

[0091] S10331: Based on the classification and recognition results, determine the first interpretation parameters of the first model.

[0092] In this context, a binary classification task means that the first model's classification task has two categories, such as identifying whether the fruit in an image is an apple. That is, the first model contains a classifier that, when the input data is an image, outputs a classification based on whether it is an apple, represented by 0 or 1. An output of 1 indicates that the fruit in the image is an apple, and an output of 0 indicates that the fruit in the image is not an apple. In other words, a binary classification task assumes that each input data point is assigned one and only one classification label, 0 or 1. When the first model's task includes a binary classification task, the first model's prediction result is a single prediction result, meaning the aforementioned target object has only one classification category.

[0093] In this embodiment, the first model is a trained binary classification image classification model, and the input data is the first image to be identified. Taking the identification of the first target object (e.g., an apple) in the first image as an example, the above steps S10311-S10331 will be explained in detail.

[0094] In specific implementation, the first image is first divided into several image units. For example, it can be divided into 10 equal parts horizontally and 10 equal parts vertically, resulting in 100 image units; or it can be divided into 100 consecutive superpixels using an algorithm. A superpixel refers to an interconnected pixel with similar color, texture, brightness, and other characteristics. Then, in step S1032, some image units are randomly hidden, and these randomly hidden image units are marked, for example, by changing their color to gray, resulting in a set of associated images (e.g., N images). The first associated image can be the image after hiding the first image unit, the second associated image can be the image after hiding the second image unit, and so on. Each randomly hidden associated image is input into the first model to obtain the classification and recognition result for each associated image. For each associated image, the first model determines whether it contains an apple with a certain probability. For example, the probability of recognizing an apple in the first associated image is 70%, the probability of recognizing an apple in the second associated image is 50%, ... the probability of recognizing an apple in the fifth associated image is 30%. Then, the classification results are compared with the prediction results of the unoccluded first image. For example, the probability of recognizing an apple in the first image is 80%, while the probability of recognizing an apple in the first associated image is 70%, showing a significant change in probability. Therefore, it can be determined that the features in the occluded image units in the first associated image can affect the prediction results. Simultaneously, the location of features can be determined based on the location of the occluded image units. By comparing the classification results of other associated images with the recognition results of the first image in a similar manner, it is possible to determine which features (superpixels) the user can effectively use to recognize the apple in the first image, thus obtaining the explanatory parameter of the feature importance of the first model.

[0095] It is understood that the hidden data units in each piece of linked data in the aforementioned set of linked data can be different. This includes situations where the hidden data units in each piece of linked data in the same set are completely different, for example, the first linked image is the image after hiding the first image unit, and the second linked image is the image after hiding the second image unit. Alternatively, it can include situations where the hidden data units in each piece of linked data in the same set are the same, for example, the first linked image is the image after hiding the first image unit, and the second linked image is the image after hiding the first and second image units. Each linked image can contain multiple hidden image units; for example, the first linked image is the image after hiding the first and second image units, and the second linked image is the image after hiding the third to fifth image units.

[0096] Steps S10311-S10331 employ a Locally Understandable, Model-Independent Interpretation (LIME) method to locally modify the input of the first model by occluding different data units on the image, perturbing the input data to obtain a perturbed dataset. The changes in the confidence (prediction probability) of the model's predictions after each data unit is occluded are recorded to determine the regions that significantly influence the prediction results (i.e., identify the input features with significant influence). If the confidence of the first model's predictions drops significantly after one or more data units are occluded, it indicates that the data unit is relatively important.

[0097] In particular, by processing the input data through steps S10311 and S10312, different data features can be displayed in different associated images, making it easier to identify input features with significant influence.

[0098] Furthermore, the method also includes:

[0099] S201: Compare each of the associated data with the input data to obtain the similarity between the associated data and the input data, wherein the similarity is determined based on the data units hidden in the associated data;

[0100] S202: Based on the similarity and the classification results, the importance of the data feature corresponding to the data unit in each of the associated data is obtained through a linear regression model;

[0101] S203: Determine at least one of the data features with the highest importance ranking as the interpretation parameter of the first model.

[0102] Specifically, each associated image is labeled, and its similarity to the input first image is calculated. For example, hidden image units in each associated image are labeled as 1, and unhidden image units are labeled as 0. When all image units in a single associated image are hidden (all turn gray), the similarity between the associated image and the first image can be determined to be 0. When all image units in a single associated image are not hidden, the similarity between the associated image and the first image can be determined to be 1. Similarly, the similarity with the first image can be determined based on the number of hidden image units (the number of units labeled as 1).

[0103] Based on the similarity with the first image, a weight is assigned to each associated image; the higher the similarity, the higher the weight; conversely, the lower the similarity, the lower the weight. These associated images are then input into a first model for classification, yielding classification results for each image. These results serve as label data for each associated image, and a training dataset is constructed using each associated image and its corresponding label data. In the training dataset, each associated image corresponds to an image unit (feature), a weight, and a classification category. A simple linear regression model is trained based on this locally weighted training dataset. In this model, each image unit corresponds to a coefficient (importance); a larger coefficient (highest positive weight) indicates a more important image unit. The feature corresponding to at least one image unit with the highest coefficient is then determined as an explanatory parameter of the first model. This explanatory parameter represents the input feature (feature importance) that significantly influences the prediction result.

[0104] Based on steps S10311-S10331, steps S201 to S205 establish a simple data representation (linear regression model) based on the similarity between the perturbation data and the input data in the perturbation dataset, as well as the prediction results of the perturbation data. They then observe which features (superpixels) have larger coefficients, thereby clarifying which part of the input features has a significant impact on the prediction results.

[0105] In this embodiment, since the output of the first model is a single prediction, the limitation of the LIME interpretation method itself in local interpretation can be overcome, without considering the influence of feature interactions. The LIME interpretation method can be applied to various complex deep learning models such as XGBoost and Support Vector Machines (SVM). Because the LIME interpretation method is inherently model-independent, it has broad applicability.

[0106] Besides interpreting image classification results, LIME interpretation methods can also be applied to natural language processing tasks such as text recognition and part-of-speech tagging. For example, LIME interpretation methods can be used to determine which keyword(s) led a message to be classified as spam.

[0107] In other embodiments, the model task of the first model includes a multi-classification task, and the input data contains a second target object to be identified, the second target object having at least two classification categories; steps S1031 to S1033 specifically include the following steps:

[0108] S10322: Input the input data into the first model to predict at least two classification categories of the second target object;

[0109] S10323: Obtain the predicted probability corresponding to each of the predicted classification categories;

[0110] S10332: The data features corresponding to at least two classification categories that meet the preset probability threshold or are determined based on the magnitude of the predicted probability are determined as the first interpretation parameter.

[0111] The term "multi-class classification task" indicates that the classification task of the first model includes multiple classification categories, meaning the input data may contain multiple classification labels. Taking the classification and recognition of target objects in an image as an example, steps S10322-S10332 will be explained in detail. For instance, the classification category of the second target object in the second image input into the first model might be more than ten building types, such as palaces or churches. Inputting the second image into the first model yields prediction results for at least two classification categories and their corresponding prediction probabilities. The category that meets the preset probability threshold or has the highest prediction probability is selected as the final classification category of the first model. The image features (input features) corresponding to the category with the highest prediction probability are then determined as the first interpretation parameter of the first model.

[0112] In this embodiment, since the model task of the first model is a multi-class classification task, the input data can be directly input into the first model without processing it in step S103. The input data is an image. In some embodiments, the first model can be a text classification model, and the text can be an article composed of words. The input features can be obtained by using natural language recognition technology to obtain the feature vectors of each word in the text. The determination of the first interpretation parameter of the first model is similar to that of the image classification described above, and will not be elaborated here.

[0113] In some embodiments, as Figure 3 As shown, the method further includes:

[0114] S1034: Adjust the input features in the input data according to the model task of the first model, and input the adjusted input features into the first model to obtain the second output result;

[0115] S1035: Determine the second interpretation parameters of the first model based on the second output result, wherein the second interpretation parameters include the way in which the input features affect the output result of the first model.

[0116] Steps S1034 to S1035 can determine the specific way in which the input features affect the output of the first model through sensitivity analysis, that is, how the prediction result changes when a certain input feature changes.

[0117] Specifically, when the first model's task is text classification (NLP), it can determine the impact of word changes on the overall semantics of the text by adding, deleting, replacing, and altering the word order. For example, in sentiment classification, which includes positive and negative sentiments, "I don't like chocolate" and "I like chocolate" express completely opposite meanings. Therefore, it can be determined that "not" has a significant impact on the prediction results, and thus, how this word affects the prediction results can be determined.

[0118] In another embodiment, the first model's task is image classification. Changing the feature parameters of one or more input features can alter the display effect of the image region corresponding to that input feature. For example, changing the brightness, color, grayscale, or blurring of the image region corresponding to a certain input feature. The processed image is then input into the first model to obtain a second prediction result. If the prediction results before and after processing are similar (having the same classification result), it can be determined that the change in the input feature does not affect the final prediction result, and the performance of the first model and the prediction result are stable. This influence method can also be used to determine the feature importance in S1031 to S1033.

[0119] In another embodiment, the model task of the first model is a regression task, which changes the values ​​of certain input features in the input data to determine the change in the final expected result. For example, the input feature of the input data is the attribute of house area. After determining the impact of the house area attribute on house price through the regression model (for example, by displaying the house area-price curve in the form of a chart, with the house area on the horizontal axis and the house price on the vertical axis), the changes in the house area-price curve are judged by continuously changing the size of the house area. In this way, it can be determined how the house area affects the price. For example, when the house area is small, its price increases with the increase of area, and the unit price per square meter may be higher; while when the house area is large, due to the long sales cycle, its unit price per square meter will decrease.

[0120] In this embodiment, steps S1034 to S1035 can be executed based on steps S1031 to S1033, or they can be executed separately. The specific execution method is not specifically limited in this disclosure.

[0121] In some embodiments, the model task of the first model may also include regression, clustering, or dimensionality reduction tasks. For example, when the model task is logistic regression, since its input features are usually discrete, after determining the importance of each feature through steps S1031 to S1033, similar features can be clustered to obtain the importance of feature groups, and the importance of feature groups can be used as the first explanatory parameter. For a logistic regression model, to calculate the importance of an input feature, this feature can be broken down into all input data, and then the area under the curve (AUC) can be calculated based on the model. The difference between the calculated AUC and the AUC of the original model can be used as the importance of this feature.

[0122] In other embodiments, the model attributes of the first model used to determine the first interpretability scheme may also include the model size of the first model. Since the size of a deep learning model is determined by the network (the number of feature parameters and the storage format of the feature parameters), it can be determined that the model contains a large number of network structures. When determining the first interpretability scheme, the first model can be compressed in advance to facilitate the analysis of the first model.

[0123] In some embodiments, the interpretation parameters of the first interpretability scheme can also be determined based on the data attributes of the input data, wherein the data attributes include data type, data size, and data label distribution, etc.

[0124] The data types can be text, images, tables, charts, audio, and video, as mentioned above. The data size can be the amount of input data. The data label distribution can be data containing special labels. For example, when the model task of the first model is classification, determining the data label distribution of the input data can help determine whether the input data is balanced. If it is unbalanced, it may affect the prediction results. Therefore, more input data can be obtained to balance the input data, and the balanced and unbalanced data can be input into the first model respectively to analyze the first model and determine the explanatory parameters of the first model. When the model task of the first model is classification or regression, determining the data label distribution of the input data can help determine whether there are extreme samples in the input data. If so, the extreme samples can be removed from the input data. Alternatively, it can be analyzed why there are extreme samples in the obtained input data, and the extreme samples can be input into the first model to obtain prediction results. Based on the prediction results, the impact of the extreme samples on the first model can be determined, and thus the explanatory parameters of the first model can be determined.

[0125] The data attributes of the input data can be used in conjunction with the model attributes of the first model to determine the first and second explanatory parameters, or they can be used alone to determine the first, second, and other explanatory parameters.

[0126] In some embodiments, the first interpretability scheme further includes an interpretation method for the first model, and the method further includes:

[0127] S1036: Based on the model attributes of the first model and / or the data attributes of the input data, determine the interpretation method of the first model, wherein the data attributes of the input data include data type and / or data size.

[0128] Specifically, the first model can be interpreted using the visualization method of the aforementioned interpretation parameters.

[0129] For example, after determining the first explanatory parameter of the binary classification model, the colors of other parts in the first image can be marked as gray to highlight the first explanatory parameter, so as to intuitively show the user the input features in the input data that have the most significant impact on the prediction result.

[0130] For example, in the process of determining the first interpretation parameter of the above multi-classification model, after obtaining the predicted probabilities corresponding to each of the predicted classification categories through step S10323, the interpretation method of the first model can be determined by the following steps:

[0131] S301: Based on the determined classification category of the second target object, obtain the heat map corresponding to the classification category through the CAM algorithm;

[0132] S302: Overlay the heat map with the input data and visualize it to determine the data features corresponding to the classification category.

[0133] Specifically, the first model can be a convolutional neural network (CNN). For the prediction results of each classification category of the second target object, the CAM (Class Activation Mapping) algorithm can be used to visualize and explain the first model, and the regions (features) that the CNN focuses on when making predictions can be explained visually.

[0134] For a classification category c, firstly, the result of the last convolutional layer of the neural network is extracted. The feature map of the last convolutional layer contains K channels (which can be simply viewed as K M×N matrices or images). These K channels can be considered to extract different features, and each channel contributes differently to the final prediction result. CAM uses Global Average Pooling (GAP) to spatially pool the feature map and performs a linear transformation to obtain the weights for each channel. The weights for each channel are w1 and w2, respectively. c ,...,w k cThen, the channels of each feature map are linearly weighted and summed (the weights are multiplied by the corresponding channels and then summed) and fused together to obtain the result as shown below. Figure 4 The heatmap (classification and localization map) shown in (b) illustrates this. After obtaining the heatmap, it can be overlaid on the second image by upsampling to the size of the original input image or by interpolation. This allows us to determine which region in the second image primarily determines the current classification based on the color depth in the heatmap, providing a visual explanation of the first model's decision-making process. In the heatmap, colors closer to red indicate a greater influence of the data features in that region on the prediction result, while colors closer to blue indicate a smaller influence.

[0135] The main purpose of upsampling, or image interpolation, is to enlarge an image so that it can be displayed on a higher resolution display device.

[0136] In a preferred embodiment, the Grad-weighted Class Activation Mapping (Grad-CAM) algorithm can be used to assign importance values ​​to each neuron using the gradient information flowing into the last convolutional layer of the CNN, thereby determining the interpretation parameters. The difference between Grad-CAM and CAM lies in the different approach to obtaining the weights for each channel.

[0137] Grad-CAM uses the gradients from the backpropagation of the network to calculate the weights of each channel in the feature map, thus obtaining a heatmap, which is equivalent to the global average pooling operation in CAM.

[0138] In Grad-CAM, the formula for calculating the weight of each channel is as follows:

[0139]

[0140] Where c represents the category, y c is the probability logits corresponding to this category (i.e., the value before the Softmax activation function), A represents the feature map output by the convolution, k represents the channel of the feature map, i and j represent the horizontal and vertical coordinates of the feature map respectively, and Z represents the size of the feature map (i.e., length × width).

[0141] Furthermore, Grad-CAM applies the ReLU function to the fused heatmap, retaining image regions (superpixels) that positively impact classification while suppressing regions that negatively affect classification. This yields the heatmap corresponding to classification category c.

[0142]

[0143] in, This indicates linear weighting, which means calculating the mean of the gradients on the feature map.

[0144] In CNN networks, the final convolutional layer captures the most specific features of the image (or other input). Therefore, Grad-CAM uses gradient information flowing to the last convolutional layer in the CNN network to understand the importance of each feature for classification. Grad-CAM can be directly applied to various CNN networks (such as VGG and ResNet) without modifying the network structure in the first model or retraining the first model, making it more widely applicable than CAM.

[0145] Furthermore, after determining the first explanatory parameters through step S10332, for example, after determining the five categories with the highest predicted probabilities as the first explanatory parameters, the heatmaps of the fused second images corresponding to the above five categories obtained through steps S301 and S302 can be directly displayed to intuitively show the input features that have the most significant impact on each prediction result and information such as positive and negative correlations, thereby explaining the first model.

[0146] For image-based input data, we can label the input features that are of high importance within the image, for example, such as... Figure 4 As shown in (a), the input feature is outlined with a dashed line; alternatively, it can be defined as follows: Figure 4 The heatmap shown in (b) illustrates each input feature, and the input features with feature importance are determined based on the color depth in the heatmap, visually representing the first interpretation parameters of the first model. For text-based input data used in natural language processing, word clouds or bar charts can be used to display the importance of each word or phrase. For example, in... Figure 4 In the word cloud shown in (c), the larger the font size of a word, the more important the word is; in... Figure 4 In the bar chart shown in (d), the higher the vertical axis of a bar, the more important the input feature corresponding to that bar. For tabular input data, tables or charts (such as bar charts) can be used to display the importance of each attribute in the table. For image-based input data, tables can be used to display the importance of features.

[0147] In other embodiments, the display method of the interpretation parameters can also be determined based on the size of the input data. For example, when there are many input features in the input data, using a word cloud approach results in significant overlap and overlap of input features within a limited display space, making it difficult to clearly determine the features with feature importance. In this case, a different approach can be used. Figure 4 The table shown in (e) displays a list of input features, along with the feature importance weights and corresponding rankings for each input feature.

[0148] By determining the interpretation parameters and interpretation methods of the first model as described above, a first interpretability scheme for interpreting the first model can be automatically generated.

[0149] In this embodiment, the first interpretability scheme can be a specific interpretable model generated based on the input data and the model attributes of the first model. By inputting the first model and the corresponding input data into the interpretable model, the interpretation result of the first model can be obtained.

[0150] In some embodiments, step S105, based on the explanation results, determines the explanatory relevance of the first model, including:

[0151] S1051: Determine the contribution of the explanation parameters to the output of the first model;

[0152] S1052: Determine the explanatory relevance of the first model based on the contribution and the preset input features of the first model;

[0153] S1053: Adjust the input features of the first model based on the explained relevance.

[0154] Specifically, the explanatory parameters include the first explanatory parameters, which are the input features that have a significant impact on the prediction results of the first model, and the contribution rate represents the contribution of each input feature to the model's prediction results. Therefore, the importance of each input feature can be determined by calculating its contribution. This importance can then be compared with the preset input features of the first model to determine whether the input features determined in the first interpretability scheme match the preset input features of the first model. For example, when a bank makes a loan, the preset input features of the credit model are information such as age, gender, income, and personal credit history, with income information having the highest importance. If the interpretability parameters in the first interpretability scheme are also the above information, and income information has the highest importance, then the preset input features of the first model can be determined to be reasonable, and the interpretability of the credit model is high, requiring no adjustment to the first model. However, when the input features in the first interpretability scheme contain unknown features, such as flight mileage or travel duration, which are not included in the preset input features, the user can adjust the input features of the first model by adding these previously unnoticed features, training the first model, and continuously adjusting the model parameters based on the prediction performance of the first model after adding these features to obtain a more reliable first model.

[0155] In some embodiments, the method further comprises:

[0156] S401: Based on the second type of parameter features, the model to be explained is determined to be the second model;

[0157] S402: Analyze the second model to obtain a second explanatory scheme for explaining the second model.

[0158] The second type of parameter features can be the same as or different from the first type of parameter features. When the second type of parameter features are the same as the first type of parameter features, the second model can be any model other than the first model. When the second type of parameter features are different from the first type of parameter features, the second model can be a model directly determined based on the second type of parameter features, such as a linear regression model or a decision tree model.

[0159] In some embodiments, step S402 specifically includes:

[0160] Determine whether the second model includes feature engineering;

[0161] If included, the second interpretability scheme is configured based on the data cleaning, feature extraction, and feature selection methods in the feature engineering; and / or

[0162] The second interpretability scheme is configured based on the task category processed by the second model, wherein the task category includes at least one of regression, classification, clustering, or dimensionality reduction; and / or

[0163] The second interpretability scheme is configured based on whether there is input data suitable for the second model and whether the input data needs to be sampled.

[0164] When determining that the second model includes feature engineering, the input features of the second model can be obtained through analysis, and the impact of each input feature on the model can be determined by calculating the feature importance. Specifically, the importance of each feature can be calculated using the permutation test method; or the importance of each input feature can be determined by calculating the SHAP value to interpret the second model. The SHAP value can also explain the interactions between the input features.

[0165] Optionally, when the task category handled by the second model is determined to be regression, classification, clustering, or dimensionality reduction, a corresponding second interpretability scheme can be configured according to the task category. For example, when the second model is a linear regression model, a second interpretability scheme can be configured by exploring the relationship between the independent and dependent variables in the second model.

[0166] Optionally, when it is determined that there is input data suitable for the second model, the input data can be input into the second model to obtain prediction results. Based on the prediction results, the second model can be analyzed to calculate the importance of each feature in the input data and how each feature affects the prediction results, thus obtaining a second interpretability scheme. After determining that there is input data, the interpretation parameters of the second interpretability scheme can be determined based on data attributes such as the data type, data size, and data label distribution of the input data, similar to the configuration of the first interpretability scheme. After interpreting the second model using the second interpretability scheme, the interpretation results are output and displayed.

[0167] In some embodiments, after determining the type of the model to be explained, the method further includes:

[0168] Based on the type of the model to be explained, it is determined whether there is a third interpretability scheme applicable to the model to be explained, wherein the third interpretability scheme is predetermined.

[0169] This involves determining whether the model to be explained is suitable for a specific interpretable model, in order to quickly match the corresponding interpretability solution to the model and improve the configuration efficiency of interpretability solutions. For example, once it is determined that the model to be explained is a decision tree model, it can be directly explained according to the preset decision tree model interpretation solution. Since the relationships between the nodes in the decision tree model are clear, a third interpretability solution can be pre-configured for the decision tree model based on the relationships between the nodes.

[0170] The model interpretation method provided in this disclosure can automatically identify the type of the model to be interpreted, automatically generate corresponding interpretability schemes for different models to be interpreted, and use the corresponding interpretability schemes to interpret the model. It can interpret not only simple models such as traditional decision trees, but also complex models such as deep learning models. Users do not need to know the business background, model information, or related content, nor do they need to know artificial intelligence knowledge or model interpretability related knowledge.

[0171] For business users without a machine learning background, the above model interpretation methods can help them understand the model's operating mechanism. Business users can compare the model interpretations with their own business experience to determine if the model is reasonable and reflects actual business needs; or they can use the model interpretations to gain insights into their business experience. For example, business users can check the importance of each feature in the calculated model to see if important factors in their business experience have a high impact on the model, and if unimportant factors have a low impact; if it matches experience, the model can be considered reasonable and effective. Furthermore, based on the model interpretations, they can identify previously overlooked features, whether these features have a significant impact on the model, whether this impact can be explained from a business logic perspective, or whether they can provide insights for business users. For users with some machine learning knowledge, the above model interpretation methods can help them understand the model more deeply, better select data cleaning methods, choose feature engineering methods, and adjust model structure or parameters. For example, in a newly constructed model, unimportant features can be removed based on the model's interpretability.

[0172] Figure 5 A schematic diagram illustrating the structure of an explanatory apparatus for a model according to an embodiment of this disclosure is shown. Figure 5 As shown, this disclosure also provides a model interpretation apparatus, including:

[0173] The first determining module 10 is configured to determine the model to be explained as the first model based on the first type of parameter features;

[0174] The acquisition module 20 is configured to acquire the input data corresponding to the first model;

[0175] Analysis module 30 is configured to analyze the first model based on the input data to obtain a first interpretability scheme for interpreting the first model, wherein the interpretation parameters of the first interpretability scheme are determined at least based on the model attributes of the first model;

[0176] Explanation module 40 is configured to use the first interpretability scheme to explain the first model and obtain an explanation result;

[0177] The second determining module 50 is configured to determine the explanatory relevance of the first model based on the explanation results, wherein the explanatory relevance is used to characterize the explanatory effect of the explanation parameters on the first model.

[0178] In some embodiments, the model attributes of the first model include the model task of the first model, and the analysis module 30 is specifically configured as follows:

[0179] The input data is processed according to the model task of the first model;

[0180] The processed input data is input into the first model to obtain the first output result of the first model;

[0181] The first explanatory parameter of the first model is determined based on the first output result. The first explanatory parameter includes the importance of the input features in the input data to the first model.

[0182] In some embodiments, the analysis module 30 is further configured to:

[0183] The input features in the input data are adjusted according to the model task of the first model, and the adjusted input features are input into the first model to obtain the second output result;

[0184] The second explanatory parameters of the first model are determined based on the second output result, wherein the second explanatory parameters include the way in which the input features affect the output result of the first model.

[0185] In some embodiments, the analysis module 30 is further configured to:

[0186] Based on the model attributes of the first model and / or the data attributes of the input data, the interpretation method of the first model is determined, wherein the data attributes of the input data include data type and / or data size.

[0187] In some embodiments, the second determining module 50 is specifically configured as follows:

[0188] Determine the contribution of the explanation parameters to the output of the first model;

[0189] The explanatory relevance of the first model is determined based on the contribution and the preset input features of the first model;

[0190] The input features of the first model are adjusted based on the explained relevance.

[0191] In some embodiments, the first determining module 10 is specifically configured as follows:

[0192] Whether the model to be explained is a deep learning model is determined based on at least one of the model complexity, model structure, or model format of the model to be explained;

[0193] If so, the model to be explained is determined to be the first model.

[0194] In some embodiments, the first determining module 10 is further configured to:

[0195] Based on the second type of parameter features, the model to be explained is determined to be the second model;

[0196] Analysis module 30 is also configured as follows:

[0197] The second model is analyzed to obtain a second explanatory scheme for explaining the second model.

[0198] In some embodiments, the analysis module 30 is further configured to:

[0199] Determine whether the second model includes feature engineering;

[0200] If included, the second interpretability scheme is configured based on the data cleaning, feature extraction, and feature selection methods in the feature engineering; and / or

[0201] The second interpretability scheme is configured based on the task category processed by the second model, wherein the task category includes at least one of regression, classification, clustering, or dimensionality reduction; and / or

[0202] The second interpretability scheme is configured based on whether there is input data suitable for the second model and whether the input data needs to be sampled.

[0203] In some embodiments, the model interpretation apparatus further includes a third determining module configured to: determine, based on the type of the model to be interpreted, whether there exists a third interpretability scheme applicable to the model to be interpreted, wherein the third interpretability scheme is predetermined.

[0204] The model interpretation apparatus provided in this disclosure corresponds to the model interpretation method in the above embodiments. Based on the above model interpretation method, those skilled in the art can understand the specific implementation and various variations of the model interpretation apparatus in this disclosure. Any option in the model interpretation method embodiments is also applicable to the model interpretation apparatus, and will not be repeated here.

[0205] This disclosure also provides an electronic device, including a processor and a memory, wherein the memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions to implement the above-described model interpretation method.

[0206] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0207] The memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0208] This disclosure also provides a computer-readable storage medium storing computer-executable instructions thereon, which, when executed by a processor, implement the above-described method for interpreting the model.

[0209] The above embodiments are merely exemplary embodiments of this disclosure and are not intended to limit this disclosure. The scope of protection of this disclosure is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this disclosure within its substance and scope, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this disclosure.

Claims

1. A method for interpreting a model, comprising: Based on the first type of parameter features, the model to be explained is determined to be the first model, which is a classification model used to classify images or text; Obtain the input data corresponding to the first model; The first model is analyzed based on the input data to obtain a first interpretability scheme for interpreting the first model, wherein the interpretation parameters of the first interpretability scheme are determined at least based on the model attributes of the first model. The first model is explained using the first interpretability scheme to obtain the explanation result; Based on the explanation results, the explanatory relevance of the first model is determined, wherein the explanatory relevance is used to characterize the explanatory effect of the explanation parameters on the first model.

2. The method according to claim 1, wherein, The model attributes of the first model include the model task of the first model. The step of analyzing the first model based on the input data to obtain a first interpretability scheme for interpreting the first model includes: The input data is processed according to the model task of the first model; The processed input data is input into the first model to obtain the first output result of the first model; The first explanatory parameter of the first model is determined based on the first output result. The first explanatory parameter includes the importance of the input features in the input data to the first model.

3. The method according to claim 2, wherein, The method further includes: The input features in the input data are adjusted according to the model task of the first model, and the adjusted input features are input into the first model to obtain the second output result; The second explanatory parameters of the first model are determined based on the second output result, wherein the second explanatory parameters include the way in which the input features affect the output result of the first model.

4. The method according to claim 1, wherein, The first interpretability scheme also includes an interpretation method for the first model, and the method further includes: Based on the model attributes of the first model and / or the data attributes of the input data, the interpretation method of the first model is determined, wherein the data attributes of the input data include data type and / or data size.

5. The method according to claim 1, wherein, Determining the explanatory relevance of the first model based on the explanation results includes: Determine the contribution of the explanation parameters to the output of the first model; The explanatory relevance of the first model is determined based on the contribution and the preset input features of the first model; The input features of the first model are adjusted based on the explained relevance.

6. The method according to claim 1, wherein, The step of determining the model to be explained as the first model based on the first type of parameter features includes: Whether the model to be explained is a deep learning model is determined based on at least one of the model complexity, model structure, or model format of the model to be explained; If so, the model to be explained is determined to be the first model.

7. The method according to claim 1, wherein, The method further includes: Based on the second type of parameter features, the model to be explained is determined to be the second model; The second model is analyzed to obtain a second explanatory scheme for explaining the second model.

8. The method according to claim 7, wherein, The analysis of the second model yields a second interpretability scheme for explaining the second model, including: Determine whether the second model includes feature engineering; If included, the second interpretability scheme is configured based on the data cleaning, feature extraction, and feature selection methods in the feature engineering; and / or The second interpretability scheme is configured based on the task category processed by the second model, wherein the task category includes at least one of regression, classification, clustering, or dimensionality reduction; and / or The second interpretability scheme is configured based on whether there is input data suitable for the second model and whether the input data needs to be sampled.

9. The method according to claim 1 or 7, wherein, After determining the type of the model to be explained, the method further includes: Based on the type of the model to be explained, it is determined whether there is a third interpretability scheme applicable to the model to be explained, wherein the third interpretability scheme is predetermined.

10. A model interpretation apparatus, comprising: The first determining module is configured to determine the model to be explained as the first model based on the first type of parameter features, wherein the first model is a classification model used for classifying images or text; The acquisition module is configured to acquire the input data corresponding to the first model; The analysis module is configured to analyze the first model based on the input data to obtain a first interpretability scheme for interpreting the first model, wherein the interpretation parameters of the first interpretability scheme are determined at least based on the model attributes of the first model. The explanation module is configured to explain the first model using the first interpretability scheme and obtain the explanation result; The second determining module is configured to determine the explanatory relevance of the first model based on the explanation results, wherein the explanatory relevance is used to characterize the explanatory effect of the explanation parameters on the first model.

Images