Method for computer-implemented analysis of classification models

Abstract

The present invention describes a method for computer-implemented analysis of a classification model adapted to map multiple input instances, each of which has a number n of features, into multiple probabilities of output classes as predictions, according to a predetermined function, as classification decisions, and adapted to determine a relevance value for each feature, which results in a saliency map. The present invention comprises the steps of identifying (S1) the influence of each feature on the prediction of the instance by determining relevance information for each feature, the relevance information representing all features of the instance, omitting contextual information of the considered feature. Then, the relevance value for each feature is determined (S2) by combining the relevance information for the features of the instance. Finally, the multiple relevance values ​​for the instance features are evaluated (S3) to identify the influence of each feature on the instance prediction.

Description

Technical Field

[0001] The present invention relates to a method for computer-implemented analysis of a classification model adapted to map, as a prediction, a plurality of input instances, each of which has a plurality of features, into a plurality of probabilities of an output class according to a predetermined function as a classification decision. Furthermore, the classification model is adapted to determine a relevance value for each feature, which results in a saliency map. Background Art

[0002] One of the weaknesses of machine learning models is the lack of explainability. Specifically, it is difficult to explain intelligent decisions to end users in practice. Models are often applied as black boxes, especially deep neural networks. Deep models achieve many tasks with human-level performance, such as image classification and detection. These neural networks with deep architectures are difficult to understand even for machine learning experts. However, intelligent decisions need to be explained to gain user trust.

[0003] In recent years, many model-aware saliency methods have been proposed to understand the relationship between the input and output of deep neural networks. Saliency methods are algorithms that identify features that are relevant to individual classification decisions by creating so-called saliency maps. However, existing model-aware saliency methods are not applicable to models with complex structures. In addition, saliency maps are unreliable because some of them are independent of both the model and the data generation process.

[0004] Model-agnostic methods appear to be alternative analytical tools compared to model-aware methods, as they can operate without access to their parameters, architecture, and the gradients of their outputs with respect to their inputs. They are able to explain any classifier, their explanations from two or more different types of models are comparable, and the so-called ensemble model can also be explained without knowledge of the model components.

[0005] Among others, LIME (Locally Interpretable Model-Agnostic Explanations) and PDA (Predictive Difference Analysis) in particular seem to be the most promising methods for analyzing black-box classification models.

[0006] LIME trains a local interpretable classifier to approximate the behavior of the original complex model on a single instance. The classification of the instance is explained using the trained interpretable classifier. Different interpretable classifiers give different explanations, and the optimization of the same type of classifiers can end with different parameters. Therefore, the explanations generated are inconsistent. In addition, for each classification, LIME is required to train a new classifier, which is not efficient.

[0007] The occlusion method identifies the relevance of each feature by occluding the feature with a predefined value. Occlusion with the predefined value cannot remove the depth evidence of the occluded feature without introducing unintended features.

[0008] To overcome these shortcomings, PDA, which is a probabilistic method, is proposed to analyze individual classification decisions. PDA observes the change in output when parts of the features of an instance are removed. If the output score decreases, the removed features are relevant, otherwise they are negative evidence. The removal of influence is achieved by replacing them with different values ​​and taking the average of their output scores. However, for each feature, it requires many classifications. It is not feasible when the dimensionality of the input is very high, such as when analyzing images. In addition, this method does not work well when many features of the classified instance support the classification. Summary of the invention

[0009] It is therefore an object of the present invention to provide an improved method for computer-implemented analysis of classification models.It is a still further object of the present invention to provide a processing unit adapted to provide an improved computer-implemented analysis of classification models.

[0010] These objects are solved by a method according to the present disclosure and a processing unit according to the present disclosure.Preferred embodiments are set out in the present disclosure.

[0011] The present invention proposes a method for computer-implemented analysis of a classification model. The classification model is adapted to map multiple input instances, each of which has multiple features, to multiple probabilities of output classes as predictions, according to a predetermined function, as classification decisions. In addition, the classification model is adapted to determine a relevance value for each feature, which leads to a saliency map. The method comprises the steps of identifying the influence of each feature on the instance prediction by determining relevance information for each feature, the relevance information representing contextual information for all features of the instance, omitting the considered feature (i.e., except for the considered feature). The method comprises the steps of determining a relevance value for each feature by combining the relevance information for the instance features. In addition, the method comprises the steps of evaluating the multiple relevance values ​​for the instance features to identify the influence of each feature on the instance prediction.

[0012] By considering the classification of contextual features, it is possible to efficiently identify the relevance of features without modeling or sampling the process. Therefore, an efficient, consistent and model-agnostic explanation method is provided. Specifically, the proposed method is able to explain saturated classifications, where the output of the assessed classification model is close to 1. The proposed method is efficient. It takes seconds to generate explanations for high-dimensional inputs, such as image classification. In addition, the relevance of contextual features rather than individual features is identified. Given an input instance, the provided method only requires the output of the model. Therefore, for black box classifier models, explanations can be generated simply by using their inputs and outputs. Such black box models can be complex models that are searched through AutoML or inaccessible remote models, for example, as an API provided by Google.

[0013] According to a preferred embodiment, the relevance value for each feature is calculated by the sum of the ratio of the relevance information and the absolute value of the relevance information, wherein the feature is the omitted considered feature.

[0014] According to a further preferred embodiment, the correlation value for each feature is equal to or greater than zero.

[0015] According to another preferred embodiment, the correlation information is determined by calculating

[0016]

[0017] where v k Is x i is the sampled value of the condition, x i is feature i of instance x, M is the number of samples of the class under consideration, and y is the output class. In this formula, p(y|x i , v k ) means that given x i and context feature v k Output probability of class y when y=0. Default is f(x)p(y|x).

[0018] According to another preferred embodiment, when analyzing the classification model of high-dimensional images on a deep neural network, image patches rather than pixels are used as individual features. Therefore, the proposed method can be applied to high-dimensional data, such as images, thereby enabling efficient analysis.

[0019] The preferred shape of the image patch is square. This makes f(x i ) calculation is possible.

[0020] According to a further preferred embodiment, instead of splitting the image into separate image patches, the patches are slid over the entire image with a fixed stride. This takes into account that not all pixels in an image patch are equally important. While patches that are too large result in a low-resolution saliency map, patches that are too small are not visually understandable. To avoid this dilemma, instead of splitting the image into separate patches, it is proposed to slide the patches over the entire image with a fixed stride, as in a convolution operation.

[0021] According to a further preferred embodiment, the classification model is a model agnostic approach.

[0022] The invention furthermore relates to a computer program product directly loadable into the internal memory of a digital computer, comprising software code portions for performing the steps of the method as stated herein when said product is run on a computer. The computer program product may be in the form of a storage medium, such as a USB memory stick, a CD, a DVD, an EEPROM memory, etc. Alternatively, the computer program product may be in the form of a signal transmittable via wireless or wired lines.

[0023] According to a further aspect, a processing unit for computer-implemented analysis of a classification model is proposed. The classification model is adapted to map multiple input instances, each of which has multiple features, into multiple probabilities of output classes as predictions according to a predetermined function as classification decisions. The classification model is further adapted to determine a relevance value for each feature, which results in a saliency map. The processing unit is adapted to: identify the influence of each feature on the prediction on the instance by determining relevance information for each feature, the relevance information representing all features for the instance, omitting contextual information of the considered feature; determine a relevance value for each feature by combining the relevance information for the instance features, and evaluate the multiple relevance values ​​for the instance features to identify the influence of each feature on the instance prediction.

[0024] The processing unit according to the invention has the same advantages as stated in connection with the method described above.

[0025] According to a further preferred embodiment, the processing unit is furthermore adapted to implement the steps of the method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Further advantages, features and embodiments will be described with reference to the drawings.

[0027] Figure 1 A program sequence according to an embodiment of the present invention is shown.

[0028] FIG. 2 shows a table illustrating explanations generated for text classification by a Naive Bayes classifier through different approaches. DETAILED DESCRIPTION

[0029] A deep neural network is a function that maps input features into a target space. Individual classification decisions of a deep neural network can be explained via saliency methods, by creating saliency maps. However, saliency maps produced by existing methods are unreliable. Two major model-agnostic explanation methods are LIME and PDA.

[0030] LIME trains small classifiers (e.g., linear classifiers) with interpretable weights to approximate the local decision boundaries of deep classifiers. This approach requires training a new classifier for each classification decision, which is not efficient. Given a single classification, the optimization of the classifier can end up with different parameters, which leads to inconsistent explanations. When the classification is explained by using different classifiers, LIME produces different explanations.

[0031] PDA is another model agnostic method for analyzing classifiers. PDA is a probabilistic method and identifies the relevance of input features by marginalizing the influence of the feature and calculating its predicted difference. PDA faces many challenges when it is applied to high-dimensional data. It requires many forward inferences, which is very slow. In order to reduce the number of inferences, it can analyze images at the superpixel level. In addition to this, PDA suffers from another problem: namely, the method does not work well when many features of the classified instance support a classification known as saturated classification.

[0032] In the following, PDA will be explained in more detail, as it will form the basis for the proposed improved method for computer-implemented analysis of classification models.

[0033] In general, a classifier is a function that maps an input instance to a probability of an output class f:x→f(x). Each instance has n features x={x 1 , x 2 , ..., x n Given a classification decision f(x) for instance x, the saliency method adds i Assign a relevance value r i The resulting saliency map is defined as R = {r 1 , r 2 , ..., r n}. To identify the impact, feature x i Given a prediction about instance x, PDA computes the difference between the two predictions:

[0034] r i =D(f(x),f(x\i )), (1)

[0035] Where f(x \i ) means that in the absence of feature x i The model prediction (marginal prediction) in the case of cognition of x, and D() is a function that measures the difference between the two predictions. Two different ways can be used to evaluate such a difference. The first way is based on information theory. For a given class y, this measure calculates the difference in the amount of information necessary to find that y is true for a given instance x.

[0036] r i =log 2 p(y|x)-log 2 p(y|x \i )[bit]. (2)

[0037] The second way directly calculates the difference between the two probabilities as follows:

[0038] r i =p(y|x)-p(y|x \i ). (3)

[0039] In the following, the difference is evaluated by using the second metric. The first term p(y|x) can be obtained by classifying the instance x using only the classifier to be explained. The second term is calculated by substituting the feature x i Set to unknown, which is only allowed in a few classifiers, such as the Naive Bayes classifier. i Another way to affect is to retrain the classifier with the feature omitted. When the input examples are high-dimensional, such as with images, such a retraining process is not feasible. A practical way to simulate the absence of a feature is to marginalize the feature:

[0040] where v k is for the feature variable x i The sampled value of. The conditional probability p(x j |x \i ) can be used to obtain the prior probability p(x j ) is approximated and a single input variable is considered as one feature.

[0041] As will be shown below, PDA suffers from classifier saturation. Given an instance x = {x 1 , x 2 , x 3} and a classifier that assigns a probability c to a class: p(c|x)=max(x 1 , x 2, x 3 ), where x i ∈[0, 1], a single classification decision is made by the classifier using PDA. The specific instance x = {1, 1, 0} is classified as p(c|x) = max(1, 1, 0) = 1. After PDA, the correlation values ​​of the three input features are as follows:

[0042] r i =p(c|x)-p(c|x \i )=0,i∈{1,2,3}. (5)

[0043] The analysis results mean that the correlation values ​​of all features are zero, which is consistent with the feature {x 1 , x 2}The fact that this supports the classification decision is the opposite.

[0044] The saturation phenomenon is widely present in existing state-of-the-art classifiers. In image classification, many visual features can contribute to a single classification decision simultaneously. For example, in an image, two zebras support that the image belongs to the class "zebra". When analyzing the classification decisions on a series of state-of-the-art classifiers by using the prediction differences, different parts of the image are removed.

[0045] Table 1 shows the predicted difference of corresponding samples, which is the relevance value of the removed visual features. All classifiers are confident that the original image is classified as a zebra. In image img_1, the head of a zebra is removed, and the rest of the visual features are classified. The confidence of all classifiers is hardly reduced, which means that these classification decisions are saturated. Similarly, if the head of another zebra in image img_2 is removed, all classifiers also show high confidence as in the original image img_0. The removal of the part of the background does not reduce the classification score (img_3). PDA assigns the difference to the classification decision as the relevance of the removed visual features. Observe that PDA does not work in this case.

[0046]

[0047] Table 1: Classification scores for different images.

[0048] The proposed method for computer-implemented analysis of classification models can be referred to as contextual prediction differential analysis (CPDA). This method is able to overcome the above-presented shortcomings. i When removed from the original instance x, according to PDA equation (3), low prediction variance means that feature x i Not relevant, i.e. r i =f(x)-f(x \i )≈0.

[0049] According to the method of the present invention, instead of directly processing a single feature x i , calculate the relevance of context information, i.e. omitting multiple features or a single feature x i the rest of the.

[0050]

[0051] Where R \i Except x i Reference values ​​for all features except k Is x i is the sampled value of the condition. \i can be distributed to the individual features in many different ways. For simplicity, we assume an equal distribution:

[0052]

[0053] An alternative formulation for identifying the relevance of individual attributes is r i =f(x i ), which calculates the impact of a single feature. However, this formulation cannot identify negative evidence because r i =f(x i )≥ 0. The order of the identified correlation values ​​is equivalent to that when all the remaining parts of the features are marginalized.

[0054] By using the demonstration distance x={1,1,0} presented above, where the distance is classified as class c with probability p(c|x)=max(1,1,0)=1, the analysis of equation (7) using contextual PDA (CPDA) is as follows:

[0055]

[0056]

[0057] Features 1 and x 2 make equal positive contributions to the decision. In this demonstration example, the last feature x 3 It does not contribute anything to the decision. The analysis results show that CPDA can overcome the saturation problem. The reason behind this is that the relevance of each feature depends on the reasoning of all context features, not the feature itself (i.e. the omitted feature).

[0058] In the case where the input is high-dimensional, such as an input image, it is not efficient to treat pixels as individual features. On the other hand, individual pixels are not visual features that can be understood by humans. Therefore, it is proposed to take image patches as features.

[0059] Neural network based classifiers usually require a fixed input size, and the selected image patch should be resized to the required size. The shape of the selected image patch is square, which makes f(x i ) is possible. Therefore, it is assumed that the image classifier is invariant to translation and scaling operations, which is satisfied in existing convolutional neural networks.

[0060] Since not all pixels within a patch are equally important, the resolution of the obtained saliency map depends on the size of the individual image patches. While patches that are too large result in low-resolution saliency maps, patches that are too small are not visually understandable because f(x i ) does not make sense. To avoid this dilemma, instead of splitting the image into separate patches, slide the patch over the entire image with a fixed stride, as in a convolution operation.

[0061] In the following, text and image classification decisions are explained using different approaches compared to the proposed method.

[0062] Text Classification Using Naive Base Classifier

[0063] Take the text dataset "20newsgroup" as described in [1], which is a collection of approximately 20,000 news group documents, evenly partitioned across 20 different news topics. Each document is an email message, including content, headers, signature blocks, and possible quote blocks.

[0064] The best trained classifier makes decisions based on information in the header and footer. The sender and receiver are associated with universities and colleges. The header or footer is informative enough to classify the document. The classifier does not make decisions based on words related to certain topics. To avoid overfitting the classifier to irrelevant information, the header and footer are removed from all documents in the dataset.

[0065] Given a class variable y and a dependent eigenvector {x 1 , x 2 , ..., x n} and the assumption that all feature variables are independent, Bayes' theorem shows the relationship

[0066]

[0067] A single instance via To be classified.

[0068] One of the classic Naive Bayes variants used in text classification is Multinomial Naive Bayes, where data is typically represented as word vector counts. TF-IDF vectors are also known to work well in practice. The TF-IDF vectors of the documents are taken as input features to estimate the parameters of the model. For each class, the model parameters are θ y ={θ y1 ,θ y2 , .., θ yn}. Parameter θ yi By maximum likelihood The smoothed version of yi is the number of times the feature appears in samples of class y in the training set, and N y is the total count of all features for that class. The smoothing factor α is empirically set to 0.01.

[0069] For the explanation of individual document classification, the parameters of the multinomial NB classifier are estimated using the training dataset of “20newsgroup” and achieve 70.09% accuracy on the test dataset. yi Assign the sensitivity of the classifier for class y to feature x i For each classification, given the output class, the most sensitive words appearing in the document are selected, which is called sensitivity analysis (SA). Other model agnostic methods LIME, PDA and the proposed CPDA are also applied to explain individual document classifications.

[0070] The explanations of the classification are shown in FIG2 . Each column corresponds to the classification of a single document D1, ..., D4. The first row R1 shows a portion of the document text; the second row R2 shows the ground truth class GTC, the predicted class PC, the probability PD, and the saturation SD (the expansion that becomes close to one). The explanations generated by the four methods are correspondingly shown in the last row R3.

[0071] In SA's method, words with high sensitivity are the top words because they will appear many times in the document. The related words identified by SA cannot explain the classification of the document. They are highlighted in bold italics. The identified related words cannot convince the user to trust the classification decision. For each document, method LIME generates 5000 distorted versions by removing some words, predicts the probability of these distorted texts by using a naive Bayes classifier, and trains logistic regression using SGD (stochastic gradient descent). SGD is a basic optimization method for training neural networks. The weights of logistic regression are used to identify words that make positive contributions to the classifier. As shown in Figure 2, the resulting explanation is more convincing.

[0072] In most classifications, especially those with low saturation, such as document D1, both PDA and CPDA give similar good explanations, which successfully identify words related to the corresponding target class. In classifications with high saturation SD, PDA suffers from saturation of the classification, where the maximum relevance value is close to zero, and many of the words in most words are exactly zero. CPDA produces better explanations without suffering from the saturation problem. For example, in the classification of document D3, PDA identifies the word "in" as the most relevant word for the topic "sci.med". In contrast, the word "yeast" in document D3 is identified by CPDA as the most relevant to the topic. CPDA produces reliable and stable explanations for individual classifications.

[0073] Image classification using deep convolutional neural networks (CNNs)

[0074] In the following, single image classification with deep convolutional neural networks is explained. Take a pre-trained version of the model (AlexNet default f()) from the pytorch framework as a classifier, which requires a fixed input size of 224×224. An input image x with size H×W is resized to the fixed input size, processed, and classified f(x). A square image patch k×k is cropped from the processed image of 224×224 and resized again to the fixed input size. A single image patch x i Classified as f(x i ). The image classifier is assumed to be invariant to image translation and scaling. When resizing an image patch, rather than resizing the cropped patch itself, the corresponding patch of the original image is resized to avoid unintended scaling effects, such as excessive blurring. Therefore, f(x i ) can describe the patch x i The impact on forecasts.

[0075] In empirical experiments it can be shown that the method according to the present invention outperforms LIME and PDA. In particular, the proposed method is more consistent and efficient.

[0076] For model-agnostic saliency methods, efficiency is a non-negligible factor. Since they do not have access to the parameters of the classifier, the algorithm must make many inferences to obtain enough information for identifying the correlation of the input features. In addition, the consistency of the explanations produced is another important indicator for evaluating saliency methods. Consistent explanations are more meaningful for understanding the classifier and gain more trust from end users.

[0077] For a single instance, LIME trains a local proxy classifier to approximate the local classification boundary of the target classifier. Although the proxy classifier takes superpixels as separate features, LIME takes several minutes to generate explanations for the classification. The explanations generated by LIME are inconsistent. The sources of inconsistency may be the modeling and sampling process, the type of proxy classifier, and the organization process of the proxy classifier.

[0078] PDA treats patches of the image as separate features for generating explanations. For each separate patch, PDA first distributes Modeling is performed and patches from the distribution are sampled. The modeling and sampling process is computationally expensive. Given an input image of size n×n and patches of size k×k, S(n-k+1) are required. 2 A forward pass is performed to evaluate the relevance of all pixels, where S is a pixel from the distribution It takes tens of minutes to generate an interpretation.

[0079] Compared to LIME, CDPA requires neither the construction of noisy variant instances nor an additional classifier. Unlike PDA, CPDA does not need to model the conditional distribution of image patches. Image patches are directly classified to identify the relevance of context pixels. With the same settings as in PDA, by using CPDA, only s is the stride. CPDA is S*s of PDA 2 When s=5, it takes only seconds to minutes to generate an explanation. Table 2 lists the average time for generating a single explanation.

[0080]

[0081] Table 2: Average time taken to produce a single explanation.

[0082] The first row shows the time to run the methods (LIME, PDA, CPDA) on AlexNet on a CPU, while the other rows show the time on a single GPU (Tesla K80).

[0083] VGG16, outlined in Table 2, has an architecture similar to AlexNet, but with more and deeper layers. The deeper VGG16 identifies the most distinctive features to classify images. Instead of the wheels shown in the original image to be assessed, for example, a sports car image is classified based on the roof, which can be used to distinguish it from taxis and buses. The most distinctive feature of the hot air balloon image is the balloon basket for VGG16. Indian elephants are different from other types of elephants in their ears. Indian elephants have smaller ears than others. In the explanation on VGG16, the corresponding distinctive visual features are highlighted. GoogleNet (also known as Inception_V3) convolves the image and filters it using multiple kernel sizes, where features of different sizes can be identified as class distinctive features, such as part of a balloon in an image. ResNet and DenseNet are equipped with jump connections, where they learn features from a global view. Together with contextual information, distinctive features are used to classify images. For example, sports cars and runways are extracted as relevant features for classification. Both the balloon and the basket are used to identify the image as a hot air balloon.

[0084] Figure 1 The steps of a method for computer-implemented analysis of a classification model as described above are illustrated. In step S1, the influence of each feature on the prediction of an instance is identified by determining correlation information for each feature, the correlation information representing all features for the instance, omitting context information of the considered feature. In step S2, a correlation value for each feature is determined by combining the correlation information for the features of the instance. In step S3, the multiple correlation values ​​are evaluated for the features of the instance to identify the influence of each feature on the prediction of the instance. Steps S1 to S3 are implemented by a processing unit PU, which is adapted to perform the steps as described above.

[0085] References

[0086] [1] Lang, K. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995, pp. 331-339. Elsevier 1995.

Claims

1. A method for computer-implemented analysis of classification models - which is adapted to map, as prediction, according to a predetermined function, a plurality of input instances, each of which has a plurality of features, into a plurality of probabilities of output classes as classification decisions; and - which is adapted to determine a relevance value for each feature, which results in a saliency map; The following steps are involved: - identifying the influence of each feature on the prediction of the instance by determining for each feature relevance information representing contextual information of all other features for the instance if the feature under consideration is omitted; - determining a relevance value for each feature by combining the relevance information of the features for the instance; - evaluating the plurality of correlation values ​​for the features for the instance to identify the impact of each feature on the prediction of the instance; When analyzing classification models of high-dimensional images on deep neural networks, image patches rather than pixels are used as separate features.

2. The method according to claim 1, wherein the relevance value for each feature is calculated by the sum of the ratio of the relevance information and the absolute value of the relevance information, wherein the feature is Features under consideration are omitted. The method according to claim 1 , wherein the correlation value for each feature is equal to or greater than 0. 4 .

4. The method according to claim 1, wherein the correlation information (R \i ) is determined by calculating: where v k Is x i is the sampled value of the condition, x i is feature i of instance x, M is the number of samples or classes considered, and y is the output class. The method of claim 1 , wherein the image patch is square in shape.

6. A method according to claim 1 or 5, wherein instead of splitting the image into separate image patches, the patches are slid over the entire image with a fixed step.

7. The method of claim 1, wherein the classification model is a model agnostic method.

8. A computer program product directly loadable into the internal memory of a digital computer, comprising software code portions for performing the steps of one of the preceding claims when said product is run on a computer.

9. A processing unit for computer-implemented analysis of a classification model adapted to map, as a prediction, a plurality of input instances, each of which has a plurality of features, into a plurality of probabilities of output classes according to a predetermined function as a classification decision; and adapted to determine a relevance value for each feature, which results in a saliency map; The processing unit is adapted to: - identifying the influence of each feature on the prediction of the instance by determining for each feature relevance information representing contextual information of all other features for the instance if the feature under consideration is omitted; - determining a relevance value for each feature by combining the relevance information of the features for the instance; - evaluating the plurality of correlation values ​​for instance features to identify the impact of each feature on the instance prediction; When analyzing classification models of high-dimensional images on deep neural networks, image patches rather than pixels are used as separate features.

10. The processing unit according to claim 9, furthermore being adapted to implement the steps of claim 2.

Images