A quantization method to improve the fidelity of rule extraction algorithms for use in artificial neural networks.

Abstract

To provide a quantization method to improve the fidelity of rule extraction algorithms for use with artificial neural networks (ANN).SOLUTION: Rules for explaining an output of an ANN are derived by: creating decision trees trained to approximate the ANN and optimize a defined criterion, a threshold value for the criterion being calculated to determine for which node of the ANN the input activations should be split between branches of the decision tree; obtaining threshold value combinations each comprising a threshold value obtained for respective nodes of the ANN; for each combination, performing a rule extraction algorithm to obtain a fidelity metric indicating the accuracy of the rule; determining which combination yields the best fidelity metric; and using the rule extraction algorithm with the combination of threshold values determined.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] This paper describes a quantization method for improving the fidelity of rule extraction algorithms used in artificial neural networks. [Background technology]

[0002] The need for interpretable artificial intelligence (AI) models is growing in both industry and academia. In many key areas where machine learning is applied, such as healthcare, autonomous driving, and finance, models need to explain their decisions in a way that is interpretable to humans in order to be trusted. For example, explaining the decision-making process of a neural network helps physicians trust the neural network's output classifications, enabling them to better assess a patient's condition and reduce human error. Furthermore, the way network training datasets are collected can introduce hidden biases or spurious correlations between different classes, which can cause the trained model to learn to make more accurate predictions for the wrong reasons and become lazy. Explaining black-box models can help field experts identify errors and potential biases in those models. After diagnosing these errors / biases, the model may be retrained to produce a trustworthy model.

[0003] Methods for extracting rules to explain the decision-making process of neural networks essentially take quantized filter / neuron activations from a subset of layers as input and measure their relevance to the target output. Because filters consist of many neurons, many post-hoc rule extraction algorithms for CNNs rely on aggregating filter activations into a single value. Thresholding is then performed to determine if a filter can be considered active (a process known as "quantization," see J. Townsend, T. Chaton, JM Monteiro, "Extracting relational explanations from deep neural networks: A survey from a neural-symbolic perspective," IEEE Transactions on Neural Networks and Learning Systems 31 (2020) 3456-3470) to detect specific patterns between images. In the case of feedforward neural networks (FFNNs), rule extraction methods directly threshold neuron activations, so aggregation is not necessary.

[0004] As discussed in H. Jacobsson, "Rule extraction from recurrent neural networks: Ataxonomy and review," Neural Computation 17 (2005) 1223-1263, many different techniques are known in this field for determining whether a filter / neuron is considered active. For example, the sigmoid activation function as a nonlinearity.

number

[0005] By associating filters / neurons with literals, rules are formed to explain the classification output of the CNN, with each atom used in the explanation corresponding to an active filter. Rule extraction may also be applied to a general ANN by replacing filters with neurons.

[0006] However, if the fidelity metric—that is, the accuracy of the extracted program relative to the original model (e.g., a measure of how closely the output of the logic program matches the model's predictions)—is not high enough, such a post-hoc rule extraction model remains essentially a black box. In this regard, existing approaches that rely on sample statistics are not always appropriate, because each filter activation may follow a different distribution, and thresholding of sample statistics can overestimate or underestimate when a filter / neuron is considered active. For the output rules to more faithfully approximate the original model, the quantization step that converts the float representation (or matrix of float values ​​in the case of a convolutional neural network (CNN)) to binary values ​​should result in minimal information loss. [Overview of the project]

[0007] Therefore, it is desirable to develop an improved quantization approach to achieve higher fidelity during rule extraction.

[0008] An embodiment according to the first aspect is a method implemented on a computer, comprising: recording node activations for each node in the layers of a trained ANN, and predictions of the ANN for each item of the training data used to train the ANN; creating at least one decision tree with the recorded node activations as input and the recorded ANN predictions as target, each decision tree being trained to approximate the ANN and optimize a defined criterion; for each node of the decision tree, a threshold for the defined criterion is calculated to determine which node of the ANN the input activations should be split among the branches of the decision tree; recording the thresholds associated with each node of the ANN; and each threshold combination for each node of the ANN A method may be provided that includes: obtaining a threshold combination including one of the obtained thresholds; for each threshold combination, using the threshold combination to run a selected rule extraction algorithm to extract from the ANN at least one rule to explain the output of the ANN layers; using the threshold combination to obtain a fidelity metric for at least one rule, the fidelity metric indicating the accuracy of the rule in predicting the ANN; determining which of the threshold combinations yields the best fidelity metric; and using the threshold combination determined to yield the best fidelity metric, using a selected rule extraction algorithm to extract at least one rule to explain the output of the ANN layers.

[0009] An embodiment according to the second aspect is a computer program having instructions, which, when executed by the computer, cause the computer to carry out a method that embodies the first aspect.

[0010] An embodiment according to a third aspect includes at least one computer processor and a memory connected to the at least one computer processor that stores node activations for each node in a layer of a trained artificial neural network (ANN), predictions of the ANN for each item of training data used to train the ANN, and instructions, wherein the instructions are to create at least one decision tree, taking the recorded node activations as input and the recorded ANN predictions as a target, each decision tree being trained to approximate the ANN and optimize a defined criterion, and for each node of the decision tree, a threshold against the defined criterion is calculated to determine which node of the ANN the input activations should be split among the branches of the decision tree, and the thresholds associated with each node of the ANN are recorded. An apparatus may be provided that performs the following: obtaining threshold combinations such that each threshold combination includes one of the thresholds obtained for each node of the ANN; for each threshold combination, using the threshold combination, running a selected rule extraction algorithm to extract at least one rule from the ANN to explain the output of the ANN layers; using the threshold combination, obtaining a fidelity metric for at least one rule, the fidelity metric indicating the accuracy of the rule in predicting the ANN; determining which of the threshold combinations yields the best fidelity metric; and using the threshold combination determined to yield the best fidelity metric, using the selected rule extraction algorithm to extract at least one rule to explain the output of the ANN layers.

[0011] Embodiments provide a quantization method aimed at finding a threshold for each filter in a CNN (or more generally, for each neuron in an artificial neural network (ANN)) that more accurately captures when a filter (or neuron) is truly active when detecting a specific pattern in an input image. Embodiments find thresholds that result in minimal information loss during the quantization process, helping subsequent rule extraction algorithms approximate complex black-box models more faithfully. According to one embodiment, this is achieved by training multiple decision trees that approximate CNN / ANN behavior, constructed by taking input filter / neuron activations (unquantized) and CNN / ANN predictions as outputs. The thresholds used when splitting nodes of each tree are separately stored in a list T for each filter f j f j separately. Thereafter, rule extraction is performed on threshold combinations (e.g., all possible threshold combinations), and the combination of thresholds that yields the highest fidelity, that is, the threshold that results in a logic program more faithfully approximating the behavior of the original model, is selected.

[0012] Performing a search in the space of thresholds found by decision trees (e.g., random forests) approximating an ANN and selecting the one that yields higher fidelity results in an extracted program that more faithfully approximates the model. The thresholds used when splitting nodes of decision trees are incentivized to reduce information loss to optimize fidelity, so embodiments may find more appropriate thresholds for each filter that more accurately capture when the filter is active.

[0013] The optimal threshold is found by optimizing user-defined criteria and constructing decisions where each tree approximates the neural network instead of relying on sample statistics, so embodiments are very versatile and do not depend on any knowledge about the distribution of filter activations.

[0014] According to the first or second aspect, storing the thresholds may comprise ranking the thresholds according to the occurrence frequency and average occurrence depth in the decision tree for each node of the ANN, and executing the selected rule extraction algorithm for each of the threshold combinations comprises first executing the selected rule extraction algorithm on a threshold combination comprising thresholds that occur at the highest frequency.

[0015] According to the first or second aspect, obtaining the threshold combinations may comprise one of: obtaining all possible combinations of thresholds; obtaining combinations using only a preset number of the most frequently occurring thresholds for each node of the ANN; obtaining combinations using only a random subset of thresholds for each node of the ANN; and obtaining combinations of only thresholds for each node of the ANN that satisfy a user-defined metric.

[0016] According to the first or second aspect, the defined criterion to be optimized may be entropy or the Gini coefficient.

[0017] According to the first or second aspect, recording the thresholds associated with the respective nodes of the ANN may comprise recording, as a threshold for a node, the average activation per sample of the node when no threshold associated with the specific node of the ANN exists.

[0018] According to the first or second aspect, generating the at least one decision tree may comprise generating a plurality of diverse decision trees using a random forest generation algorithm.

[0019] The method, program or device according to the embodiments may be used for extracting at least one rule for an ANN for use in one of an autonomous driving algorithm and a healthcare algorithm.

[0020] A method, program, or apparatus according to an embodiment may be used to either (i) extract at least one rule for a CNN used to control an autonomous vehicle, or (ii) use the extracted at least one rule to determine whether an ANN is functioning correctly.

[0021] The embodiment can boost the fidelity of a post-rule extraction algorithm that attempts to explain the decisions of a black-box model by providing extracted rules that more faithfully explain the model's decisions in a human-understandable, interpretable language. Furthermore, an extraction program with high fidelity may be used to detect hidden biases or errors in the original model, which may then be retrained to eliminate these problems and obtain a more robust model.

[0022] Many consumers may be fascinated by black-box models accompanied by rule-extraction algorithms that are highly accurate and faithfully explain the model's decisions to the consumer. Accurately interpreting the decisions of black-box models is crucial for applying deep learning to critical areas such as healthcare, fully autonomous driving, criminal justice, and finance.

[0023] Finally, the proposed method may contribute to XAI research and support neurosymbolic integration by reducing the gap between neural and symbolic representations. This is because, after applying the proposed method symbols, the model state can be captured more accurately due to the reduction of approximation losses during quantization.

[0024] The rule extraction algorithm takes filter activations from a subset of layers as input and measures their association with the target output. Embodiments may be applied to any rule extraction algorithm that relies on neuron / filter activation to extract knowledge from a neural network and explain its decisions. For example, embodiments may be combined with a post-hoc rule extraction program proposed by J. Townsend, T. Kasioumis, and H. Inakoshi in "ERIC: extracting relations inferred from convolutions," 15th Asian Conference on Computer Vision, Kyoto, Japan, Revised Selected Papers, Part III, volume 12624 of Lecture Notes in Computer Science, Springer, Nov. 30 - Dec. 4, 2020, pp. 206-222) to achieve better and more interpretable rules. However, any other existing or future methods of mapping filters / neurons to literals and generating rules across those literals may also be used. [Brief explanation of the drawing]

[0025] For example, a reference to the attached drawing is provided.

[0026] [Figure 1] This graph shows the magnitude of filter activation as a function of count for an exemplary image dataset using mean activation as the threshold. [Figure 2] This is a graph of the magnitude of filter activation against counts for an exemplary image dataset using the mode of distribution as the threshold. [Figure 3] This flowchart outlines the quantization methods applied to CNN filter activation. [Figure 4] This example shows the process of outputting feature maps from filters in a CNN layer. [Figure 5] This is a flowchart of the method according to one embodiment. [Figure 6] A module of the system according to one embodiment is shown. [Figure 7] Here's an example of how road signs can be altered to fool CNN. [Figure 8] A decision tree used to illustrate the embodiment is shown. [Figure 9] This is a block diagram of a computing device suitable for carrying out a method according to one embodiment. [Modes for carrying out the invention]

[0027] Without loss of generality, the proposed method is described below for convolutional neural networks, but it should be noted that everything described in this section can be generalized with little effort to ANNs by replacing filters with neurons or nodes in technical terms.

[0028] The fidelity of a rule extraction program that relies on quantization of filter activations to describe the output classification of a trained CNN depends heavily on the quantization procedure. Quantization can be considered a two-step process: a) application of aggregate functions (mean, sum, maximum, etc.), and b) thresholding. The goal of quantization is to convert the output feature map generated by convolutional kernel multiplication into binary values ​​in order to detect whether a kernel is considered active, i.e., to detect patterns in the input image. The aggregate function is necessary to convert the multidimensional feature map into a single value indicating the intensity of kernel activation, while thresholding is necessary to determine at what value θ a kernel is considered active. For the output rule to faithfully approximate the original CNN (i.e., have high fidelity), the quantization step of converting the float representation (or matrix of float values ​​in the case of a CNN) to binary values ​​should result in minimal information loss. If the quantization step introduces a high approximation error ε, the rule extraction cannot reach its highest potential, and the fidelity of the extracted program is capped by 1-ε. Therefore, developing a more theoretical quantization approach is key to achieving higher fidelity during rule extraction.

[0029] As mentioned above, many approaches in the literature use sample statistics to determine whether a filter is considered active and detect specific patterns in the input images. For example, in the case of a CNN, different images from the dataset pass through the network and filter activation is recorded.

number

number

number

number

number

number

number

number

number

[0030] However, such thresholds do not actually determine whether a filter is truly active. One reason for this inefficiency is that such thresholds are only meaningful if the distribution of activations for each filter is Gaussian-like. This assumption can actually be strongly violated if the distribution has heavy tails, if the distribution is highly skewed, or if it is, for example, multimodal.

[0031] Another problem with using the mean as a threshold is that it is heavily influenced by extreme values. For example, consider a dataset of images where a subset of images has many traffic signs (these signs may have been captured from a very busy area of ​​town or a training driving course), the majority of images contain one traffic sign, and the rest contain zero traffic signs (e.g., captured in a forest). Images with multiple traffic signs are expected to have very large feature maps for the filter that detects traffic signs (because many traffic signs are detected in multiple locations). As shown in Figure 1, these extreme values ​​"push" the mean μ to a higher value, and if there are many such extreme values, it can even result in a bimodal distribution. For pictures with only one traffic sign, the magnitude of the activation of the filter that detects traffic signs may not be high enough to exceed the mean. In that case, if the threshold is the mean, the threshold will not correctly determine whether the filter is active or not.

[0032] Replacing the mean with the mode of the distribution (since the mode is not affected by extreme values) solves the previous problem but introduces another. Consider a dataset where most of the images are captured in one town and contain multiple traffic signs. In these images, the activation magnitude of the filter for detecting traffic signs will be very high because there are many peaks in the feature map. Suppose there are some images with only one traffic sign that is far away (and therefore appears small in the image). This means the activation magnitude is small but greater than zero. Also, suppose there are some images with no traffic signs. This dataset could result in an activation distribution for the filter for detecting traffic signs, as shown in Figure 2. If the mode of the distribution is chosen as the threshold to determine whether the filter is active, the filter will be incorrectly considered not active for images containing only one traffic sign. This introduces an error in subsequent rule extraction algorithms that rely on quantization of literals, which degrades fidelity. Similar examples may be constructed to demonstrate that other thresholds that rely on sample statistics can indeed fail.

[0033] Another reason why thresholds that rely on sample statistics can actually fail relates to the size of the patterns detected by the filter. Objects farther away from the camera will appear smaller in the image and, therefore, after many convolution and max pooling operations, will occupy a smaller area in the feature map in the layer of interest. For example, suppose a filter is fired in response to a traffic sign, and the image contains a traffic sign far from the camera. Then, after convolution and max pooling operations, the corresponding traffic sign activation region will be much smaller. As a result, the activation magnitude will also be small, due to many aggregates with zero activation regions around the traffic sign. This means that the activation magnitude of the filter may be lower than the average activation threshold across different images, and the filter may be incorrectly considered unactivated even if the image contains a traffic sign.

[0034] From the examples above, it is clear that mean or mode-dependent thresholds do not generalize well to any arbitrary distribution of filter activation. These types of thresholds introduce high approximation errors during the quantization process, which affects the rule extraction fidelity.

[0035] This proposal introduces a quantization method aimed at finding a threshold that accurately captures when a filter is truly active and maximizes the fidelity of the subsequent rule extraction program. This is achieved by training a random forest in which each tree approximates neural network behavior and is constructed greedily (i.e., according to a greedy algorithm) to optimize a user-defined criterion. Each tree takes filter activations (unquantized) from a subset of CNN layers as input and CNN predictions as its target. The threshold used when splitting the nodes of the tree in the random forest is T for each filter f. f They are stored separately. Rule extraction is then performed for each possible threshold combination (an exhaustive grid search may be performed to find the threshold that yields the best fidelity, but other approaches, such as random search or Bayesian approaches (i.e., guided search), are possible and select thresholds that yield higher fidelity, i.e., thresholds that result in a logic program that more faithfully approximates the behavior of the original CNN). Information loss can be reduced by choosing a better policy for quantizing the kernel, and the fidelity of the rule extraction algorithm that describes the decisions of the neural network may be increased.

[0036] The embodiments are distinguished from the prior art by one or more of the following: 1. One or more decision trees, such as a random forest, are used to find reasonable thresholds for extracting rules that optimize user-defined criteria. 2. The threshold used when splitting the nodes of each tree is selected to maximize a user-defined criterion (e.g., entropy, Gini coefficient) in order to minimize information loss during thresholding, and is stored for each filter. 3. By exploring the threshold space defined by the random forest, the optimal threshold is selected to maximize the fidelity of the subsequent rule extraction algorithm. 4. Each tree in the forest approximates the original CNN, and is trained to take filter activations from a subset of the layers used in rule extraction as input, and to predict CNNs as the target. 5. A list of thresholds is searched, and each global threshold is selected separately for each filter, so that subsequent rule extraction algorithms using such thresholds have maximum fidelity.

[0037] Figure 3 is a flowchart 30 outlining the quantization method 34 applied to the filter activation 33 of the CNN 32. A filter is considered active if it exceeds a predetermined threshold (the illustrated image 31 is taken from the places365 dataset: B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, A. Torralba, "Places: A 10 million image database for scene recognition", IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (2018) 1452-1464) for the input image 31. To apply the proposed thresholding policy to the CNN, first, as shown in Figure 3, an aggregate step 34A should be performed to convert the filter activations into single values. The aggregate step 34A is not necessary for FFNNs, because in this case, this method is applied directly to the neuronal activations to determine the optimal threshold. After the aggregate step 34A, the quantization method is applied to find the optimal threshold for each filter, determining the value above which the filter is considered active.

[0038] Before describing a method according to one embodiment, we will introduce some necessary prerequisites and clarify the notation.

[0039] To improve the fidelity of subsequent rule extraction algorithms that approximate the behavior of CNNs, J filters are added to layer l of the CNN.

number

number

[0040] C represents the total number of classes in the classification problem at hand, and c i This refers to image X in the dataset. i Let's show the correct label / class for each image. This is the training dataset for the images.

number

number

number

number

number

[0041] * represents a convolution operator followed by ReLU (and in some cases, maximum pooling depending on the architecture),

number

[0042] Definition: filter

number

number

[0043] As mentioned above, many previously proposed approaches use sample statistics to determine when a filter is active on a given image. The method proposed herein differs in that a random forest is used to find filter-specific thresholds, and each tree approximates the original CNN. Each filter [Math.]] a plausible threshold θ j after finding a list of thresholds, a search through the threshold space is performed to find the optimal threshold that provides the highest fidelity for the subsequent selective rule extraction algorithm.

[0044] In the remainder of this section, the steps of the proposed quantization method are discussed with reference to CNN for image data, but the described method may be generalized to any ANN using non-picture data by replacing filter activation (spatial average of feature maps) with neuron activation.

[0045] A flow diagram of the method is shown in FIG. 5, and modules of a system for implementing the proposed method are shown in FIG. 6. As will be described later with reference to FIG. 9, modules 61 to 63 in FIG. 6 may be implemented in one or more processors 993 of one or more computing devices 10, and the computing device may include one or more memories 994 for storing data input to the processor 993, data used by the processor 993, and / or data output from the processor 993.

[0046] Quantization process (numbering corresponds to steps in FIG. 5 and modules in FIG. 6): 1) Obtain training images. 2) Image

number

number

number

number

number

number

number

number

number

number

number

number

[0047] The algorithm for implementing the above method is described below: Quantization algorithm 1: Initialization: A layer l to apply quantization, and a dictionary T = {} that stores the threshold used to split the tree nodes in a random forest for each filter. 2. Construct a random forest that approximates a CNN: (As mentioned in step (3) above, it is preferable to use feature subsampling and bootstrap). 3: For loop: For each tree in a random forest, 4: - Train the tree to approximate a CNN using filter activation from layer l as input and CNN prediction as the target. 5: -After constructing the tree, list T f =[Threshold for filter f] stores the threshold used when splitting the tree node. 7: End of For loop 8: For each filter f (grid search), measure the fidelity of the extracted programs for all threshold combinations in list T[f]. For each filter, n filters and k n If there is a threshold, the total

number

[0048] paradigm The experiment was conducted using the quantization policy described above on a toy subset of the Placesse365 dataset, which includes three categories: "Forest Roads," "Highways," and "Streets." Hereafter, for convenience, this dataset of three classes will be referred to as the "Roads" dataset. The training-validation-test splits used were 10,445-1,500-3,055, with 500 images per class for validation and approximately 1018 images per class for testing. This dataset was selected because scenes can be described through the sub-objects and topics present within them, making it a good candidate for rule extraction.

[0049] Scene classification can be particularly useful in autonomous driving scenarios where neural networks are used to make decisions such as steering, braking, and accelerating. It is well known that neural networks are susceptible to adversarial attacks. A real-world example, as shown in Figure 7, is the extremely dangerous trick of a vehicle's autopilot into believing a 35 mph speed limit sign is actually an 85 mph sign by using adhesive tape over it. Having an additional network to classify scenes (or adding another classification head to an existing model) can be useful in this situation because the model can estimate from the scene whether it is wise to accelerate to 85 mph.

[0050] First, a network for multi-class classification was trained on the road dataset. The VGG-16 network was initialized with weights pre-trained on ImageNet, and then pre-processed and augmented as performed by A. Krizhevsky, I. Sutskever, and GE Hinton ("Imagenet classification with deep convolutional neural networks," Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012). The final high-density layer was fine-tuned over 50 epochs using the Adam optimizer (DP Kingma, J. Ba, Adam: "A method for stochastic optimization," 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015).

number

[0051] After fine-tuning VGG-16 for the "Roads" dataset, rule extraction was performed using the ERIC technique (J. Townsend, T. Kasioumis, H. Inakoshi, "ERIC: extracting relations inferred from convolutions", 15th Asian Conference on Computer Vision, Kyoto, Japan, Revised Selected Papers, Part III, volume 12624 of Lecture Notes in Computer Science, Springer, Nov. 30 - Dec. 4, 2020, pp. 206-222). The fidelity of the extracted programs was measured using the average activation per sample for each kernel as a baseline. The results are shown in the second column of Table 1. [Table 1] To measure its effectiveness, the proposed quantization policy was applied after fine-tuning VGG-16 on the "Roads" dataset. First, all images from the Roads dataset were passed through a CNN, and the filter activation in layer conv13 of VGG-16 was recorded.

number

number

[0052] After passing all images through the CNN, three tables are created (one for training, one for validation, and one for test datasets), with the i-th row of each table containing the activation for each sample.

number

number

[0053] Next, a random forest was trained on the training dataset (10,445 rows) using feature subsamples and bootstraps for a diversified tree. Figure 8 shows an example of a tree in the random forest. The lowest levels of the tree are the leaves, which are the predictions of the CNN model.

number

number

[0054] For each tree, the threshold for splitting the nodes of each tree is the list T. j The positions / depths of each tree, which are stored in the memory, are determined by each filter f. j Regarding List L jIt was stored in memory.

[0055] After training the random forest, a list of thresholds for each filter was obtained:

number

number

[0056] If the list is empty (i.e., the filter does not appear as a split node in the random forest), the average per sample of filter activations is added to the list.

[0057] Next, filter f j That is the list T j In this ranking, filters with longer lists were ranked in terms of the number of thresholds:

number

[0058] Then, all threshold combinations

number

number

[0059] The results of rule extraction using the best threshold combination described above are shown in the third column of Table 1. A 3% improvement in fidelity was observed after using the thresholds obtained from the proposed quantization method, compared to the baseline per average filter threshold of the samples.

[0060] Alternative examples In a medical scenario, the input image could be one or more medical images, such as an X-ray or CT scan, and the CNN is trained to identify the presence of one or more features in the input image. Based on this information, combined with an explanation of why the CNN made that decision, a medical professional may be able to identify the presence of an abnormality, make a diagnosis, and determine a course of treatment. For example, if the input image is an X-ray of the lung, the y labels in Figure 8 might instead represent the following: 0 = No decision made 1 = Presence of a mass in the lung or diagnosis of lung cancer 2 = Presence of fluid around the lungs or diagnosis of pneumonia

[0061] Application fields Achieving a high level of fidelity while maintaining interpretability is extremely beneficial for using high-accuracy black-box models in key areas such as:

[0062] Autonomous vehicles In autonomous driving, there is a great need for interpretable models, as deep neural networks may be responsible for steering, braking, stopping, accelerating, and other actions of the vehicle based on input images from the environment. Rule extraction algorithms may be used to explain the decisions of black-box models in a human-interpretable way in order to build trust in these models. The proposed method finds a more appropriate threshold to boost rule extraction fidelity, which means that the extracted program will represent the black-box model more faithfully.

[0063] Furthermore, if the model achieves high accuracy during training by relying on spurious correlations and biases, the extracted program may allow the user to ignore or debug the model's predictions (assuming the extracted program has sufficiently high fidelity). The model may then be retrained to mitigate these errors and biases.

[0064] The extracted rules can also be useful for audit purposes. Insurance agents may be interested in knowing why a vehicle misbehaved, what caused the accident, and who should be held responsible. Furthermore, the extracted rules can be useful for taxis, freight companies, and other transportation companies auditing their vehicles / employees, for example, when a customer or witness makes a complaint that is not bad enough to warrant an insurance or legal claim, but is still bad enough to damage the company's reputation.

[0065] Therefore, logic programs with high fidelity are highly desirable; otherwise, they cannot faithfully represent the black-box model and cannot explain their decisions in a human-interpretable way. The proposed quantization method helps in this direction by selecting a more appropriate threshold for creating faithful logic programs.

[0066] Healthcare In healthcare, while state-of-the-art deep neural networks can achieve high accuracy, their predictions may not be trusted by physicians, especially in life-threatening situations, when it is unclear what guides those decisions. Rule extraction algorithms approximate these models in a human-interpretable way, helping physicians understand the predictions of black-box models. Achieving high fidelity is key to understanding these models, as the fidelity of the extracted rules measures how well the output logic program approximates the black-box model; the proposed method assists in this direction. Similar to autonomous driving, these rules may also be used to point out bugs or errors in the model to a physician, who may then ignore their decisions or have the model retrained to correct them.

[0067] Other fields Highly faithful rule extraction algorithms can be extremely useful in financial applications, for example, in explaining decisions made by black-box models used for automated loan decisions or in predicting who is likely to default on their debts. They can also be very useful in criminal justice applications, for example, in suggesting sentences to judges making final verdicts, where judges need to understand how the sentence is derived from the facts about the crime and the defendant.

[0068] Figure 9 is a block diagram of a computing device 10, such as a data storage server, which can be used to embody the present invention, implement some or all of the operations of the method of embodying the present invention, and perform some or all of the tasks of the apparatus of the embodiment.

[0069] The computing device 10 includes a processor 993 and memory 994. Optionally, the computing device also includes a network interface 997 for communication with other such computing devices, e.g., other computing devices in the embodiments of the invention. Optionally, the computing device also includes one or more input mechanisms, such as a keyboard and a mouse 996, and one or more display units, such as a monitor 995. The components are connectable to each other via a bus 992.

[0070] Memory 994 may include computer-readable media, and the term may refer to a single or multiple media configured to carry computer-executable instructions (e.g., a centralized or distributed database and / or associated caches and servers). Computer-executable instructions may include instructions and data that are accessible by, for example, a general-purpose computer, a special-purpose computer, or a special-purpose processing device (e.g., one or more processors) and cause them to perform one or more functions or operations. For example, computer-executable instructions may include the method of Figure 5, or any other method steps disclosed herein, and / or data such as training, verification, and / or test datasets, the structure of an ANN, feature activation, decision tree derivation thresholds, and / or instructions for implementing a selected rule extraction algorithm. Thus, the term “computer-readable storage medium” may also include any medium that stores, encodes, or carries a set of instructions for machine execution and causes a machine to perform one or more steps of the methods of the present disclosure. Therefore, the term “computer-readable storage medium” includes, but is not limited to, solid memory, optical media, and magnetic media. Such computer-readable media may include, but are not limited to, non-temporary computer-readable storage media, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory devices (e.g., solid memory devices).

[0071] The processor 993 is configured to control the computing device and perform processing operations, for example, executing a computer program stored in memory 994, to implement some or all of the steps of the method described herein. For example, the code may implement only steps 2, 3, or 5-12 of the method in Figure 5, or all of steps 1-12 in Figure 5. Memory 994 stores data to be read and written by the processor 993. As referred to herein, the processor may include one or more general-purpose processing devices, such as a microprocessor or a central processing unit. The processor may include a complex instruction set compute (CISC) microprocessor, a reduced instruction set compute (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements a processor or combination of instruction sets. The processor may also include one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a network processor. In one or more embodiments, the processor is configured to execute instructions for performing the operations and steps discussed herein.

[0072] The display unit 995 may display a representation of data stored by the computing device, such as images input to a CNN, rules extracted from a CNN, and / or any other inputs / outputs described above, and may display a cursor, dialog boxes, and screens that enable interaction between the user and programs and data stored in the computing device. The input mechanism, 996 may enable the user to input data or instructions to the computing device, such as selecting a rule extraction algorithm to be used, criteria to be optimized, and / or a dataset to be used for training.

[0073] The network interface (network I / F) 997 may be connected to a network such as the Internet and may be connected to other such computing devices via the network. The network I / F 997 may control data input and output with other devices via the network. Other peripheral devices such as microphones, speakers, printers, power supply units, fans, cases, scanners, and trackballs may be included in the computing device.

[0074] A method for embodying the present invention may be implemented using a computing device / apparatus 10 as shown in Figure 9. Such a computing device does not need to have all the components shown in Figure 9, but may consist of a subset of those components. For example, apparatus 10 may include a processor 993 and a memory 994 connected to the processor 993. Alternatively, apparatus 10 may include a processor 993, a memory 994 connected to the processor 993, and a display 995. A method for embodying the present invention may also be implemented using a single computing device that communicates with one or more data storage servers over a network. The computing device may be a data storage device itself that stores at least a portion of the data.

[0075] The method for realizing the present invention may be carried out by a plurality of computing devices working together. One or more of the plurality of computing devices may be a data storage server that stores at least a portion of the data.

[0076] The present invention can be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The present invention can also be implemented as a computer program or computer program product, i.e., as a computer program tangibly embodied in a non-temporary information carrier, such as a machine-readable storage device, or in a propagating signal, for execution by or control of the operation of one or more hardware modules.

[0077] A computer program may be in the form of a standalone program, a computer program portion, or multiple computer programs, and may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, either as a standalone program or as modules, components, subroutines, or other units suitable for use in a data processing environment. A computer program may be deployed to run on a single module or on multiple modules distributed across one or more sites, and interconnected by a communication network.

[0078] The method steps of the present invention may be performed by one or more programmable processors that execute a computer program to perform the functions of the present invention by acting on input data and generating an output. The apparatus of the present invention may be implemented as programmed hardware or as a special-purpose logic circuit, including, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0079] Processors suitable for executing computer programs include, for example, both general-purpose and dedicated microprocessors, and any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory or random-access memory, or both. The essential element of a computer is a processor for executing instructions, coupled with one or more memory devices for storing instructions and data.

[0080] The above-described embodiments of the present invention may, advantageously, be used independently of any other embodiment of the embodiments, or in any feasible combination with one or more other embodiments of the embodiments.

[0081] Brief explanations of the technical terms used - Glossary FFNN = Feedforward Neural Network Fidelity = The faithfulness of the extracted logic program to the original model. This is the accuracy of the logic program when the target is the prediction of the original model instead of the correct label. ·

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0082] This disclosure includes the following inventions. (Note 1) A method implemented in a computer, Record the node activations for each node in the layers of the trained artificial neural network (ANN), and the ANN's predictions for each item of the training data used to train the ANN. The method involves taking the recorded node activations as input and the recorded ANN predictions as a target to create at least one decision tree, each decision tree being trained to approximate the ANN and optimize a defined criterion, and for each node of the decision tree, a threshold for the defined criterion is calculated to determine which node of the ANN the input activations should be split among the branches of the decision tree. Record the threshold associated with each node of the ANN, To obtain a threshold combination such that each threshold combination includes one of the thresholds obtained for each node of the ANN, For each of the threshold combinations, the selected rule extraction algorithm is executed using the threshold combination to extract at least one rule from the ANN to explain the output of the layer of the ANN. For each of the threshold combinations, the fidelity metric for at least one rule is obtained using the threshold combination, wherein the fidelity metric represents the accuracy of the rule regarding the prediction of the ANN. The objective is to determine which of the aforementioned threshold combinations yields the best fidelity metric, A method comprising: extracting at least one rule to explain the output of the layer of the ANN using the selected rule extraction algorithm with the threshold combination determined to yield the best fidelity metric. (Note 2) The method according to Appendix 1, wherein recording the thresholds includes ranking the thresholds for each node of the ANN according to their frequency of occurrence and average occurrence depth in the decision tree, and executing the selected rule extraction algorithm for each of the threshold combinations includes first executing the selected rule extraction algorithm for the threshold combination containing the threshold that occurs most frequently. (Note 3) The method according to Appendix 1 or 2, wherein obtaining the threshold combination includes one of obtaining all possible combinations of the thresholds, obtaining a combination for each node of the ANN using only a preset number of most frequently occurring thresholds, obtaining a combination for each node of the ANN using only a random subset of the thresholds, and obtaining a combination of only thresholds for each node of the ANN that satisfy a user-defined metric. (Note 4) The method according to Appendix 1, 2, or 3, wherein the defined criterion for optimization is entropy or the Gini coefficient. (Note 5) The method according to Appendix 1, 2, 3, or 4, wherein recording the threshold associated with each node of the ANN includes recording the average activation per sample of the node as a threshold for the node when no threshold associated with a particular node of the ANN exists. (Note 6) Creating at least one decision tree is any of the methods described in the preceding appendix, including creating multiple diverse decision trees using a random forest generation algorithm. (Note 7) Using any of the methods described above to extract at least one rule for an ANN for use in one of the autonomous driving algorithms and healthcare algorithms. (Note 8) (i) Extract at least one rule for a CNN used to control autonomous vehicles, (ii) Using any of the methods described in Appendix 1 to 6 to determine whether the ANN is functioning correctly using at least one of the extracted rules. (Note 9) A computer program having instructions, wherein, when executed by a computer, the instructions cause the computer to perform any of the methods described in Appendix 1 to 6. (Note 10) At least one computer processor, Connected to the at least one computer processor, the system includes a memory for storing node activations for each node in the layers of a trained artificial neural network (ANN), predictions of the ANN regarding each item of training data used to train the ANN, and instructions. The aforementioned instruction is given to the computer processor, The process involves taking the stored node activations as input and the stored ANN predictions as a target to create at least one decision tree, each decision tree being trained to approximate the ANN and optimize a defined criterion, and for each node of the decision tree, a threshold for the defined criterion is calculated to determine which node of the ANN the input activations should be split among the branches of the decision tree. The threshold associated with each node of the ANN is recorded, To obtain a threshold combination such that each threshold combination includes one of the thresholds obtained for each node of the ANN, For each of the threshold combinations, the selected rule extraction algorithm is executed using the threshold combination to extract at least one rule from the ANN to explain the output of the layer of the ANN. For each of the threshold combinations, the fidelity metric for at least one rule is obtained using the threshold combination, wherein the fidelity metric represents the accuracy of the rule regarding the prediction of the ANN. The objective is to determine which of the aforementioned threshold combinations yields the best fidelity metric, An apparatus that, using the threshold combination determined to yield the best fidelity metric, extracts at least one rule to explain the output of the layer of the ANN using the selected rule extraction algorithm. (Note 11) The apparatus according to Appendix 10, wherein recording the thresholds includes ranking the thresholds for each node of the ANN according to their frequency of occurrence and average occurrence depth in the decision tree, and executing the selected rule extraction algorithm for each of the threshold combinations includes first executing the selected rule extraction algorithm for the threshold combination containing the threshold that occurs most frequently. (Note 12) The apparatus according to Appendix 10 or 11, wherein obtaining the threshold combinations includes one of obtaining all possible combinations of the thresholds, obtaining combinations for each node of the ANN using only a preset number of most frequently occurring thresholds, obtaining combinations for each node of the ANN using only a random subset of the thresholds, and obtaining combinations of only the thresholds for each node of the ANN that satisfy a user-defined metric. (Note 13) The apparatus according to Appendix 10, 11, or 12, wherein the defined criterion for optimization is entropy or the Gini coefficient. (Note 14) The apparatus according to Appendix 10, 11, 12, or 13, wherein recording the threshold associated with each node of the ANN includes recording the average activation per sample of the node as a threshold for the node when no threshold associated with a particular node of the ANN exists. (Note 15) Creating at least one decision tree is an apparatus described in any of the appendices 10 to 14, which includes creating multiple diverse decision trees using a random forest generation algorithm.

Claims

1. A method implemented in a computer, Record the node activations for each node in the layers of the trained artificial neural network (ANN), and the ANN's predictions for each item of the training data used to train the ANN. The method involves creating at least one decision tree using the recorded node activations as input and the recorded ANN predictions as the target, wherein each decision tree is trained to approximate an ANN and optimize a defined criterion, and for each node of the decision tree, a threshold value for the defined criterion is calculated to determine which node of the ANN the input activations should be split among the branches of the decision tree. Record the threshold associated with each node of the ANN, The goal is to obtain a threshold combination such that each threshold combination includes one of the thresholds obtained for each node of the ANN, For each of the threshold combinations, the selected rule extraction algorithm is executed using the threshold combination to extract at least one rule from the ANN that explains the output of the layer of the ANN. For each of the threshold combinations, the fidelity metric of at least one rule is obtained using the threshold combination, wherein the fidelity metric represents the accuracy of the rule regarding the prediction of the ANN. The objective is to determine which of the aforementioned threshold combinations yields the best fidelity metric, A method comprising: extracting at least one rule to describe the output of the layer of the ANN using the selected rule extraction algorithm with the threshold combination determined to yield the best fidelity metric.

2. The method according to claim 1, wherein recording the thresholds includes ranking the thresholds for each node of the ANN according to their frequency of occurrence and average occurrence depth in the decision tree, and executing the selected rule extraction algorithm for each of the threshold combinations includes first executing the selected rule extraction algorithm for the threshold combination containing the threshold that occurs most frequently.

3. The method according to claim 1 or 2, wherein obtaining the threshold combination includes one of obtaining all possible combinations of the thresholds, obtaining a combination for each node of the ANN using only a preset number of most frequently occurring thresholds, obtaining a combination for each node of the ANN using only a random subset of the thresholds, and obtaining a combination of only the thresholds for each node of the ANN that satisfy a user-defined metric.

4. The method according to claim 1 or 2, wherein the defined criterion for optimization is entropy or the Gini coefficient.

5. The method according to claim 1 or 2, wherein recording the threshold associated with each node of the ANN includes recording the average activation per sample of the node as a threshold for the node when there is no threshold associated with a particular node of the ANN.

6. The method according to claim 1 or 2, wherein creating at least one decision tree comprises creating multiple diverse decision trees using a random forest generation algorithm.

7. A computer program having instructions, wherein, when executed by the computer, the instructions cause the computer to carry out the method described in claim 1 or 2.

8. At least one computer processor, Connected to at least one computer processor, and including a memory for storing node activations for each node in the layers of a trained artificial neural network (ANN), the ANN's predictions for each item of the training data used to train the ANN, and instructions, The aforementioned instruction is given to the computer processor, The process involves creating at least one decision tree, taking the stored node activations as input and the stored ANN predictions as a target, wherein each decision tree is trained to approximate an ANN and optimize a defined criterion, and for each node of the decision tree, a threshold value for the defined criterion is calculated to determine which node of the ANN the input activation should be split among the branches of the decision tree. The thresholds associated with each node of the ANN are recorded, The goal is to obtain a threshold combination such that each threshold combination includes one of the thresholds obtained for each node of the ANN, For each of the threshold combinations, the selected rule extraction algorithm is executed using the threshold combination to extract at least one rule from the ANN that explains the output of the layer of the ANN. For each of the threshold combinations, the fidelity metric of at least one rule is obtained using the threshold combination, wherein the fidelity metric represents the accuracy of the rule regarding the prediction of the ANN. The objective is to determine which of the aforementioned threshold combinations yields the best fidelity metric, An apparatus that, using the threshold combination determined to yield the best fidelity metric, extracts at least one rule to explain the output of the layer of the ANN using the selected rule extraction algorithm.

Images