Rule-based deep neural network explanation method, system, and media

Abstract

Rule-extraction based deep neural network interpretation methods, systems, and media. The interpretation of the deep neural network is based on extracting one or more rules that approximate the network's classification behavior. The rules are defined by identifying a set of hyperplanes in the data space that collectively define a convex polytope that separates the target class of input samples from different classes of input samples. Each rule corresponds to a set of decision boundaries between two different decision outcomes. Human understandable rule representations can be generated. The one or more rules can be used to generate a classifier. The representations and interpretations exhibit fidelity, stability, and comprehensiveness relative to other known methods.

Description

[0001] This patent application claims the benefit of priority to U.S. Patent Application No. 17 / 086,124, filed October 30, 2020, entitled “METHODS, SYSTEMS, AND MEDIA FOR DEEP NEURAL NETWORK INTERPRETATION VIA RULE EXTRACTION”, which is incorporated herein by reference in its entirety. Technical Field

[0002] This invention relates to artificial neural networks, including deep learning neural networks and their interpretation and representation. Background Technology

[0003] Artificial neural networks are computational structures used for predictive modeling. A neural network typically consists of multiple layers of neurons, each receiving input from the previous layer, applying a set of weights to the inputs, and combining these weighted inputs to generate an output, which in turn serves as input to one or more neurons in subsequent layers.

[0004] A layer of neurons uses filters to define the relationship between the outputs of neurons in the previous layer and the outputs of neurons in the current layer. A layer of a neural network receives data input, typically in the form of a data array of known dimensions: when the neural network operates on 2D data such as image data, the data input to a layer of the network is usually a 3D array consisting of a set of multiple 2D input activation maps (also called input feature maps or input channels). By applying a set of filters to a set of input activation maps, the layer generates data output, which is typically a data array of known dimensions; similarly, when the neural network operates on 2D data, the data output of a layer of the network is usually a 3D array consisting of a set of multiple 2D output activation maps (also called output feature maps or output channels).

[0005] Deep neural networks (DNNs) are a special type of neural network trained using deep learning techniques. Over the past decade, DNNs have achieved tremendous success in a wide range of tasks. However, despite their capabilities, DNNs are largely considered black-box models—their internal logic is often opaque or incomprehensible to humans. This lack of human-understandable explanations of DNNs makes it difficult to promote the use of artificial intelligence (AI) systems in risk-sensitive fields such as healthcare, finance, and security.

[0006] Therefore, it is necessary to communicate information about the internal decision-making nature of DNNs to human users, especially non-experts, to reassure them of the security and reliability of deploying DNNs for various critical applications. To convey meaningful information about the internal workings of DNNs, the model embodied by the DNN must be explained.

[0007] The meaning of "interpretation" in the relevant literature is very broad. In some cases, it may include generating or presenting indicators of factors that are understandable to humans, such as the reliability and / or transparency of a DNN. Reliability refers to the characteristics embodied in the DNN that suggest the model makes reliable decisions based on sound logic. Transparency is a means of building trust between the model and the user. Simply making the right decisions may not be enough to convince a user to use and rely on the model; the user may also need to know how to make decisions.

[0008] These factors (reliability and transparency) are crucial for certain areas where AI might be deployed. A first example is the field of medical diagnostics. Doctors often need to make diagnoses based on a patient's medical images. DNNs can achieve significant accuracy in this area, but recommending treatment or surgery could be risky if doctors cannot understand the DNN's reasoning. For example, doctors might want to know which part of the image triggered the decision; a DNN capable of identifying which part of the image is particularly relevant or prominent to their decision would allow doctors to assess the reasonableness of the DNN's reasoning. Doctors could also carefully examine the diagnosis by referring to the indicated parts of the image.

[0009] The second example is the field of autonomous vehicles. AI-driven self-driving cars may need to recognize objects they see in a road environment. For instance, when a car sees a stop sign, the user or developer must understand whether the car truly understands the "stop" instruction based on the letters displayed on the sign, or whether the car is making a decision based on memorized, unreliable features, such as the color or shape of the sign.

[0010] Various methods for interpreting DNNs have been proposed in the research literature. Gradient-weighted Class Activation Mapping (Grad-CAM) is one such method described by Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra in Int J Comput Vis 128, 336-359 (2020), titled "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization." Based on sample images provided as input to the neural network, a set of output feature maps from the penultimate layer of the neural network are multiplied by the average gradient of the output feature maps, and the results are summed to generate a heatmap. However, the gradients involved only apply to a small region in the input image, which may make the interpretation unreliable. Furthermore, Grad-CAM does not provide information about the model's decision-making logic; even if multiple factors lead to the decision, it only returns a single interpretation. Furthermore, it can only interpret one image at a time; therefore, the fidelity and stability of this interpretation are questionable.

[0011] The second approach, Local Interpretable Model-Agnostic Explanation (LIME), was described in 2016 in Knowledge Discovery and Data Mining (KDD) in “Why Should I Believe You?: Explaining the Predictions of Any Classifier” by Ribeiro, MT; Singh, S.; and Guestrin, C. In LIME, a sample image is segmented into multiple parts (called “superpixels”), and image classification performed by a neural network model is performed on a perturbed version of the sample image, where some superpixel image data has been removed. A loss function is minimized to compute an interpretable version of the neural network model. LIME has several drawbacks. The perturbed image cannot perfectly simulate a natural image, which may affect the credibility of the neural network model's confidence. LIME does not provide information about the decision logic of the neural network model because it is model-agnostic. Furthermore, the applicability of LIME explanations is actually quite limited, casting doubt on its fidelity. Due to the use of image segmentation and sampling, LIME can be unstable, an effect demonstrated in other techniques. Finally, like Grad-CAM, LIME returns only one explanation even when multiple factors lead to a decision.

[0012] The third approach, similar in some respects to LiIME, is called the anchor method, described by Ribeiro, MT; Singh, S.; and Guestrin, C. in their 2018 presentation at the AAAI Artificial Intelligence conference, "Anchors: High-precision model-agnostic explanations." In this approach, "anchors" are identified in the image using segmentation and perturbation methods similar to LiIME. In this case, an "anchor" refers to a set of superpixels, so the model's prediction remains unchanged regardless of what the other pixels in the image show. Using bottom-up structure or beam search to identify anchors provides the greatest coverage. The anchor method also has several drawbacks, shared with LiIME. The applicable images for anchors do not perfectly simulate natural images, which can affect the reliability of the model's confidence. The anchor method does not provide information about the model's decision logic because it is model-agnostic. The applicability of the anchor method's explanations can be very small, thus its realism is questionable. Due to the use of image segmentation, the anchor method is not stable. Finally, like LIME and Grad-CAM, even if multiple factors lead to the decision, only one explanation is returned.

[0013] The fourth method, called the meaningful perturbation method or masking method, was described by RCFong and A. Vedaldi in 2017 in arXiv preprint arXiv:1704.03296, "Interpretable explanations of black boxes by meaningful perturbation." A mask is used to apply a perturbation to a sample image. The mask is defined by a natural masking operation, such as a constant, noise, or blur. An objective function is minimized via gradient descent. The objective function is a weighted sum of multiple image pixels, the sum of confidence scores after applying the mask, and the total variation norm of the mask. The optimized mask should cover the image pixels most relevant to the model's decision. The masking method also exhibits some drawbacks. The masked image cannot perfectly simulate a natural image, which may affect the reliability of the model's confidence scores. The masking method does not provide information about the logic of the model's decision. The masking method can only interpret one image at a time, which raises questions about the fidelity and stability of this interpretation. Finally, like the other methods described above, the masking method returns only one interpretation even if multiple factors lead to the decision.

[0014] Therefore, there is a need for a technique to interpret the decisions of deep neural networks that overcomes one or more of the shortcomings of the existing methods mentioned above. Summary of the Invention

[0015] In various examples, this invention describes methods, systems, and computer-readable media for interpreting deep neural networks based on rule extraction. The interpretation of the deep neural network is based on the extraction of one or more rules, wherein the deep neural network makes decisions, each rule corresponding to a set of decision boundaries between two distinct decision outcomes. In some examples, human-understandable representations of one or more rules can be generated, helping human users to view or otherwise understand the nature of the rules applied to a given input sample. In some examples, one or more rules can be used to generate a classifier. Experimental data comparing the exemplary embodiments described herein with other known methods of interpreting deep neural networks demonstrate that the currently disclosed embodiments exhibit fidelity, stability, and comprehensiveness relative to other known methods. Furthermore, some embodiments provide interpretation methods that can be used before the neural network is fully trained (e.g., when the neural network is fine-tuned).

[0016] As used herein, the term "model" can refer to an optical or computational model. A model can be said to be implemented, embodied, run, or executed by an algorithm, computer program, or computational structure or device. In this exemplary embodiment, a model may refer to a predictive model designed to model human perception or interpretation of inputs such as images, and may be implemented using algorithms trained using deep learning techniques (e.g., deep neural networks (DNNs)). The terms DNN, deep neural network, neural network, artificial neural network, and network are used interchangeably herein unless otherwise stated.

[0017] Activation maps or feature maps can refer to the inputs (e.g., "input activation maps") or outputs (e.g., "output activation maps") of layers in an artificial neural network. A layer of a neural network can receive multiple input activation maps as inputs and can produce multiple output activation maps as outputs.

[0018] "Input sample" can refer to any data sample used as input to a neural network, such as image data. It can refer to training data samples used to train the neural network, or to data samples provided to a trained neural network that infer (i.e., predict) the output based on the data samples from the task performed by the trained neural network. Therefore, for a neural network performing an image classification task, the input sample can be a single digit image.

[0019] As used herein, "updated input sample" refers to a set of data generated by a trained neural network or a portion thereof based on received input samples. In some examples, it may refer to a set of output activation maps generated by layers of a trained DNN based on a single input sample provided as input to the DNN. In some examples, as further described below with reference to Figure 6, the updated sample may be an input sample.

[0020] As used herein, a statement that an element is “used for” a particular purpose may mean that the element performs a particular function or is used to perform one or more particular steps or operations, as described herein.

[0021] As used herein, the statement that the second element is "based on" the first element can mean that the features of the second element are at least partially influenced by or determined by the features of the first element. The first element can be considered as the input of an operation or computation or a series of operations or computations that produce the second element as an output associated with the first element.

[0022] As used herein, a “representation,” “simplified representation,” “interpretation,” or “human-understandable representation” of a neural network, model, or decision boundary can refer to a human-perceptible (e.g., visual) representation of data or a simplified data structure that approximates or corresponds to a more complex structure or behavior of a neural network, model, or decision boundary, or represents a decision boundary. A “representation” or “simplified representation” of a neural network, model, or decision boundary can also refer to a simplified model or classifier that approximates the behavior of a more complex neural network, model, or decision boundary. By generating simplified representations of models (e.g., inference models implemented by deep neural networks), the embodiments described herein can thereby generate simplified classifiers that approximate the behavior of the represented more complex model, and / or generate human-understandable representations of the represented model. Unless otherwise stated, “DNN representation” refers to generating rules or classifiers that approximate the behavior of said DNN, while “DNN interpretation” refers to generating a human-understandable representation of said DNN behavior.

[0023] As used herein, the terms “classification” and “categorization” are used interchangeably and synonymously (e.g., “classify and categorize”, “classifier and categorizer”, etc.). In some examples, the behavior of a classification model can be described as classifying an input sample into a first category or a second category; it should be understood that both the first and second categories can include multiple categories. For example, the first category could be “dog”, while the second category could include multiple categories (e.g., “cat”, “person”, and “truck”, none of which are “dog”, so the second category could be considered “non-dog”).

[0024] In some aspects, the present invention provides a method for generating a simplified representation of a neural network trained to perform image classification on input samples. The neural network includes a truncated portion and a tail portion, wherein the truncated portion includes one or more layers, and the tail portion includes one or more layers. The method includes the step of using the truncated portion to generate updated input samples, the input samples including one or more output activation maps based on the input samples. The method includes the step of generating labels using the tail portion, wherein the labels classify the input samples into a first category or a second category according to the one or more output activation maps. The method includes the step of generating rules comprising a convex polyhedron defined by a plurality of hyperplanes in a multidimensional Euclidean space defined according to the plurality of updated samples, such that updated input samples within the convex polyhedron are likely to be classified into a first category by the trained neural network, and updated input samples outside the convex polyhedron are likely to be classified into a second category by the trained neural network. In some aspects, the present invention describes a system. The system includes a processing system. The processing system includes one or more processor devices and a memory storing instructions. When these instructions are executed by the processing system, the system performs multiple steps. The system generates a simplified representation of a neural network trained to perform image classification on input samples. The trained neural network includes a truncated portion and a tail portion, wherein the truncated portion includes one or more layers, and the tail portion includes one or more layers. The simplified representation of the trained neural network is generated by the following steps: using the truncated portion to generate updated input samples, the updated input samples including one or more output activation maps based on the input samples. The simplified representation of the trained neural network is also generated by the following steps: using the tail portion to generate labels, wherein the labels classify the input samples into a first category or a second category based on the one or more output activation maps. The simplified representation of the trained neural network is further generated by the following steps: generating rules comprising a convex polyhedron defined by multiple hyperplanes in a multidimensional Euclidean space defined according to the multiple updated samples, such that updated input samples within the convex polyhedron are likely to be classified into a first category by the trained neural network, and updated input samples outside the convex polyhedron are likely to be classified into a second category by the trained neural network.

[0025] In some aspects of the method, the plurality of input samples includes seed samples classified into a first category by the trained neural network. The truncated portion generates updated seed samples based on the seed samples, and the number of comparison input samples is equal to a predetermined candidate pool size. Each comparison input sample is classified into a second category by the trained neural network. The truncated portion generates updated comparison samples based on each comparison input sample. For each updated comparison sample, generating the rule includes performing a binary search between the updated seed samples and the updated comparison samples to generate base values ​​and bias values. For each updated comparison sample, generating the rule further includes calculating a decision boundary between the first category and the second category based on the base values ​​and bias values ​​from the binary search, wherein the decision boundary defines one of the plurality of hyperplanes. The decision boundary defines one of the plurality of hyperplanes.

[0026] In some aspects of the method, the plurality of input samples further includes one or more positive samples and one or more negative samples. Each positive sample is classified into a first category by the trained neural network. The truncated portion generates an updated positive sample based on each positive sample. Each negative sample is classified into a second category by the trained neural network. The truncated portion generates an updated negative sample based on each negative sample. For each hyperplane, the generation rule further includes generating a positive sample binary matrix. For each updated positive sample, the positive sample binary matrix includes a row of binary values. Each binary value indicates whether the updated positive sample is on a first or second side of one of the plurality of hyperplanes. Each column of the positive sample binary matrix corresponds to a hyperplane. For each updated negative sample, generating a negative sample binary matrix includes a row of binary values. Each binary value indicates whether the updated negative sample is on a first or second side of one of the plurality of hyperplanes. Each column of the negative sample binary matrix corresponds to a hyperplane. Generating the rule further includes: using submodulus cost submodulus coverage optimization to select the plurality of hyperplanes from the columns of the positive sample binary matrix and the columns of the negative sample binary matrix, such that the number of hyperplanes in the plurality of hyperplanes of the convex polyhedron defining the rule is less than the predetermined candidate pool size.

[0027] In some aspects of the method, the positive sample binary matrix further includes a row of binary values ​​corresponding to the updated seed sample, and the negative sample binary matrix further includes a row of binary values ​​corresponding to each updated comparison sample.

[0028] In some aspects of the method, the plurality of input samples includes explanatory samples classified into the first category by the trained neural network. The truncated portion generates updated explanatory samples based on the explanatory samples. The method further includes generating an explanation of the rule. The rule is generated by identifying one of the plurality of hyperplanes of the rule. The rule is also generated by the steps of: for each activation mapping in one or more activation mappings of the updated explanatory samples, each activation mapping including a plurality of activation values, calculating the average of the partial derivatives of the hyperplane with respect to each activation value of the activation mapping; setting the weights of the activation mappings equal to the calculated average. The rule is also generated by the steps of: generating a heatmap by calculating the sum of each activation mapping weighted by its respective weights, the heatmap representing the decision boundary corresponding to the hyperplane.

[0029] In some aspects of the method, each input sample is an image. The method also includes generating a human-understandable representation of the rule by combining the heatmap with the interpretation sample.

[0030] In some aspects of the method, combining the heatmap with the explanatory sample includes generating an image that includes at least a portion of the explanatory sample, wherein the heat value of a visual indicator of a portion of the heatmap is above a significant threshold.

[0031] In some aspects of the method, generating a representation of the rule further includes repeating the following steps: identifying hyperplanes; calculating the average value of each activation map; setting the weight of each activation map; and generating a heatmap for each of the remaining hyperplanes of the convex polyhedron of the rule.

[0032] In some aspects of the method, updated input samples and generated labels are generated for each of the first plurality of input samples to generate the first plurality of updated samples and the first plurality of labels. The generation rules are executed based on the first plurality of updated samples and the first plurality of labels to generate a first rule. Updated samples and generated labels are generated for each of one or more additional plurality of input samples to generate one or more additional plurality of updated samples and one or more additional plurality of labels. The generation rules are executed based on one or more additional plurality of updated samples and one or more additional plurality of labels to generate one or more additional rules, the first rule and the one or more additional rules collectively constituting a plurality of rules. The method further includes generating a multi-rule classifier comprising the plurality of rules.

[0033] In some aspects, the present invention provides a computer-readable medium having instructions tangibly stored thereon, which, when executed by a processing system, cause the processing system to perform the above-described method steps. Attached Figure Description

[0034] The accompanying drawings, by way of example, illustrate exemplary embodiments of this application, wherein:

[0035] Figure 1A This is a block diagram of an exemplary processing system that can be used to implement the examples described herein;

[0036] Figure 1B A schematic diagram of an exemplary architecture for a deep neural network (DNN);

[0037] Figure 1C yes Figure 1B A schematic diagram of the convolutional layer of a DNN shows the dimensions of the input data array, the output data array, and a set of convolutional filters applied by the convolutional layer;

[0038] Figure 2 This is a simplified representation of the rules extracted from a deep neural network provided in an embodiment of the present invention;

[0039] Figure 3 The present invention provides a set of human-understandable representations of three decision boundaries of rules extracted from a deep neural network trained to classify images as displaying cats or dogs, which are applied to four separate explanatory samples.

[0040] Figure 4 This is a further simplified representation of the human-understandable representation of the three decision boundaries of the rules extracted from a deep neural network trained to classify images as displaying healthy or diseased retinas, provided by the present invention.

[0041] Figure 5 This is a block diagram of an exemplary deep neural network provided by the present invention. The exemplary deep neural network includes a truncated portion for generating updated input samples and a tail portion for perceiving updated input samples to generate inferences.

[0042] Figure 6A This is a flowchart of an exemplary single-rule extraction method provided by the present invention for generating a simplified representation of a deep neural network including rules;

[0043] Figure 6B yes Figure 6A The flowchart of the adjusted binary search steps of the method;

[0044] Figure 6C It is used for optimization Figure 6AA flowchart illustrating an exemplary method for using binary matrices;

[0045] Figure 6D Is using Figure 6A A flowchart illustrating an exemplary multi-rule extraction method that generates multiple rules from a single-rule extraction method;

[0046] Figure 7A This invention provides a two-dimensional visualization of the data space of input samples, including seed samples, displayed by category;

[0047] Figure 7B This invention provides Figure 7A The two-dimensional visualization shows multiple decision boundaries between input sample categories;

[0048] Figure 7C This is the representation provided by the present invention. Figure 7B A pair of binary matrices representing multiple decision boundaries;

[0049] Figure 7D This invention provides Figure 7A of Figure 7B Two-dimensional virtualization of rules defined by subsets of multiple decision boundaries;

[0050] Figure 8 This is a flowchart of an exemplary method for generating an interpretation of one or more rules generated by the method in Figure 6.

[0051] Similar reference numerals were used in different accompanying drawings to indicate similar components. Detailed Implementation

[0052] A method, system, and computer-readable medium for interpreting deep neural networks based on rule extraction will now be described with reference to exemplary embodiments. The interpretation of a deep neural network is based on extracting one or more rules, whereby the network makes decisions, each rule corresponding to a set of decision boundaries between two distinct decision outcomes. In some examples, human-understandable representations of one or more rules can be generated, helping human users to view or otherwise understand the properties of the rules applied to a given input sample. In some examples, one or more rules can be used to generate a classifier.

[0053] Some of the embodiments described herein exhibit fidelity, stability, and comprehensiveness relative to other known methods. Fidelity means that the neural network's interpretation accurately describes the model's local behavior and has the potential to combine several local interpretations to form a global representation of the model. The example embodiments described herein can be based on the model's decision boundaries, thus accurately describing the model's behavior. Stability means that the interpretation is stable to small changes in input samples, model parameters, interpretation hyperparameters, and adversarial attacks. The exemplary embodiments described herein can be stable to changes that have little impact on the model's decisions, such as mild transformations in the input image, fine-tuning of model parameters, interpretation hyperparameters, and even adversarial attacks. Comprehensiveness means that the interpretation identifies all the major factors that contribute to the model's decision-making. The exemplary embodiments described herein can consider multiple decision boundaries and interpret all of these boundaries.

[0054] Some embodiments provide interpretation methods that can be used before the neural network is fully trained (e.g., when the neural network is fine-tuned). Existing DNN interpretation methods typically require the weight values ​​of the DNN to converge (i.e., the value of the loss function applied during training is small) before a meaningful interpretation can be generated. However, some embodiments described herein can allow the generation of a representation or interpretation of the DNN while the DNN is still being fine-tuned and the weight values ​​have not yet converged.

[0055] The exemplary embodiments described herein may include rule extraction methods that generate rules corresponding to a relatively small number of decision boundaries within a possible input space. Some embodiments may use multi-rule extraction methods to supplement a first rule with one or more additional rules to further define the decision boundaries within the possible input space. Some embodiments may use model interpretation methods to generate human-understandable representations of one or more decision boundaries of one or more rules, thereby enabling human users to understand the basis of the neural network's decisions made with respect to a given input sample.

[0056] This paper describes exemplary embodiments with reference to deep neural networks (DNNs) trained using supervised learning for specific perturbation tasks. After training, the trained DNN is used to perform specific inference tasks, such as object classification using image data as input. However, it should be understood that the techniques described herein may be applicable to training various artificial neural networks to perform various inference tasks using supervised or unsupervised learning with various input sample types.

[0057] Now refer to Figures 1A to 1C Describe the context in which the exemplary embodiments can be operated.

[0058] Exemplary processing system

[0059] First, describe the systems or devices that can be used in the examples disclosed herein, such as computing systems.

[0060] Figure 1A For illustrative and simplified purposes, the block diagram of computing system 100 may be a device for training a convolutional neural network or executing instructions of a trained deep neural network to perform a specific task (e.g., image classification, object recognition, etc.) that the deep neural network has been trained on, according to the examples disclosed herein. Other computing systems suitable for implementing the embodiments described herein may be used, which may include components different from those described below. In some examples, the computing system may be implemented across multiple physical hardware units, such as in parallel computing, distributed computing, virtual servers, or cloud computing configurations. Although Figure 1A A single instance of each component is shown, but multiple instances of each component may exist in the computing system 100.

[0061] The computing system 100 may include one or more processing units 102, such as a central processing unit (CPU) with hardware accelerators, a graphics processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, a dedicated artificial intelligence processor unit, or a combination thereof.

[0062] The computing system 100 may also include one or more optional input / output (I / O) interfaces 104, which may support connection to one or more optional input devices 114 and / or optional output devices 116. In the example shown, one or more input devices 114 (e.g., keyboard, mouse, microphone, touchscreen, and / or keypad) and one or more output devices 116 (e.g., display, speaker, and / or printer) are shown as optional and external to the computing system 100. In other examples, one or more of the one or more input devices 114 and / or one or more output devices 116 may be included as components of the computing system 100. In other examples, there may be no one or more input devices 114 and one or more output devices 116, in which case one or more I / O interfaces 104 may not be required.

[0063] The computing system 100 may include one or more optional network interfaces 106 for wired or wireless communication with a network (e.g., an intranet, the Internet, a P2P network, a WAN, and / or a LAN) or other nodes. The one or more network interfaces 106 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for intra-network and / or inter-network communication.

[0064] The computing system 100 may further include one or more storage units 108, wherein the one or more storage units 108 may include mass storage units such as solid-state drives, hard disk drives, disk drives, and / or optical disk drives. The processing system 100 may include one or more memories 110, wherein the one or more memories 110 may include volatile or non-volatile memories (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The one or more non-transient memories 110 may store instructions executed by the one or more processing devices 102, for example, to perform the examples described herein. The one or more memories 110 may include other software instructions, such as software instructions for implementing operating systems and other applications / functions. In some examples, the memories 110 may include software instructions executed by the processing device 102 to train neural networks and / or implement trained neural networks, as disclosed herein.

[0065] In some other examples, one or more datasets and / or modules may be provided by external memory (e.g., an external drive that communicates with computing system 100 via wired or wireless communication) or by transient or non-transient computer-readable media. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable storage devices.

[0066] A bus 112 may exist to provide communication between components of the computing system 100, including one or more processing devices 102, one or more optional I / O interfaces 104, one or more optional network interfaces 106, one or more storage units 108, and / or one or more memories 110. The bus 112 may be any suitable bus architecture, including, for example, a memory bus, a peripheral bus, or a video bus.

[0067] Deep Neural Network Examples

[0068] Figure 1B An example architecture of a deep neural network (DNN) 120 is shown. The DNN 120 in this example is constructed and arranged to perform specific tasks during training and inference, such as object detection, object recognition and classification (i.e., object localization and classification), audio recognition and classification, text recognition and classification, etc. The DNN 120 has been simplified and is not intended to be limiting, and is for illustrative purposes only. For example, input data to the DNN 120 can be image data representing digital images, audio data representing audio files, or text data (i.e., embeddings representing words or sentences). A single sample of the input data (e.g., a single image) may be referred to herein as an "input sample". Optional elements are shown with dashed lines.

[0069] In this example, DNN 120 is a convolutional neural network (CNN). It includes one or more convolutional blocks 124 for performing feature learning, and fully connected blocks 132 for performing specific tasks. DNN 120 may also optionally include a preprocessing block 122, which can perform various operations, such as normalization, to prepare the input data for the convolutional blocks 124.

[0070] For example, Figure 1B The convolutional block 124 shown may include convolutional layers 126, combined nonlinear and rectified layers (e.g., ReLU layers 128), and optional pooling layers 130. The output from each layer in the convolutional block 124 is used as the input to the next layer in the convolutional block 124. In other words, the output of convolutional layer 126 can be used as the input to a subsequent ReLU layer 128, the output of ReLU layer 128 can be used as the input to a pooling layer 130, or it can be used as the input to the convolutional layer 126 of another convolutional block 124 to continue the convolutional operation.

[0071] The following is combined Figure 1C The internal workings of convolutional layer 126 are described in detail. In summary, convolutional layer 126 performs convolution operations on its input activation maps to generate an output consisting of output activation maps generated from the input activation maps.

[0072] Since the number of parameters (e.g., weights) of the filters in convolutional layer 126 typically needs to be reduced, pooling layer 130 can be placed after convolutional layer 126 in convolutional block 124. During image processing, the purpose of pooling layer 130 is to reduce the size of the output activation map generated by ReLU layer 128. Pooling layer 130 may include average pooling and / or max pooling operators to sample the input activation map, resulting in a relatively small output activation map. Average pooling can calculate pixel values ​​within a specific range of activation maps to generate an average value as the average pooling result. Max pooling can obtain the pixel with the largest value within a specific range as the max pooling result. The size of the activation map output after pooling layer processing can be smaller than the size of the activation map input to the pooling layer. Each pixel in the activation map output by the pooling layer indicates the average or maximum value of the sub-region corresponding to the activation map input to the pooling layer.

[0073] After the input data is processed by convolutional block 124, DNN 120 still cannot output the required output information. In order to generate the final output information (e.g., object category (i.e., class), bounding box, etc.), fully connected block 132 of CNN 120 generates the output from the output of convolutional block 124. The output of convolutional block 124 is the output activation map as described above.

[0074] Fully connected block 132 includes an input layer 134, one or more hidden layers 136, and an output layer 138. The output layer 138 follows the last hidden layer 136 of the fully connected block 132. In other words, the output layer 138 is the last layer in the entire DNN 120. During the training of DNN 120, training data and a supervised learning algorithm are used to train DNN 120. During forward propagation (… Figure 2 After the forward propagation (from 124 to 138) is completed, a loss function similar to the category (i.e., class) cross-entropy is used to calculate the prediction error of the DNN 120, and backpropagation is performed (in... Figure 2 The propagation in the direction from 138 to 124 is backpropagation) to update the parameters (e.g., weights) of layers 128, 130, 132, 134, and 136 of DNN 120, the weights of filters in DNN 120 based on the computational prediction error, in order to reduce the prediction error between the ideal result (i.e., the ground truth in the training data) and the prediction result output by the output layer 138.

[0075] In the DNN 120 trained to perform a classification task, the last hidden layer 136 of the fully connected block 132 can be a logits layer immediately preceding the output layer 138. The logits layer generates a set of logits corresponding to the probability distribution of the classification task, indicating the inferred (i.e., predicted) probability of each of the multiple possible classes into which the input sample can be classified. The output layer 138 can apply a Softmax function to the logits generated by the logits layer to normalize the logits to a probability distribution from 0 to 1. Thus, the Softmax function generates the output of the DNN 120 consisting of two or more classes, each with an associated classification probability associated with its corresponding input sample. Therefore, in response to an input sample consisting of photos of dogs, the DNN 120 trained to classify images into the categories of "dog" and "cat" can produce an output at its output layer 138 consisting of classification probabilities ("dog", 0.993 | "cat", 0.007).

[0076] It should be noted that, Figure 1B The DNN 120 shown is only used as an exemplary architecture for deep neural networks. In practical applications, the architecture of the DNN 120 can exist in different forms.

[0077] The above discussion provides an example illustrating how a trained DNN can be used to perform a specific task during inference. Generally, input data can have one, two, or three (or more) dimensions, and the output can have any suitable format, depending on the task. The exemplary embodiments described herein are to be described in the context of a DNN used to perform a specific task, such as a classification task, including the classification of images or objects in natural language processing (NLP). In the exemplary embodiments described herein, the input to DNN 120 is an image that can be preprocessed by an input layer and fed to the input layer, receiving an input activation map and generating an output activation map in the form of a multi-channel 2D pixel array (i.e., a 3D array defined by pixel height, pixel width, and channel depth). However, it should be understood that in some embodiments, data arrays of different dimensions can be used as input or output, such as a multi-channel 1D array for tasks involving audio or text input.

[0078] Examples of convolutional layers and activation maps

[0079] Figure 1C The convolutional layer 126 is shown, along with the dimensions of the input data array 144, the output data array 148, and a set of convolutional filters 146 applied by the convolutional layer 126. The input data array 142 is shown herein as having a value equal to C. inA multi-channel activation map set with multiple input channels (i.e., activation maps). Each channel of the input data array 144 consists of a 2D array, such as an image composed of a 2D pixel array, with a height H. in and width W in Therefore, the number of values ​​stored in the input data array 142 is equal to (H in ×W in ×C in The convolutional filters 146 applied to the input data array 144 each have a height h, a width w, and a channel depth C. in Convolutional layer 126 uses the value C. out Multiple convolutional filters 146.

[0080] Convolutional layer 126 applies convolutional filters 146 to the input data array 144 in a series of convolutional operations. Each convolutional filter 146 applied to the input data array 144 generates channels of the output data array 148, shown here as having a value equal to C. out A multi-channel activation map set with multiple output channels (i.e., output activation maps). Each channel of the output data array 148 consists of a 2D array, such as an image composed of a 2D pixel array, with a height H. out and width W out H in and H out Between and W in and W out The relationship between them is determined by the kernel dimensions h and w, as well as the stride, padding, and other convolution configuration values ​​or hyperparameters used in the convolutional operations of convolutional layer 126. In some embodiments, H in =H out W in =W out For example, an exemplary embodiment can use kernel dimensions h=3 and w=3, padded with 1 pixel and stride 1, to generate an output data array, where H in =H out W in =W out For example, in embodiments using hardware or software components optimized to handle input channels with a fixed dimension, H is used. in =H out and W in =W out Convolutional layer 126 can exhibit certain advantages.

[0081] Now refer to Figures 2 to 8 This describes exemplary embodiments that provide DNN representations and interpretations.

[0082] Example representation of DNN decision boundary

[0083] Figure 2 This is a simplified representation of the rules extracted from a deep neural network (DNN). In this example, the DNN has been trained and receives input samples 210 in the form of images. Figure 2 Each input sample 210 shown is an image of a dog or a cat. The DNN in this example has been trained to classify the input sample into either a first category "dog" or a second category "cat" (or, in some examples, "not a dog"). Therefore, the trained DNN, in response to receiving a given input sample, will generate a label (i.e., the label "dog" or the label "cat") as output. The mapping between the input samples and the output labels can here be referred to as the "classification behavior" of the trained DNN. Any function, algorithm, or process that replicates the mapping between the input and output spaces of a trained DNN can be said to replicate the classification behavior of the trained DNN. Similarly, if a function, algorithm, or process results in a mapping between the inputs and outputs that approximates but does not completely replicate the mapping of the trained DNN (i.e., most or many input samples map to the same output labels that the DNN will generate), it can be said to approximate the classification behavior of the trained DNN. As used herein, the term "behavior," applied to a DNN or another machine learning model, refers to its classification behavior.

[0084] For clarity and visibility, Figure 2 While shown as a two-dimensional space, the actual data space defined by the input sample 210 is multi-dimensional. In some embodiments, the number of actual dimensions of the data space defined by the input sample may be equal to the number of pixels multiplied by the number of channels per input sample.

[0085] In this example, the classification behavior of the trained DNN causes the input sample to be classified as either "dog" or "cat," as shown by the DNN decision boundary 202: when given an input sample 210, if the input sample is within the DNN decision boundary 202, the DNN will output a predicted label for the first category, "dog," or if the input sample is outside the DNN decision boundary 202, the trained DNN will output a predicted label for the second category, "cat." The exemplary embodiments described herein can generate a simplified representation of the DNN decision boundary 202, such as a single rule 204, which is shown as a triangle defined by three decision boundaries 241, 242, and 243, and is shown as a straight line. However, the single rule 204 can actually be a convex polyhedron in a multidimensional data space, where each decision boundary 241, 242, and 243 passes through a hyperplane of the data space. References below... Figure 6A and Figure 6B as well as Figures 7A to 7D Describes the generation of a single rule.

[0086] It should be understood that a single rule 204 may not precisely replicate every classification decision made by the DNN. For example, input sample 220 displaying a picture of a dog is outside of a single rule 204, but within the DNN's decision boundary 202. In some embodiments, a single rule 204 is generated such that it closely approximates the classification behavior of the DNN while using a small number of decision boundaries 241, 242, 243. In some embodiments, the behavior of the DNN can be more closely matched by generating one or more additional rules in addition to the single rule 204. Figure 2 The diagram shows several additional rules 230, where the input sample within each additional rule 230 is classified into the second category, "cat". The additional rules are also shown in the reference below. Figure 6A and Figure 6B as well as Figures 7A to 7D The described method is generated. Although Figure 2 The additional rules 230 in the document all include input samples of the second category "cat", but additional rules generated by the method described herein can be defined to include input samples of the first category, the second category, or any other category or set of categories.

[0087] It should be understood that the number of decision boundaries defining a single rule 204 or any additional rule 230 is not limited to three. Some embodiments may use a greater number of hyperplanes to define the convex polyhedra of the rules, and in some embodiments, different rules generated by the multi-rule extraction method may each have a different number of decision boundaries. For example, Figure 2 The additional rule 232 shown has four decision boundaries instead of three.

[0088] Based on the following reference Figures 6A to 6B as well as Figures 7A to 7D The described method (a process that may be called "DNN representation") extracts one or more rules from a DNN, which can then be used as a simplified model or classifier that approximates the reasoning behavior of the DNN. Furthermore, or alternatively, the rules can be used to generate human-understandable representations of one or more decision boundaries of one or more rules (a process that may be called "DNN interpretation"). References will now be made to... Figure 3 and Figure 4 Describe an exemplary DNN explanation.

[0089] An exemplary explanation of the decision boundary of a DNN

[0090] Figure 3 Human-understandable representations of three decision boundaries are shown from rules extracted from a deep neural network trained to classify images as either cats or dogs. Figure 3Each of the four rows 322, 324, 326, and 328 shows the input sample (on the left), followed by a human-understandable representation of each of the three decision boundaries of the rule applied to the input sample. The input samples in each row 322, 324, 326, and 328 are referred to here as "interpretation samples," meaning the input samples provided as input to the DNN to generate one or more human-understandable interpretations of the rules extracted from the DNN. Referring to the first row 322, the first interpretation sample 302 is shown as a photograph of a dog. See Figures 6 to 7 below. Figure 7D The described method is used to generate at least one rule, for example Figure 2 The single rule 204 is then used. A first interpretive sample 302 is then used to generate one or more human-understandable representations of the decision boundaries of the rule extracted from the DNN (shown here as three human-understandable representations 304, 306, and 308). Each human-understandable representation 304, 306, and 308 corresponds to one of the three decision boundaries of the rule (e.g., the first human-understandable representation 304 could correspond to the decision boundary 241 of the single rule 204 visualized using the first interpretive sample 302). Each of the other rows 324, 326, and 328 also displays an interpretive sample (in each case, a picture of a dog) on ​​the left, followed by three human-understandable representations of the decision boundaries of the same rule used in the first row 322, as visualized using the representation samples of the corresponding rows. Figure 3 The 4×4 image matrix shown can be called a heatmap matrix. The following section will combine... Figure 8 Further description.

[0091] In this example, in each line 322, 324, 326, 328, the corresponding explanatory sample (e.g., 302) is compared with the following... Figure 8 The heatmaps generated by the DNN interpretation method are combined to generate each human-understandable representation (e.g., 304, 306, or 308) of the decision boundaries of the rules extracted from the DNN. The heatmaps are combined with their corresponding interpretation samples (e.g., 302) to produce human-understandable representations showing that the original interpretation sample 302 has been modulated, perturbed, overlaid, superimposed, or otherwise combined with its corresponding heatmap. For example, the heatmap may recolor the interpretation sample 302 such that pixel regions with high heat levels defined by the heatmap are red-shifted, while pixel regions with low heat levels defined by the heatmap are blue-shifted or remain unmodified. In some embodiments, the heatmap may recolor or color-shift multiple pixel regions using multiple colors in the spectrum, such as a first color (e.g., red) for very high heat regions, a second color (e.g., orange) for high heat regions, and a third color (e.g., yellow) for medium heat regions, while keeping low heat regions unmodified. In some embodiments, consecutive color values ​​may be used to represent consecutive heatmap values.

[0092] The following is combined Figure 8 The generation of the heatmap is described in detail. In summary, the heatmap indicates pixel regions of the input sample (i.e., explanatory sample 302) that are highly correlated or obvious with a given decision boundary of the rules extracted from the DNN. This can mean that, in some examples, changing the pixel values ​​of pixels in highly correlated regions of the input sample (e.g., the first explanatory sample 302) relative to changes made to low-correlation pixel regions of the input sample (e.g., the first explanatory sample 302) will likely lead to a change in the DNN classification on the other side of the decision boundary for the input sample (e.g., the modified first explanatory sample 302).

[0093] exist Figure 3 In each row 322, 324, 326, 328, each human-understandable representation (e.g., 304, 306, or 308) of the decision boundary of the rule extracted from the DNN is shown as an explanatory sample with the corresponding row (e.g., 302) of the superimposed heatmap corresponding to the respective decision boundary (e.g., decision boundary 241 for human-understandable representation 304, decision boundary 242 for human-understandable representation 306, and decision boundary 243 for human-understandable representation 308). For Figure 3 The other rows 324, 326, and 328 repeat this correspondence between the decision boundary and the human-understandable representation from left to right. Although the three human-understandable representations 304, 306, and 308 are shown as heatmaps with similar shapes to their counterparts in the other rows 324, 326, and 328, this may not be the case for some DNNs and / or some sets of interpreted samples.

[0094] Each human-understandable heatmap includes a very highly relevant pixel region 310 that can be recolored according to a first color, a highly relevant pixel region 312 that can be recolored according to a second color, and a moderately relevant pixel region 314 that can be recolored according to a third color. The remainder of the image can be considered as unrecolored, low-relevance regions. It should be understood that different embodiments may modify the image in different ways to visually represent the heatmap values ​​of each pixel or each pixel region.

[0095] By presenting one or more human-understandable representations (e.g., 304, 306, and 308) to a human user in association with one or more input samples (e.g., interpretation sample 302), various embodiments can help the user interpret the behavior of the DNN to confirm that the DNN has met one or more of the aforementioned criteria (e.g., fidelity, stability, and comprehensiveness).

[0096] Now refer to Figure 4 Alternative embodiments of DNN interpretation are described in the context of medical imaging.

[0097] Figure 4 Two different types of human-understandable representations of decision edges for rules extracted from a deep neural network trained to classify images as real healthy or diseased retinas are shown. On the left, interpretation sample 402 shows an image of a portion of the human retina (e.g., a CAT scan, X-ray, or MRI image). Interpretation sample 402 includes a region of interest 408 that shows potential markers of retinal damage or disease.

[0098] According to the reference below Figure 8 The described technique provides interpretive samples 402 to a DNN trained to classify retinal images as displaying healthy or diseased retinas. (See reference below.) Figure 8 The described method 800 generates a first human-understandable representation 404, as shown in the reference above. Figure 3 The image described is an interpretation sample 402 of the heatmap combination. The heatmap uses the aforementioned visual representation techniques to identify highly relevant pixel regions 410 and moderately relevant pixel regions 412 of the image. The highly relevant pixel regions 410 and moderately relevant pixel regions 412 can jointly correspond to the region of interest 408.

[0099] In another embodiment, a second human-understandable representation 406 may be generated, showing a bounding box 420 of pixels surrounding the region of interest 408. In some embodiments, according to the following references Figure 8 The described method 800 generates a second human-understandable representation 406, after which a bounding box 420 is generated by the following steps: identifying pixel locations in the heatmap that are above a heat threshold, and overlaying the bounding box 420 on the image, the bounding box restricting the pixel locations above the heat threshold. It should be understood that other techniques and other visual representations of the region of interest can also be used in conjunction with the DNN interpretation method described herein.

[0100] Now refer to Figure 5 The example DNN is described, demonstrating how the truncated output of the DNN can be used through the various DNN representations and interpretations described in this paper.

[0101] Examples of truncated DNNs for representation and interpretation

[0102] Figure 5 A block diagram of an exemplary deep neural network 500 includes a truncated portion 504 for generating an updated input sample 512 based on an input sample 502 and a tail portion 506 for generating classification data 514 based on the updated input sample 512.

[0103] In this example, the DNN 500 is trained to perform image classification, such as classifying each input sample of a set of input samples 502 into a first category "dog" or a second category "cat". The DNN 500 is used to perform image classification on multiple input samples 502, thereby generating updated input samples 512 and a classification for each input sample. The classification includes classification data 514 generated by the output layer 138 of the tail portion 506. The updated input samples 512 include one or more output activation maps generated by the truncated portion 504 of the DNN 500 based on the input samples.

[0104] In this example, DNN 500 is shown as a convolutional neural network (CNN) with multiple convolutional layers 126 organized within convolutional blocks 124, as referenced above. Figure 1B The truncated portion 504 (also referred to herein as F) is described. trunc The diagram is shown to include several convolutional blocks 124 and a partial convolutional block at its right end consisting of a convolutional layer 126 and a ReLU layer 128. Therefore, in this example, the updated input sample 512 is the output of the ReLU layer 128.

[0105] The tail section 506 (also referred to herein as F) tail The diagram shows a pooling layer 130 comprising a partial convolutional block 124 located at the left end of the truncated portion 504, followed by a fully connected block consisting of an input layer 134 and a hidden layer 136. For the purposes of the currently described embodiment, the output layer 138 of the DNN 500 is not considered part of the tail portion 506 of the DNN 500. In this example, the input layer 134 and the hidden layer 136 can be fully connected layers, and the hidden layer can be used as a logits layer, generating a set of logits 516 as its output. The logits 516 are received as input by the output layer 138, which can apply a Softmax function to normalize the logits 516, thereby generating categorical data 514. For the purposes of the currently described embodiment, the logits 516 can be considered as the output of the DNN 500. It should be understood that the term "hidden layer" is used to maintain consistency with... Figure 1B Consistency with the exemplary neural network; however, in the example DNN 500, all layers except the first convolutional layer 124 and the final hidden layer 136 at the left end of the truncated portion 504 can be considered "hidden" layers because they are not the final input layer or the final output layer 138.

[0106] When a specific input sample 502(1) consisting of photos of dogs is provided, the DNN 500 can generate classification data 514, which includes the probability that the specific input sample 502(1) is classified in the first category or the probability that the specific input sample 502(1) is classified in the second category, shown as classification probabilities (“dog”, 0.993 | “cat”, 0.007).

[0107] The updated input sample 512 generated by the truncated portion 504 is shown as a set of activation maps, which are the outputs of the ReLU layer 128 (i.e., output activation maps). In some embodiments, the updated input sample 512 is generated from different layers of the DNN 500. The updated input sample 512 can be used to simplify the computation of the representation and interpretation of the DNN 500, as shown below with reference to Figures 6 to 7. Figure 8 However, in some embodiments, input sample 502 itself can be used as updated input sample 512.

[0108] In some embodiments, DNN 500 may have a structure consistent with one of the DNN architectures described by Karen Simonyan and Andrew Zisserman in “Very Deep Convolutional Networks for Large-Scale Image Recognition”, 2014, arXiv:1409.1556 [cs.CV], available at https: / / arxiv.org / abs / 1409.1556. For example, DNN 500 may have a structure consistent with the DNN architecture of configuration “A” from Simonyan’s reference: 11 weight layers, including an input layer that accepts a 224×224 RGB image as input; a convolutional layer with 64 kernels of dimension 3×3 (i.e., an receptive field size of 3 and 63 output channels); a maxpool layer; a convolutional layer with 128 kernels of dimension 3×3; another maxpool layer; and two convolutional layers, each with 256 dimensions. The structure consists of: a 3×3 convolutional kernel; another maxpool layer; two convolutional layers, each with 512 kernels (3×3 kernels); another maxpool layer; two more convolutional layers, each with 256 kernels (3×3 kernels); another maxpool layer; a fully connected layer with 4096 output channels; another fully connected layer with 4096 output channels; a fully connected layer with 1000 output channels; and a softmax layer (although the softmax layer can be omitted from the DNN 500 as described above, producing a final fully connected layer with 1000 output channels and a tail logits output of 516). It should be understood that each maxpool layer listed above occurs before or after a ReLU layer. More potential configurations are described in Simonyan's references.

[0109] In some embodiments, DNN 500 is not a CNN. It can be any neural network trained using machine learning and implemented by a computational structure or algorithm, as long as the neural network accepts numerical input (e.g., a set of pixel values ​​of an image) and returns output scores (e.g., normalized probabilities) for each of multiple classes or categories.

[0110] Now refer to Figures 6 to... Figure 7D An exemplary method for describing DNN representations will be described, and references will be made to... Figure 8 An exemplary method for describing DNN interpretation.

[0111] Example of DNN rule extraction method

[0112] Figure 6AThis is a flowchart of an exemplary single-rule extraction method 600 for generating a simplified representation of a DNN including rules. Method 600 may be referred to herein as a "DNN representation," a "model representation," or a "rule extraction." Reference will also be made to... Figures 7A to 7D Describe the steps of method 600. Figures 7A to 7D A visualization of the various steps of method 600 is shown.

[0113] Method 600 generates rules that approximate the classification behavior of a neural network trained to perform a classification task, such as a DNN 500 trained to perform image classification. The rules comprise a convex polyhedron defined by multiple hyperplanes in a multidimensional Euclidean space defined by multiple updated input samples 512 used to generate the rules. The updated input sample 512 within the convex polyhedron is likely to be classified by the neural network into a first category (e.g., "dog").

[0114] Method 600 is executed on a neural network trained to perform the classification task described above. In this example, it will... Figure 5 Method 600 is described in the context of DNN 500.

[0115] The steps of Method 600 can be referred to herein by terms indicating their overall function. Steps 602 and 603 can be collectively referred to as “initialization” of the DNN 500, as they generate an updated sample set for generating rules. Steps 604 and 606 can be collectively referred to as “candidate pool generation”, as they generate a decision boundary candidate pool, which may or may not be included in the rules. Steps 608 and 610 can be collectively referred to as “binary matrix reformatting”, as they reformat the decision boundary set from the candidate pool into a binary matrix. Step 611 can be referred to as “submodule optimization”, as it selects the best subset of columns of the binary matrix to generate a smaller set of hyperplanes for defining the rules. Steps 602 through 612 can be collectively referred to as “single rule extraction”, as they generate a single rule that can be used as a representation of the DNN 500. Step 614 can be referred to as “interpretation”, as it generates a human-understandable representation of the classification behavior of the DNN 500 based on the single rule generated through the single rule extraction.

[0116] In 602, DNN 500 is used to perform image classification on multiple input samples 502, thereby generating updated input samples 512 and classification probabilities of a first class and a second class, such as classification data 514, for each input sample (e.g., the first input sample 502(1)). Each updated input sample 512 includes one or more output activation maps generated by the truncated portion 504 based on the corresponding input sample 502.

[0117] In some embodiments, the same input samples used in method 600 can be used to train the DNN 500. Each updated input sample 512 can be relabeled according to its corresponding classification data 514 generated by the DNN 500: for example, the first input sample 502(1) classified by the DNN 500 as the most likely first category "dog" in the classification data 514 (which is shown as indicating the probabilities of 0.993 and 0.007 in the category "dog" and belonging to the category "cat") can be labeled with its corresponding updated input sample 512 as the first category "dog".

[0118] Figure 7A A two-dimensional visualization of the data space is shown, illustrating input samples 502 labeled with tags associated with categories output by the DNN 500. A first set of labeled input samples 706 includes tags output by the DNN 500 associated with a first category (e.g., "dog"), and a second set of labeled input samples 704 includes tags output by the DNN 500 associated with a second category (e.g., "cat"). Figure 2 As shown, the data space is actually multidimensional, but for the sake of simplicity and visibility, it is displayed as two-dimensional.

[0119] In step 603, a seed sample and multiple contrast samples are selected from the labeled input samples 704 and 706. The seed sample can be any input sample from either the first group 707 or the second group 704. In this example, the DNN 500 is provided with the seed sample and outputs a label associated with the first category (e.g., "dog") of the seed sample, meaning the seed sample is an input sample selected from the first group 706. The seed sample serves as the starting point for generating rules. The updated input sample 512 generated by the truncated portion 504 of the DNN 500 based on the seed sample is called the updated seed sample.

[0120] Since the seed samples are in the first group 706, the multiple contrast samples can be any input samples selected from the second group 704, i.e., input samples of the second category (e.g., "cat") output by DNN 500. Method 600 selects multiple contrast input samples equal to a predetermined candidate pool size. The predetermined candidate pool size effectively defines an upper limit on the number of decision boundaries included in the rules generated by method 600. The updated input sample 512 generated by the truncated portion 504 of DNN 500 based on each contrast sample is called the updated contrast sample.

[0121] In some examples, an updated set of contrast samples is randomly selected from the updated samples of the training data, with the constraint that the updated contrast samples must be classified (i.e., relabeled) into a second category (i.e., different from the relabeled updated seed samples). Importantly, for the purposes of Method 600, the labels attached to the updated contrast samples are not the predetermined training labels attached to the training data samples; instead, they are the labels predicted by the DNN 500. Similarly, for the purposes of Method 600, the labels attached to the seed samples are the labels predicted by the DNN 500.

[0122] Figure 7B It shows Figure 7A The two-dimensional visualization shows that the updated seed sample 708 and five updated contrast samples 712, 714, 716, 718, and 719 are identified. Therefore, in this example, the predetermined candidate pool size is 5.

[0123] In step 604, a binary search algorithm (referred to herein as "binary search") is performed between the updated seed sample 708 and each updated comparison sample 712, 714, 716, 718, 719. The binary search will be described with reference to the example of the updated comparison sample 712 occurring over range 732. It should be understood that the binary search algorithm can be used to locate the desired value at an unknown point located between two known initial values ​​in an ordered sequence of items, such as an ordered list. The binary search algorithm first determines the value of the item at the midpoint between the two initial values. Based on the value determined at the midpoint item, the next step of the binary search algorithm selects a second item at the midpoint between the midpoint item and the first or second initial value, and determines the value of the second item. Other steps repeat this process until an item with or close to the desired value is found. In this case, the desired value is a multidimensional value (e.g., a vector) representing the boundary between two categories in a multidimensional space.

[0124] In some embodiments, the binary search is a modified binary search. Many symbols used in this step and subsequent steps of method 600 will be defined first. x is the seed sample. F is a DNN 500. The output is the logits of 136 hidden layers, which is 516. trunc It's the truncated portion, 504. F tail It represents the tail portion 506, and can also represent the logits output 516 of the hidden layer 136 of the tail portion 506. X = F trunc (x) represents the updated seed sample of 508. K is the predetermined candidate pool size. max This is the maximum boundary size of the rule, i.e., the maximum number of decision boundaries that the rule must include. X1, X2, ..., X KThese are the updated comparison samples, for example, updated comparison samples 712, 714, 716, 718, and 719. s is the search point for the adjusted binary search. H = {H1, H2, ..., H...} K Let} be a set of hyperplanes in the candidate pool. R is a subset of H. S is the solution optimized by the submodule. r is the ratio parameter of the adjusted binary search. e, e1, e2 are some minima with predetermined values. max i (v) is the i-th maximum value of vector v.

[0125] Figure 6B Is Figure 6A The flowchart illustrates the adjusted binary search performed in step 604 of method 600 between the updated seed sample 708 and each updated comparison sample 712, 714, 716, 718, 719.

[0126] In 622, the adjusted binary search is performed by defining the value x. pos =Updated seed sample 708 and x neg =The updated comparison sample 712 begins.

[0127] In 624, calculate the search point s = (1-r)x neg +rx pos .

[0128] In 626, if max1(F tail (s))-max2(F tail (s))<e and argmax(F tail (X))=argmax(F tail (s)), then the adjusted binary search is terminated, where F tail (i) represents the classification data 514 generated by the tail portion 506 based on the updated sample i, and e represents the predetermined search parameter with a relatively small value. The first condition means that s is very close to the decision boundary between the two categories (e.g., "dog" and "cat"). The second condition means that the search point s and the updated seed sample X are on the same side of the decision boundary.

[0129] If the termination condition is not met, then in step 628, the adjusted binary search checks the classification of the search point. If argmax(F tail (X))=argmax(F tail (s)), that is, if the classification of search point s is the same as the classification of the updated seed sample 708 (i.e., the first category "dog" in this example), the binary search proceeds to step 630, where x posThe value of is replaced by s. Otherwise, the adjusted binary search proceeds to step 632, where the value of s is replaced by s. The adjusted binary search then returns to step 624 to recalculate the search point s = (1-r)x neg +rx pos .

[0130] When the binary search terminates, the base value and bias value of the binary search are calculated in step 634. The base value W is calculated as follows:

[0131]

[0132] And the bias value b is calculated as

[0133] b = max1(F tail (s))-max2(F tail (s))-W T s.

[0134] Therefore, a binary search is performed between the updated seed sample and the updated comparison sample to generate base values ​​and bias values;

[0135] Back Figure 6A Once the binary search between the updated seed sample 708 and each updated comparison sample 712, 714, 716, 718, 719 is completed in step 604, method 600 proceeds to step 606. In 606, based on the results of the binary search, decision boundaries between the updated seed sample 708 and each updated comparison sample 712, 714, 716, 718, 719 are calculated. Each decision boundary between input sample categories (e.g., between the updated seed sample 708 in the first category "dog" and the updated comparison sample 712 in the second category "cat") is calculated based on the base value b and the bias value b calculated through the binary search.

[0136] The decision boundary defines one of several hyperplanes in the candidate pool. Hyperplane H i It can be derived from the following equation W T The definition of v+b=0 is given, where v is an arbitrary vector, and the associated label of the hyperplane is F. tail The index of the second maximum value of (s). The "association label" of the boundary refers to the label of the updated sample on the other side of the boundary.

[0137] The candidate pool H is all H's for i = 1, 2, ..., K i A set of.

[0138] Figure 7BThe decision boundaries generated based on various binary searches are shown, such as a first decision boundary 722 defined between the updated seed sample 708 and the updated comparison sample 712 based on a binary search on range 732, and a second decision boundary 724 defined between the updated seed sample 708 and the updated comparison sample 714 based on a binary search on range 734. The set of five decision boundaries (shown as dashed lines) defines the candidate pool H.

[0139] Return again Figure 6A Method 600 proceeds to step 608. In 608, a positive sample binary matrix is ​​generated based on the candidate pool and the updated input samples 512. Each updated input sample 512 has been relabeled with a label associated with either the first or second category. The set of updated input samples 512 including the label associated with the first category can be referred to as the updated positive samples, and the set of updated input samples 512 including the label associated with the second category can be referred to as the updated negative samples.

[0140] The positive sample binary matrix A is generated as follows: each row represents the updated positive sample, and each column represents the hyperplane (i.e., the decision boundary) in the candidate pool. A i，j =1 If the updated positive sample i and the updated seed sample X708 lie in the hyperplane H j On the same side, otherwise A i，j =0. It will be understood that the updated seed sample 708 can be included in the updated positive sample set, such that the rows of A corresponding to the updated seed sample X708 are all the same.

[0141] Figure 7C The binary matrix A740 of positive samples is shown. Each column 746 corresponds to one of the hyperplanes in the candidate pool, such as the first decision boundary 722. The first row 742 corresponds to the updated seed sample X708, and therefore has all bits set to 1. Each additional row 744 corresponds to another updated positive sample, i.e., in... Figure 7A and Figure 7B One of the updated samples 706 is shown as a square.

[0142] Therefore, for each updated positive sample 706, the positive sample binary matrix 740, denoted as A, includes a row of binary values, each binary value indicating whether the updated positive sample 706 is on the first or second side of one of the multiple hyperplanes (e.g., the first hyperplane 722).

[0143] In step 610, a negative sample binary matrix is ​​generated based on the candidate pool and the updated input samples 512, in a process very similar to that of generating the positive sample binary matrix 740. The negative sample binary matrix B is generated as follows: each row represents an updated negative sample, and each column represents a hyperplane (i.e., a decision boundary) in the candidate pool. Ai，j =1 If the updated negative sample i and the updated seed sample X708 lie in the hyperplane H j On the same side, otherwise A i，j =0.

[0144] Figure 7C The binary matrix B750 of negative samples is shown. Each column 746 corresponds to one of the hyperplanes in the candidate pool, such as the first decision boundary 722. Each row 752 corresponds to the updated negative sample, i.e., in... Figure 7A and Figure 7B The triangle shown is one of the updated samples 704.

[0145] Therefore, for each updated negative sample 704, the negative sample binary matrix 750, denoted as B, includes a row of binary values, each binary value indicating whether the updated negative sample 704 is on the first or second side of one of the multiple hyperplanes (e.g., the first hyperplane 722).

[0146] In step 611, the optimal subset of columns is selected from the negative sample binary matrix 740 and the negative sample binary matrix 750. Some embodiments can use submodule-cost submodule coverage optimization, as referenced below. Figure 6C The description states that the number of hyperplanes in the multiple hyperplanes of the convex polyhedron defining the rules is less than the predetermined candidate pool size.

[0147] Figure 6C It is used from Figure 7C The flowchart illustrates an exemplary method for selecting the best subset of columns from a binary matrix. It specifies the sub-step of step 611 in method 600.

[0148] In 642, for a subset of column R, G A (R) is defined as the number of rows in a 740-column binary matrix of positive samples in R. This means the number of positive samples updated outside the convex polyhedron defined by the hyperplane in R. R is defined in a similar way for a 750-column binary matrix of negative samples.

[0149] In 644, the combinatorial optimization problem is expressed by a formula. In some embodiments, this could be an example of the Submodular Cost Submodular Cover (SCSC) optimization problem: min R G A (R)subjectto G B (R)=G B (H)

[0150] In 646, a greedy algorithm is used to solve the SCSC optimization. The set of techniques S is initialized to empty. Then, according to... Repeat selecting the new column and adding j to S. When G B (S)=G B (H) means that all updated samples from other categories are not included, or |S| = s max This means the algorithm has reached the maximum number of boundary values ​​to be included in the rules, and the algorithm terminates. The purpose of e1 and e2 is to ensure that the numerator and denominator are not zero.

[0151] It should be understood that in other embodiments, other known algorithms may be used to solve SCSC.

[0152] Alternatively, in 648, this can be achieved by adding a parameter that does not reduce G. A (S) and G B Tighten the rule by taking all columns of H\S of the value of (S). This may reduce the volume of the polyhedron and make it more accurate. If the rule is to be used for model representation, i.e. as a simplified classifier, it is recommended to use this step 648, as described below.

[0153] Alternatively, in 650, G can be increased by deleting S. A (S) and G B Tighten the rule by examining all columns of the (S) value. This will remove any redundant boundaries. If the rule is used to generate a simplified interpretation of a model approximated by a DNN (also known as "model interpretation," "DNN interpretation," or "DNN model interpretation"), step 650 is recommended, as referenced below. Figure 8 As stated above.

[0154] Alternatively, in some embodiments, different algorithms may be applied to remove redundant linear inequality constraints.

[0155] Back Figure 6A In step 611, after selecting the optimal subset of columns from binary matrices 740 and 750, method 600 proceeds to step 612. In 612, a rule is generated. This rule is defined as a convex polyhedron containing the updated seed sample X 708 and defined by hyperplanes in the set S. The associated label of this rule is the updated label of X 708. In some embodiments, the number of hyperplanes in the plurality of hyperplanes S defining the rule (up to s) max () is less than the predetermined candidate pool size K.

[0156] Figure 7D It shows the result of Figure 7BThe rules are defined by subsets of multiple decision boundaries. In this example, the rules are defined by three decision boundaries 722, 726, and 728 included in S after binary matrix reconstruction and submodule optimization. As mentioned above, each decision boundary 722, 726, and 728 can have associated labels that identify the labels of updated samples on the other side of the boundary. The rules as a whole can also have associated labels indicating the categories that all updated samples covered by the rule are classified by the DNN 500.

[0157] Optionally, in step 614, this rule can be used to perform the following reference Figure 8 Explanation of the described DNN model.

[0158] In some embodiments, one or more additional rules may be extracted, such as Figure 2 Additional rule 230 in the DNN500. The multi-rule representation of DNN500 can be used as a simplified classifier to approximate the classification behavior of DNN500. Now refer to Figure 6D Describe an exemplary multi-rule extraction method.

[0159] Example of a multi-rule extraction method

[0160] Figure 6D It is used for iteration Figure 6A The flowchart of an exemplary multi-rule extraction method 660 for generating multi-rule representations of DNNs is provided.

[0161] The notation and terminology used to describe method 660 will now be defined. n is the size of the set of all input samples. D = {x1, x2, ..., x...} n ) is the set of all input samples (e.g., the union of sets 704 and 706). p is the desired coverage. S represents a single rule extracted using single-rule extraction method 600. all It is the set of all rules extracted at a given time when multi-rule extraction method 660 is executed. cov(S) all ) is S all The input sample coverage is T. T is the maximum number of rules to be included in the multi-rule representation. A "rule" is a convex polyhedron in Euclidean space with associated labels. The "scope of application" is S. all The union of all regions covered by the rule. "Input sample coverage" refers to the number of input samples within the scope of at least one rule.

[0162] In 662, the multi-rule extraction method 660 is initialized. A set of decision boundaries (i.e., hyperplanes) S all It is set to be equal to the empty set. The index value j is set to be equal to the initial value, for example, 1.

[0163] In 664, the seed sample x corresponding to the index value j is used.j Execute single-rule extraction method 600. This generates a single rule defined by a convex polyhedron, which is updated in the seed sample X. j The surrounding area has decision-making boundaries.

[0164] In step 666, the decision boundary set S of the rules generated in step 664 is added to the set S. all In the middle, making S all =S all ∪S. The index value j is incremented to the value j+1.

[0165] In step 668, a termination condition is checked. The multi-rule extraction method 660 terminates if either of the following two conditions is met: First, if a predetermined input sample coverage threshold is met (e.g., cov(S)). all )>pn), or secondly, if the number of rules extracted is equal to the maximum number of rules to be included in the multi-rule representation (e.g., |S all |≥T).

[0166] As described above, input sample coverage refers to the number of updated input samples 512 included within the scope of one or more rule sets. A given input sample 502 is also considered to be within the scope of a set of one or more rules if its respective updated input sample 512 is within that scope. Therefore, if a rule includes 50 updated input samples 512 within its decision boundary, its input sample coverage is 50. If three rules are generated that collectively include 80 updated input samples, the input sample coverage of the multi-rule representation defined by these three rules is 80. A predetermined input sample coverage threshold can be a parameter controlling the multi-rule extraction method 660 used to generate the multi-rule representation of the DNN; in some embodiments, it can be set to a fixed percentage equal to the number of input samples.

[0167] A set of rules extracted using the multi-rule extraction method 660 can be used to generate a multi-rule representation of a DNN. This rule set S all A simplified classifier that approximates the classification behavior of a DNN 500 can be used: An updated input sample 512, within the applicability of the multi-rule representation of the DNN, can be associated with a predicted label or a set of probabilities related to one or more categories by voting in the rules covering the updated input sample 512. It should be understood that the new input sample 502 to be classified by the multi-rule representation must first be updated to generate the updated input sample 512 using the truncated portion 504 of the DNN 500.

[0168] It should be understood that, according to DNN 500, for each new seed sample x j The generated label, S allThe various rules can include updating the input sample 512 of different categories, for example Figure 2 The first rule 204 (which includes the first category "dog") and the supplementary rule 230 (each of which includes the second category "cat") are shown.

[0169] When the multi-rule extraction method 660 for generating multi-rule representations of CNNs is executed, the CNN implementation for DNN 500 can exhibit additional advantages: the decision boundary of the CNN is piecewise linear. Therefore, when computing the candidate hyperplanes (e.g., 241) of the candidate pool, the hyperplanes sometimes actually overlap with the actual decision boundary 202. This property of CNNs is not a necessary property for generating simplified multi-rule representations of DNN 500 as described above, but it can lead to simpler and / or more accurate multi-rule representations of CNNs. For non-CNN DNNs, each candidate hyperplane is typically a linear approximation of the decision boundary (usually non-linear). This piecewise linearity property of CNNs is not inherent to CNNs; rather, it is a result of the ReLU function used in the ReLU layer 128 of the CNN, e.g., the ReLU layer 128 located to the right of the truncated portion 504 of DNN 500. Any DNN using a piecewise linear activation function can be represented or interpreted using the techniques described herein.

[0170] Example methods of DNN interpretation

[0171] Figure 8 It is used to generate by Figure 6A Single rule extraction method 600 or Figure 6D A flowchart of an exemplary interpretation method 800 for interpreting one or more rules generated by the multi-rule extraction method 660.

[0172] The notation and terminology used to describe method 800 will now be defined. x is the input sample 502 used to generate a human-understandable explanation, called the explanation sample. F is a DNN or a DNN approximation model, such as DNN 500. One of the outputs of the DNN 500 used in method 800 is the logits 516 generated by the hidden layer 136. trunc It is the truncated portion 504, and can also represent the updated input sample 512 generated by the truncated portion 504. tail It is the tail portion 506, and can also represent Logits 516 generated by the hidden layer 136 of the tail portion 506. X = F trunc (x) is the updated explanatory sample. L is the number of activation maps included in the updated explanatory sample X. K is the number of boundaries of the rule being explained. H1, H2, ..., H KThis refers to the boundary (i.e., hyperplane) of the rule. A "rule" is a convex polyhedron in Euclidean space with associated labels. "Scope" refers to the region encompassed by the rule. A "heatmap" is a single-channel image of the same size as the interpretation sample; in this example, the interpretation sample is an image. The pixel values ​​of the heatmap are called heat values, where the larger the value, the more important the pixel is to the rule being interpreted.

[0173] Using method 800, a variety of useful outputs can be generated. One or more interpretations can be generated in the form of human-understandable representations of one or more corresponding decision boundaries of the rule: for example, a heatmap can be generated for each boundary of the rule for the interpretation sample, and each heatmap can be combined with the interpretation sample to generate a human-understandable representation of the decision boundaries of the rule (e.g., Figure 3 and Figure 4 As shown in the figure). This applies to cases where all samples are covered by the rules, such as heatmap matrices (e.g.). Figure 3 (As shown) can be generated using multiple explanatory samples. The similarity of each explanatory sample to the main explanatory sample can be used to provide a ranking. Furthermore, boundary statistics can be generated to provide information on how each boundary of the rule is enforced, as well as the distribution of boundary-associated labels.

[0174] Method 800 starts with the rules generated by single rule extraction method 600.

[0175] In step 801, an updated input sample 512 is identified within the scope of the rule's applicability. The updated input sample 512 is used as an interpretation sample to generate one or more heatmaps and / or human-understandable representations of the rule's decision boundary.

[0176] In step 802, a hyperplane (i.e., a decision boundary) is selected or identified from among the multiple hyperplanes of the rule. This decision boundary will be interpreted in the first iteration of steps 804 to 810 of method 800.

[0177] In 804, the activation mapping X of the updated explanatory sample is computed. l The weights are determined by taking... Relative to X l It is calculated by averaging the partial derivatives of the hyperplane, that is, by averaging the partial derivatives of the hyperplane with respect to each activation value of the activation map. The formula is: The weights are set to be equal to the calculated average.

[0178] In 806, the calculated weights are associated with the activation mapping X. l Multiply to generate a weighted activation map. Then, method 800 can repeat steps 804 and 806 for each of one or more additional activation maps (in this example, all activation maps) of the updated interpretation sample.

[0179] In 808, a heatmap is generated by computing the sum of each activation map weighted by its respective weight. The heatmap represents the decision boundary corresponding to the hyperplane. In this example, the heatmap is a shrinking heatmap computed using ReLU of the weighted sum of the activation maps. The formula is:

[0180]

[0181] In this example, as another sub-step of step 808, the shrinking heatmap can be scaled back to the original size of the input image. A total of K heatmaps are generated, thus providing more than one interpretation as described above.

[0182] In 810, optionally, as referenced above. Figure 3 and Figure 4 As described in detail, a human-understandable representation of the decision boundary is generated based on the interpreted samples. For example, the human-understandable representation could be an image modified from a heatmap or an image with bounding boxes around the region of interest, such as... Figure 3 and Figure 4 As shown.

[0183] Method 800 may repeat steps 802 to 808 or 802 to 810 to generate one or more additional heatmaps and / or human-understandable representations of one or more additional decision boundaries for interpreting rules.

[0184] Method 800 may also repeat steps 801 to 808 or 802 to 810 to generate one or more additional heatmaps and / or human-understandable representations to interpret one or more additional decision boundaries of the rules applied to one or more additional interpretive samples.

[0185] In 812, optionally, a heatmap matrix can be generated, where various explanatory samples are ordered according to their similarity to the first or primary explanatory sample. The heatmap matrix can be similar to... Figure 3 The image matrix shown contains explanatory samples in each row and human-understandable representations in each column. It requires returning to step 801 at least once to identify new explanatory samples (i.e., the second row of the matrix), and may also involve returning to step 802 at least once for each explanatory sample to generate additional human-understandable representations after the first explanatory sample (i.e., adding more columns after the first two explanatory samples, the first column being the unmodified explanatory sample, and the second column being the first human-understandable representation). Rows can be sorted based on the similarity between the explanatory samples in that row and the explanatory samples in the first row.

[0186] To calculate a given explanatory sample X other The similarity between the sample X and the first explanatory sample is expressed by the formula:

[0187]

[0188] This can be used to define the similarity level. The lower the value, the higher the similarity.

[0189] In some embodiments, the heatmap matrix or other displays of human-understandable representations may be captioned, labeled, or otherwise identified for each human-understandable representation, such as each image. The identification label attached to each human-understandable representation can be used to identify the associated label of the corresponding decision boundary, i.e., the category that the decision boundary attempts to separate from the category within the association range of the rule. This information may be less meaningful in binary classification of new input samples (i.e., classifying input samples into a set of categories containing only two classes), but may be valuable in multi-class settings (i.e., classifying input samples into a set of categories containing more than two classes).

[0190] In step 814, optionally, boundary statistics can be generated, for example, in the form of a report, to understand the role of each boundary of the rule. Boundary statistics may include the order in which the boundaries were selected in the greedy algorithm of step 646, the number of updated input samples separated by each boundary, and the distribution of associated labels. The distribution of associated labels can be described as the effort expended to separate a particular class from the target class (i.e., the class included in the rule).

[0191] In some embodiments, the average or combination of multiple heatmaps may be sensed to generate a single human-understandable representation. In some embodiments, the first boundary selected in the greedy algorithm of step 646 may be sensed to select a single heatmap for a human-understandable representation. It should be understood that there are many other alternative methods to summarize multiple heatmaps into a single heatmap.

[0192] Experimental data comparing the exemplary embodiments described herein with other known methods of deep neural network interpretation show that method 800 can be used to generate other representations of human-understandable visualizations or classification behavior of deep neural networks, which exhibit fidelity, stability and comprehensiveness relative to other known methods, and in some examples can be generated before the deep neural network is fully trained.

[0193] Overview

[0194] Although the methods and processes are described in a specific order in this invention, one or more steps of the methods and processes may be omitted or modified as appropriate. One or more steps may be performed sequentially, but not in the order described (as the case may be).

[0195] Although the invention has been described, at least in part, those skilled in the art will understand that the invention also relates to various components for performing at least some aspects and features of the described methods by means of hardware components, software, or any combination of both. Accordingly, the technical solutions of the invention can be embodied in the form of a software product. Suitable software products can be stored in pre-recorded storage devices or other similar non-volatile or non-transitory computer-readable media, such as DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes examples of instructions stored thereon that enable processing devices (e.g., personal computers, servers, or network devices) to perform the methods disclosed herein.

[0196] This invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative in all respects and not restrictive. Selected features from one or more of the foregoing embodiments may be combined to create alternative embodiments not explicitly described, and it is understood that features suitable for such combinations are within the scope of this invention.

[0197] All values ​​and sub-ranges within the scope of disclosure are also disclosed. Furthermore, while the systems, devices, and processes disclosed and shown herein may include a specific number of elements / components, the systems, devices, and assemblies may be modified to include more or fewer of these elements / components. For example, although any element / component disclosed may be referenced in the singular, embodiments disclosed herein may be modified to include multiple such elements / components. The subject matter described herein is intended to cover and encompass all appropriate technical changes.

[0198] All published papers disclosed in this invention are incorporated herein by reference.

[0199] Other aspects and examples of the invention are set forth in the appendix, the entire contents of which are incorporated herein by reference.

Claims

1. A method for generating a simplified representation of a neural network trained to perform classification, characterized in that, The trained neural network includes a truncated portion and a tail portion, the truncated portion including one or more layers, and the tail portion including one or more layers; the method includes: The truncated portion is used to generate updated input samples, the updated input samples including one or more output activation maps based on the input samples, the input samples including image data; Labels are generated using the tail portion, wherein the labels classify the input sample into a first category or a second category based on one or more output activation maps, the first category being "dog" and the second category being "non-dog"; The rule generates a convex polyhedron defined by a plurality of hyperplanes in a multidimensional Euclidean space defined according to the plurality of updated samples, such that updated input samples within the convex polyhedron are classified into a first category by the trained neural network, and updated input samples outside the convex polyhedron are classified into a second category by the trained neural network. An explanation is generated based on a human-understandable representation of one or more corresponding decision boundaries of the rules.

2. The method according to claim 1, characterized in that, The multiple input samples include: Seed samples classified in the first category by the trained neural network, wherein the truncated portion generates updated seed samples based on the seed samples; Multiple contrast input samples equal to a predetermined candidate pool size, wherein each contrast input sample is classified in the second category by the trained neural network, and the truncated portion generates updated contrast samples based on each contrast input sample; For each updated comparison sample, the rules are generated as follows: A binary search is performed between the updated seed sample and the updated comparison sample to generate base values ​​and bias values; The decision boundary between the first category and the second category is calculated based on the base value and the bias value of the binary search, wherein the decision boundary defines one of the plurality of hyperplanes.

3. The method according to claim 2, characterized in that, The plurality of input samples also includes: One or more positive samples, wherein each positive sample is classified in the first category by the trained neural network, and the truncated portion generates updated positive samples based on each positive sample; One or more negative samples, wherein each negative sample is classified in the second category by the trained neural network, and the truncated portion generates an updated negative sample based on each negative sample; For each hyperplane, the rules for generating the above rules also include: For each updated positive sample, a positive sample binary matrix is ​​generated, comprising a row of binary values, each binary value indicating the updated positive sample on the first or second side of a hyperplane in the hyperplane, and each column of the positive sample binary matrix corresponding to a hyperplane; For each updated negative sample, a negative sample binary matrix is ​​generated, comprising a row of binary values, each binary value indicating the updated negative sample on the first or second side of a hyperplane in the hyperplane, and each column of the negative sample binary matrix corresponds to a hyperplane; Using submodular cost submodular coverage optimization, the plurality of hyperplanes are selected from the columns of the positive sample binary matrix and the columns of the negative sample binary matrix, such that the number of hyperplanes in the plurality of hyperplanes of the convex polyhedron defining the rule is less than the predetermined candidate pool size.

4. The method according to claim 3, characterized in that, The positive sample binary matrix also includes a row of binary values ​​corresponding to the updated seed sample; The negative sample binary matrix also includes a row of binary values ​​corresponding to each updated comparison sample.

5. The method according to any one of claims 1 to 4, characterized in that, The plurality of input samples includes explanatory samples classified into the first category by the trained neural network, the truncated portion generates updated explanatory samples based on the explanatory samples, and the method further includes generating an explanation of the rule through the following steps: Identify one of the plurality of hyperplanes according to the rule; For each activation map in one or more activation maps of the updated interpretation sample, each activation map includes multiple activation values: Calculate the average of the partial derivatives of the hyperplane with respect to each activation value of the activation map; Set the weight of the activation map to be equal to the calculated average value; A heatmap is generated by calculating the sum of each activation map weighted by the respective weights of each activation map, the heatmap representing the decision boundary corresponding to the hyperplane.

6. The method according to claim 5, characterized in that, Each input sample is an image, and the method further includes generating a human-understandable representation of the rule by combining the heatmap with the interpretation sample.

7. The method according to claim 6, characterized in that, Combining the heatmap with the explanatory sample includes generating an image that includes at least a portion of the explanatory sample, wherein the heat value of a visual indicator of a portion of the heatmap is above a significant threshold.

8. The method according to claim 5, characterized in that, Generating a representation of the rule further includes repeating the following steps: Identify hyperplanes; Calculate the average value for each activation map; Set the weight for each activation map; Generate a heatmap for each of the remaining hyperplanes of the convex polyhedron of the rule.

9. The method according to claim 1, characterized in that, For each of the first plurality of input samples, generate an updated input sample and generate a label to generate the first plurality of updated samples and the first plurality of labels; The generation rule is executed based on the first plurality of updated samples and the first plurality of labels, thereby generating the first rule; Generate updated samples and generated labels for each of one or more additional multiple input samples, to generate one or more additional multiple updated samples and one or more additional multiple labels; The generation rule is executed based on one or more additional update samples and one or more additional labels to generate one or more additional rules, wherein the first rule and the one or more additional rules together constitute multiple rules; The method also includes generating a multi-rule classifier that includes the multiple rules.

10. A system, characterized in that, include: A processing system that includes one or more processor devices; A memory storing instructions, which, when executed by the processing system, cause the system to generate a simplified representation of a neural network trained to perform a classification task on input samples. The trained neural network includes a truncated portion and a tail portion, the truncated portion comprising one or more layers, and the tail portion comprising one or more layers, comprising the following operations: The truncated portion is used to generate updated input samples, the updated input samples including one or more output activation maps based on the input samples, the input samples including image data; Labels are generated using the tail portion, wherein the labels classify the input sample into a first category or a second category based on one or more output activation maps, the first category being "dog" and the second category being "non-dog"; The rule generates a convex polyhedron defined by a plurality of hyperplanes in a multidimensional Euclidean space defined according to the plurality of updated samples, such that updated input samples within the convex polyhedron are classified into a first category by the trained neural network, and updated input samples outside the convex polyhedron are classified into a second category by the trained neural network. An explanation is generated based on a human-understandable representation of one or more corresponding decision boundaries of the rules.

11. The system according to claim 10, characterized in that, The multiple input samples include: Seed samples classified in the first category by the neural network, wherein the truncated portion generates updated seed samples based on the seed samples; Multiple contrast input samples equal to a predetermined candidate pool size, wherein each contrast input sample is classified into the second category by the neural network, and the truncated portion generates updated contrast samples based on each contrast input sample; For each updated comparison sample, the rules are generated as follows: A binary search is performed between the updated seed sample and the updated comparison sample to generate base values ​​and bias values; The decision boundary between the first category and the second category is calculated based on the base value and the bias value of the binary search, wherein the decision boundary defines one of the plurality of hyperplanes.

12. The system according to claim 11, characterized in that, The plurality of input samples also includes: One or more positive samples, wherein each positive sample is classified in the first category by the trained neural network, and the truncated portion generates updated positive samples based on each positive sample; One or more negative samples, wherein each negative sample is classified in the second category by the trained neural network, and the truncated portion generates an updated negative sample based on each negative sample; For each hyperplane, the rules for generating the above rules also include: For each updated positive sample, a positive sample binary matrix is ​​generated, comprising a row of binary values, each binary value indicating the updated positive sample on the first or second side of a hyperplane in the hyperplane, and each column of the positive sample binary matrix corresponding to a hyperplane; For each updated negative sample, a negative sample binary matrix is ​​generated, comprising a row of binary values, each binary value indicating the updated negative sample on the first or second side of a hyperplane in the hyperplane, and each column of the negative sample binary matrix corresponds to a hyperplane; Using submodular cost submodular coverage optimization, the plurality of hyperplanes are selected from the columns of the positive sample binary matrix and the columns of the negative sample binary matrix, such that the number of hyperplanes in the plurality of hyperplanes of the convex polyhedron defining the rule is less than the predetermined candidate pool size.

13. The system according to claim 12, characterized in that, The positive sample binary matrix also includes a row of binary values ​​corresponding to the updated seed sample; The negative sample binary matrix also includes a row of binary values ​​corresponding to each updated comparison sample.

14. The system according to any one of claims 10 to 13, characterized in that, The plurality of input samples includes explanatory samples classified into the first category by the trained neural network. The truncated portion generates updated explanatory samples based on the explanatory samples. When the instruction is executed by the processing system, the system also generates an explanation of the rule through the following steps: Identify one of the plurality of hyperplanes according to the rule; For each activation map in one or more activation maps of the updated interpretation sample, each activation map includes multiple activation values: Calculate the average of the partial derivatives of the hyperplane with respect to each activation value of the activation map; Set the weight of the activation map to be equal to the calculated average value; A heatmap is generated by calculating the sum of each activation map weighted by the respective weights of each activation map, the heatmap representing the decision boundary corresponding to the hyperplane.

15. The system according to claim 14, characterized in that, Each input sample is an image, and the system includes generating a human-understandable representation of the rule by combining the heatmap with the interpretation sample.

16. The system according to claim 15, characterized in that, Combining the heatmap with the explanatory sample includes generating an image that includes at least a portion of the explanatory sample, wherein the heat value of a visual indicator of a portion of the heatmap is above a significant threshold.

17. The system according to claim 14, characterized in that, Generating a representation of the rule further includes repeating the following steps: Identify hyperplanes; Calculate the average value for each activation map; Set the weight for each activation map; Generate a heatmap for each of the remaining hyperplanes of the convex polyhedron of the rule.

18. The system according to claim 10, characterized in that, For each of the first plurality of input samples, generate updated samples and generate labels to generate the first plurality of updated samples and the first plurality of classifications; The generation rule is executed based on the first plurality of updated samples and the first plurality of labels, thereby generating the first rule; Generate updated samples and generated labels for each of one or more additional multiple input samples to generate one or more additional multiple updated samples and one or more additional multiple classifications; The generation rule is executed based on one or more additional update samples and one or more additional labels to generate one or more additional rules, wherein the first rule and the one or more additional rules together constitute multiple rules; Generate a multi-rule classifier that includes the multiple rules.

19. A computer-readable medium including instructions, characterized in that, When the instruction is executed by the processing unit, the processing unit performs the method according to any one of claims 1 to 9.

20. A computer program comprising instructions, characterized in that, When the instruction is executed by the processing system, it causes the processing system to perform the method according to any one of claims 1 to 9.

Images