Interpretable hyper-features based on perturbation methods

Abstract

In an embodiment, a computer hosts a machine learning (ML) model that inferes a particular inference for a particular tuple based on a number of features. These features are grouped into predefined hyper-features, each hyper-feature containing a subset of non-intersecting (i.e., non-intersecting, mutually exclusive) features. For each hyper-feature, the computer: a) randomly selects a number of permutation values from the original values of the hyper-feature in the original tuple, b) generates permutation tuples based on the particular tuple and the respective permutation values, and c) causes the ML model to infer a respective permutation inference for each permutation tuple. A proxy model is trained based on the permutation inference. For each hyper-feature, a respective importance of the hyper-feature is calculated based on a proxy model. The hyper-feature importance may be used to rank hyper-features by influence and / or to generate local ML interpretability (MLX) interpretations.

Description

Technical Field

[0001] The present invention relates to machine learning (ML) explainability (MLX). This paper introduces a local explanation technique for black-box ML models based on hyper-feature importance built by feature permutation of dataset samples. Background Art

[0002] Machine learning (ML) and deep learning have become ubiquitous for two main reasons: their ability to solve complex problems in a wide variety of different domains, and the increasing performance and efficiency of modern computing resources. However, as the complexity of problems continues to increase, so does the complexity of the ML models applied to these problems.

[0003] Deep learning is a classic example of this trend. While other ML algorithms (such as neural networks) may contain only a few layers of densely connected neurons, deep learning algorithms (such as convolutional neural networks) may contain tens to hundreds of layers of neurons that perform very different operations. Increasing the depth of neural models and the heterogeneity of layers brings many benefits. For example, greater depth can increase the capacity of the model, improve the generalization of the model, and provide the model with the opportunity to filter out unimportant features, while including layers that perform different operations can greatly improve the performance of the model. However, these optimizations come at the cost of increased complexity and reduced human interpretability of the model's operations.

[0004] Compared to many other ML models, interpreting and understanding the results of complex deep learning models is a challenging task. For example, a decision tree can perform binary classification based on N input features. During training, the features that have the greatest impact on class prediction are inserted near the root of the tree, while features that have less impact on class prediction are placed near the leaves of the tree. Feature importance can be directly determined by measuring the distance of the decision node to the root of the decision tree.

[0005] Such models are often referred to as inherently interpretable. However, as the complexity of a model increases (e.g., the number of features or the depth of a decision tree increases), interpreting the explanation of the model's inferences becomes increasingly challenging. Similarly, even relatively simple neural networks with a few layers can be difficult to interpret because multiple layers combine the effects of features and increase the number of operations between the model's input and output. Therefore, alternative techniques are needed to help interpret complex ML and deep learning models.

[0006] ML Explainability (MLX) is the process of explaining and interpreting ML and deep learning models. MLX can be roughly categorized into local explainability and global explainability:

[0007] Local: Explain why the ML model made a specific prediction corresponding to a given sample to answer questions such as why the ML model made a specific prediction.

[0008] Big picture: Understand the general behavior of the ML model as a whole to answer questions such as how the ML model works or what the ML model has learned from the training data.

[0009] ML models accept as input instances, such as feature vectors based on many features of various data types, each with many or infinite number of possible values. Each feature provides a dimension in the huge multi-dimensional problem space, where a given multi-feature input is just a point. Even though a global explanation may be based on many input instances, these instances miss most of the multi-dimensional problem space, and there are huge spatial gaps between these instances. Therefore, the accuracy of a global explanation may be low for explaining a specific inference of the ML model for a specific input, which almost always falls within this spatial gap of the unknown behavior of the ML model. Local explanation methods such as Shapley require a large number of input instances and output inferences, which grow exponentially with the number of features because Shapley's design explores relationships between features, which are combinatorially intractable. In other words, best-of-breed local explainers are not scalable and may be computationally overwhelmed by wide feature vectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the attached picture:

[0011] Figure 1 is a block diagram depicting an example computer for providing machine learning (ML) explainability (MLX) for a black-box ML model based on permuting tuples to be explained to generate permuted tuples;

[0012] Figure 2 is a flow chart depicting an example computer process that can provide a local MLX for a black-box ML model based on permuting tuples to be interpreted to generate permuted tuples;

[0013] Figure 3 is a block diagram illustrating a computer system upon which embodiments of the present invention may be implemented;

[0014] Figure 4 is a block diagram illustrating a basic software system that may be used to control the operation of a computing system. DETAILED DESCRIPTION

[0015] In the following description, for the purpose of explanation, many specific details will be set forth in order to provide a comprehensive understanding of the present invention. However, it is apparent that the present invention can be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form to avoid unnecessary obfuscation of the present invention.

[0016] General Overview

[0017] Prior art machine learning (ML) explanation (MLX) techniques are based on a problem space with many dimensions, called features. This space may contain hundreds of features. Subsets of features may be related based on modalities, such as features coming from the same source or features that collaborate to describe the same aspect. Analyzing and presenting related features together as modalities allows humans to interpret more intuitively and quickly, and are better able to connect MLX explanations to possible follow-up actions and investigations, such as troubleshooting the source of the modality. For example, each modality may have its own semantics that humans can understand.

[0018] Existing techniques, MLX, are not modality-aware and instead treat hundreds of features as independent features that are uncorrelated with each other. The result is that MLX interpretation is more complex, and since complexity and usability are inversely proportional to the human factor, the interpretation is more difficult to understand and therefore less useful. Features in the same modality are more likely to be statistically correlated, which can make existing MLX techniques slower or less reliable, as described in this paper.

[0019] Super features are a subset of features based on the same modality and processed together for MLX. This paper is an explanation technique to extract the importance of local super features of a trained ML or deep learning (DL) model (called a black-box model). To locally explain the behavior of an ML model, a permutation-based MLX technique evaluates how the ML model's predictions change on permuted versions of the instance to be explained. Super features that have a much larger impact on the ML model's predictions after permutation are considered more important than permuted super features that cause little change in the ML model's predictions. This approach includes a highly stable, linear-time, permutation-based, model-independent MLX local feature attribution.

[0020] As combinations of individual eigenvalues, our technique draws hyper-eigenvalues ​​from the empirical marginal distribution of a reference dataset to improve realism and avoid evaluating the importance of combinations of eigenvalues ​​that are completely outside the range of realistic value combinations, even if the individual values ​​in the combination are themselves realistic. By permuting with the underlying data distribution, the overall quality (i.e., accuracy) of the explanation may increase quantitatively because the generated data instances can explore parts of the ML model's multidimensional latent space that may be encountered by future real-world instances that are not observed in the reference dataset. Similarly, this approach can reduce the number of instances that must be generated to obtain explanations of equivalent quality, thereby reducing the consumption of time and space.

[0021] Data distribution is critical to realism. Perturbing original instances to generate new instances can result in samples that deviate from the distribution. Unrealistic data is problematic because it can confuse ML models, reducing the accuracy of inferences such as classification. Unrealistic instances arise in regions of the multidimensional problem space where ML models are unreliable or even unstable, such as being prone to unpredictable discontinuities in the prediction solution space, preventing the instances from being modified or used as-is in the real world as predicted. As a result, unrealistic instances have little explanatory value and can undermine confidence in MLX.

[0022] Important local MLX use cases are interactive and do not tolerate latency well. Customer experience (CX) may suffer. For example, local MLX may be used during a phone conversation such as with a support or sales agent. Localized neighborhoods of permutation instances should be generated quickly. Optimizing the above concerns and criteria is expensive for high-dimensional datasets with many constituent data types.

[0023] Embodiments may generate and train additional ML models that are non-black-box ML models. Additional ML models are referred to herein as surrogate models, or, due to human understandability, as interpretable models.

[0024] In practice, black-box ML models learn the vast multidimensional space of a large training corpus and are more complex than surrogate models that only need to learn a small neighborhood based on a specific MLX call. Surrogate models can have a straightforward and simplified architecture, such as a decision tree or linear regression. For example, the coefficients of a linear regression or the number of levels of a decision tree may be more or less directly used as the importance score of a super feature.

[0025] The surrogate model may or may not use feature vectors that directly encode superfeatures instead of features. The value of a superfeature is encoded as a row offset in the corpus of the original tuple. So even though there may be dozens of features within each superfeature, the value of the surrogate model may be encoded as a few integers for a few superfeatures. This dimensionality reduction means that the surrogate model can be trained in less time and less space without sacrificing accuracy.

[0026] In an embodiment, a computer hosts a machine learning (ML) model that infers a specific inference for a specific tuple based on many features. The features are grouped into predefined super-features, each super-feature containing a disjoint (i.e., non-intersecting, mutually exclusive) subset of features. For each super-feature, the computer: a) randomly selects a number of permuted values ​​from the original values ​​of the super-feature in the original tuple, b) generates permuted tuples based on the specific tuple and the corresponding permuted values, and c) causes the ML model to infer the corresponding permuted inference for each permuted tuple. A proxy model is trained based on the permuted inference. For each super-feature, the corresponding importance of the super-feature is calculated based on the proxy model. The super-feature importance can be used to rank the super-features by influence and / or generate local ML interpretability (MLX) explanations.

[0027] 1.0 Sample Computer

[0028] Figure 1 1 is a block diagram depicting an example computer 100 in an embodiment. The computer 100 provides machine learning (ML) explainability (MLX) for a black box ML model 160 based on permuting the tuple to be explained T5 to generate a permuted tuple 141. Based on the permuted inference 143, the super features 131-133 can be scored and ranked for relative importance, according to which a local MLX explanation can be generated. The computer 100 can be one or more of a rack server such as a blade, a personal computer, a mainframe, a virtual computer, a smart phone, or other computing devices.

[0029] 1.1 Black Box Model

[0030] In various embodiments, hosted in the memory of computer 100 is a trained ML model 160 that can operate for classification, regression, prediction, anomaly detection, clustering, or other ML purposes. In operation, ML model 160 is applied to tuples (such as tuple 150) to generate inferences (such as inference 170), which can be classes or values ​​for regression or prediction. In an embodiment, inference 170 contains one or more numeric scores or probabilities, such as corresponding probabilities for each of a plurality of classes. In an embodiment, inference 170 is a number and is compared to a threshold to detect whether tuple 150 is abnormal. Tuples will be explained later in this document.

[0031] ML model 160 may be a black box model having an unknown, opaque or obfuscated architecture, which more or less prevents direct inspection and interpretation of the internal operation of ML model 160. In an embodiment not shown, ML model 160 is hosted in a different computer than computer 100, and computer 100 applies the techniques herein by remotely using ML model 160. For example, computer 100 may send tuple 150 to ML model 160 via a communication network, and responsively receive inference 170 via the communication network. For example, computer 100 and ML model 160 may be owned by different parties and / or hosted in different data centers. In various embodiments where ML model 160 is hosted in computer 100, the techniques herein may or may not share an address space and / or operating system process with ML model 160. For example, calling ML model 160 may or may not require inter-process communication (IPC).

[0032] 1.2 Machine Learning Explainability (MLX)

[0033] The method herein generates local explanations of the ML model 160. As discussed later herein, a local explanation is an explanation of the reasoning I2 of the ML model 160 for a tuple T5 to be explained, which may be known or new. As discussed below, the corpus 110 and / or the ML model 160 participate in a series of stages, including: training of the ML model 160 and MLX calls, which generate the neighborhood 140 based on the tuple T5 to be explained before generating the local explanation.

[0034] In various scenarios, for various reasons, the tuple T5 and its inference I2 and / or ML model 160 are examined for interpretation. The MLX herein may provide any combination of the following functions:

[0035] Explainability: The ability to explain the local reason why the inference I2 occurs in the tuple T5 to be explained

[0036] Interpretability: The level at which humans can understand the explanation

[0037] What-If explanation: understanding how a change in the tuple T5 to be explained may or may not lead to the same inference I2

[0038] Model-agnostic explanations: explanations treat the ML model 160 as a black box, rather than using internal properties of the ML model 160 to guide the explanation

[0039] For example, explanations may be required to comply with regulatory requirements. Likewise, explanations may reveal edge cases that caused the ML model 160 to fail, requiring retraining using different data or a different hyperparameter configuration.

[0040] 1.3 Corpus of raw tuples

[0041] Training of ML model 160 requires a training corpus containing training tuples. In various embodiments, the training corpus is or is not corpus 110. In various embodiments, training of ML model 160 is unsupervised or supervised, meaning that the tuples of the training corpus are unlabeled or each tuple is labeled with a corresponding known correct inference. In any case, ML model 160 has been trained on Figure 1 Training was conducted.

[0042] The corpus 110 may or may not be used for any training, validation, and testing of the ML model 160. In essence, the original tuple 121 is a small portion of the multi-dimensional problem space, where each feature F1-F7 provides a corresponding dimension, which the ML model 160 can map to inferences, which will provide additional dimensions to the multi-dimensional solution space.

[0043] 1.4 Corpus Metadata

[0044] Corpus 110 includes metadata and data stored or accessible by computer 100. In an embodiment, corpus metadata is stored or cached in volatile memory, and corpus data is stored in a local or remote non-volatile storage device. Corpus data defines a portion of a multidimensional problem space and includes original tuples 121. Original tuples 121 are corresponding points in the multidimensional problem space. Original tuples 121 include individual original tuples T1-T4, which together contain original values ​​122, and original values ​​122 include individual values ​​V1-V17. Each original tuple 121 contains the corresponding value of each feature F1-F7. For example, as shown in the figure, the value of feature F1 in original tuples T1-T2 is value V1.

[0045] Corpus metadata generalizes or otherwise describes the corpus data. Corpus metadata includes features F1-F7 that may describe tuple 150, which is shown in a dashed outline to indicate that tuple 150 may be any single tuple in tuples 121, 141, or T5.

[0046] 1.5 Feature Engineering

[0047] Tuple 150 contains a corresponding value for each of features F1-F7. In an embodiment, tuple 150 is or is used to generate a feature vector that is accepted by ML model 160 and contains a more or less densely encoded corresponding value of features F1-F7. Each of features F1-F7 has a corresponding data type. For example, features F1 and F3 may have the same data type. The data type may be of various types: a) a number that is an integer or real number, b) a primitive type that can be easily encoded as a number, such as a Boolean value or a text character, c) a sequence of discrete values ​​with a semantic order (such as a month), such as a text literal, which can be easily encoded as a corresponding number preserving the original order, or d) an enumeration of categories of different categorical values ​​that are semantically unordered.

[0048] Categories are prone to discontinuities, which may or may not significantly destabilize the ML model 160, such that different categorical values ​​for the same feature may or may not result in very different inferences being generated by the ML model 160. A categorical feature may be hash-encoded as a single number in the feature vector, or n-hot or 1-hot encoded as multiple numbers. For example, 1-hot encoding generates 1 for categorical values ​​that actually appear in the tuple, and also generates 0 for each possible categorical value that does not appear in the tuple.

[0049] Tuple 150 may represent various objects in various embodiments. For example, tuple 150 may be or represent a network packet, a record (such as a database table row), or a log entry (such as a line of text in a console output log file). Similarly, features F1-F7 may be corresponding data fields, attributes, or columns that may appear in each object instance.

[0050] Inference 170 is shown in dashed outline to indicate that inference 170 can be any individual inference that replaces inference 143 or inference I2 of tuple T5 to be explained. In some examples, inference 170 can be a binary classification or anomaly score that indicates whether tuple 150 is abnormal (such as based on a threshold). When ML model 160 detects an anomaly in a production environment, an alert can be generated to trigger a manual or automated security response, such as terminating a session or network connection, denying tuple 150 further processing, and / or logging, diverting, and / or alerting tuple 150 for more intensive manual or automated inspection and analysis.

[0051] 1.6 Sampling Neighborhood

[0052] ML model 100 generates or has previously generated inference I2 for the original tuple T5 to be interpreted. In an embodiment that classifies tuple 150 into one of four mutually exclusive classes, inference 170 may be any of inferences I1-I4. However, the accuracy of ML model 160 may not be perfect, sometimes causing inference 170 to be wrong and not match the label of tuple 150, which is actually the known correct class of tuple 150.

[0053] Each original tuple 121 may be a point in the multidimensional problem space defined by features F1-F7. Although there may be hundreds of thousands of original tuples 121, each of which may be a different combination of feature F1-F7 values ​​and a different point in the multidimensional problem space, most or almost all possible points in the multidimensional problem space will not appear in the original tuple 121. Therefore, for most or almost all possible points in the multidimensional problem space, the inference 170 is unknown. Therefore, the global interpretation based on the original tuple 121 may have limited accuracy, especially because the known points in the multidimensional problem space are often separated by regions with many possible tuples, and the inference 170 of these tuples is unknown.

[0054] The computer 100 generates a local explanation that is more accurate than the global explanation as follows. Inference 170 depends on the values ​​of features F1-F7 in tuple 150. The accuracy of the local explanation can be improved by focusing the generation of explanations on a neighborhood of possible points around the tuple T5 to be explained in the multidimensional problem space. Neighborhood 140 uses a sampling of the original value 122 to explore the interval (locale) around the tuple T5 to be explained as follows.

[0055] Neighborhood 140 generates permuted tuples 141 based on original values ​​122, which are probabilistic variants of tuple T5 to be interpreted. Perturbations require that new tuples be made into imperfect copies of old tuples. Permutations differ from other kinds of perturbations because permutations do not require randomly generated values ​​for features and do not require predefined meaningless special values ​​for features, such as null or zero.

[0056] 1.7 Beyond LIME and Kernel SHAP

[0057] How the prior art generates perturbation tuples depends on how these perturbation tuples are used to detect feature importance and generate ML explanations (MLX), as described later in this paper. Other kinds of perturbations cannot achieve the benefits of permutation due to the following design limitations.

[0058] For example, Shapley additive explanation (SHAP) is proposed in the non-patent literature (NPL) "Aunified approach to interpreting model predictions" by Scott Lundberg et al., Advances In Neural Information Processing Systems 30 (2017), which is incorporated herein in its entirety. Due to SHAP's additive approach, SHAP's so-called "missing value" perturbations are predefined meaningless special values ​​of features, such as null values ​​or zeros, which may confuse and destabilize the inference of the ML model 160.

[0059] SHAP is not reliable in other ways. Most embodiments of ML models 160 are nonlinear, such as neural networks. As explained later in this document, related features are grouped into super-features that correspond to natural modes. The distribution of values ​​for related features is generally not independent, which means that there may be correlations between features. If SHAP is used with a nonlinear ML model or with non-independent features, then SHAP is not reliable unless perturbation tuples are exhaustively generated based on combinations that may be difficult to compute.

[0060] As another example, Locally Interpretable Model-Agnostic Explanations (LIME) was proposed by Marco Ribeiro et al. in NPL "Why should I trust you? Explaining the predictions of any classifier" published in the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016), which is incorporated herein in its entirety. The perturbations of LIME require so-called "grayed-out" values, which are equivalent to the "missing values" of SHAP.

[0061] In a manner discussed later in this article, embodiments of this article can combine SHAP or LIME or both (i.e., kernel SHAP proposed in SHAP NPL) with the following novel permutation techniques. Prior art perturbations of LIME and kernel SHAP tend to generate unrealistic perturbation tuples because missing values ​​(e.g., null values ​​or zeros) are always or almost always unrealistic. The lack of realism is exacerbated when multiple features have missing values ​​in the perturbation tuple, which is unavoidable in SHAP.

[0062] Realistic combination of 1.8 values

[0063] To some extent, the lack of realism can be mitigated by using natural values ​​for perturbation rather than predefined blank values. For example, the perturbation values ​​can be randomly generated from the natural range of the feature. If the feature is the wheel count of vehicles, and vehicles range from single-wheeled vehicles to freight trucks, there may be value(s) that are in the range but still unnatural, such as five wheels.

[0064] Even if the values ​​are restricted to natural values, such as the values ​​that feature F5 actually appears in the original values ​​122, a set of perturbed tuples may have an unrealistic distribution. For example, values ​​V10-V11 are the only values ​​of feature F5 in the original values ​​122, but since value V10 dominates feature F5 in the original values ​​122, it would be unrealistic to generate perturbed tuples that have equal counts of values ​​V10-V11 of feature F5, respectively. In other words, realistic perturbed feature values ​​may still result in the generation of unrealistic neighborhoods with realistic perturbed tuples.

[0065] Even if the value distribution is preserved, the composite values ​​of the feature combinations may not be realistic. For example, feature F1 may be the score of the team that won the American professional football game, and feature F2 may be the score of the team that lost the game. A score of 6-0 or 6-1 is possible, but a score of 1-1 is naturally impossible.

[0066] In these various approaches, the perturbation tuples may have unrealistic or unnatural values ​​or combinations of values, and the neighborhoods composed of realistic perturbation tuples may themselves be unrealistic due to unrealistic or unnatural frequencies. The permutation herein avoids these pitfalls. Thus, neighborhood 140 is more realistic than the prior art and provides the benefits discussed later herein.

[0067] 1.9 Example Super Features

[0068] The permutation herein is based on super features 131-133, each of which contains disjoint (i.e., non-intersecting, mutually exclusive) subsets of features F1-F7. Each super feature contains multiple features. A feature is contained in exactly one super feature. For example, super feature 131 contains features F1-F2. Super features 131-132 contain features of different corresponding counts.

[0069] Super features are based on modalities, which are information domains containing related features that have a common origin or purpose, or describe objects or components of objects. Super features correspond one to one with modalities. Table 1 below provides examples of modalities, super features, and features that may appear in a tuple, each of which represents a corresponding database statement in the log of a database server.

[0070]

[0071]

[0072] In Table 1 above, for super-feature 131, feature F1 may be or include a portion of an Open Database Connectivity (ODBC) or Java ODBC (JDBC) uniform resource locator (URL) used to establish a network connection and database session. Example connection string portions include standard URL portions (e.g., protocol, server host, and network port number) and ODBC / JDBC-specific portions in the path or query parameters, such as the name of the database, schema, or user account. Feature F2 may indicate how old the database session in which the database statement was issued is.

[0073] In Table 1 above, for super features 132, feature F3 may be a one-hot encoding of a dialect of a structured query processing language (SQL) of a database statement, such as data definition language (DDL), data manipulation language (DML), data query language (DQL), and transaction control language (TCL). Feature F4 may be a one-hot encoding of verbs such as SELECT, INSERT, DELETE, UPDATE, CREATE, DROP, GRANT, BEGIN, and COMMIT of a database statement. Feature F5 may be an n-hot encoding of a state or context of a database statement, such as outside a transaction, inside a marked transaction, an auto-committed transaction, and a prepared statement.

[0074] In Table 1 above, for super feature 133, feature F6 may be a return code of a database statement, such as an error code. Feature F7 may be a count of rows in a result set returned by the database statement.

[0075] Other examples not shown in Table 1 include schema super features, which contain features such as n-hot encodings of database tables referenced by database statements. Query condition super features may contain features describing the WHERE clause, such as a specified join count, a LIMIT clause for results, sort direction, and a DISTINCT keyword.

[0076] 1.10 Replacement of Super Features

[0077] As discussed above, the substitution is based on super features 131-133, such as those in Table 1 above. Substitution tuple 141 contains multiple (e.g., three as shown) substitution variants of the tuple T5 to be interpreted for each super feature 131-133. Each substitution tuple is almost a perfect copy of the tuple T5 to be interpreted, except that the value of one of the super features 131-133 is substituted (i.e., not a copy). Substitution 144 displays "YES" to indicate that super feature 132 is substituted in the subset of substitution tuples 141 shown. However, substitution tuple 141 also contains an equal number of undisplayed tuples that replace each other super features 131 and 133, respectively. In other words, the tuple count of substitution tuple 141 can be a multiple of the count of super features 131-133.

[0078] The substitution 144 shows "NO" for the super features 131 and 133, which means that the super features 131 and 133 in the substitution tuples P1-P2 and P4 are not substituted. The values ​​of the super features 131-133 of the substitution tuples P1-P2 and P4 are respectively displayed in the corresponding rows of the value 142. The values ​​of the super features 131 and 133 of the substitution tuples P1-P2 and P4 are both displayed as "T5", which means that the values ​​of the super features 131 and 133 of the substitution tuples P1-P2 and P4 are the same as the value of the tuple T5 to be interpreted.

[0079] Super features 131 and 133 contain features F1-F2 and F6-F7, respectively. Therefore, the values ​​of features F1-F2 and F6-F7 are the same in tuples T5, P1-P2, and P4. For example, the value of feature F6 is V14 in all tuples T5, P1-P2, and P4.

[0080] Since the super feature 132 of the replacement tuples P1-P2 and P4 is replaced, the value of the super feature 132 of the replacement tuples P1-P2 and P4 is not taken from the tuple T5 to be interpreted, but is taken from the corresponding original tuples 121 of the replacement tuples P1-P2 and P4, respectively. For example, the value of the super feature 132 of the replacement tuple P2 is displayed as "T2", which means that the values ​​of the super feature 132 of the tuples P2 and T2 are the same.

[0081] Super feature 132 includes features F3-F5. Therefore, the values ​​of features F3-F5 are the same in tuples P2 and T2. For example, the value of feature F3 is V6 in both P2 and T2.

[0082] The permuted values ​​of super-feature 132 are randomly sampled from the values ​​of super-feature 132 in original values ​​122. For example, as shown for super-feature 132 in values ​​142, permuted tuples P1-P2 and P4 have values ​​of super-feature 132 from original tuples T1-T2 and T4, respectively. Thus, the distribution of values ​​of super-feature 132 in permuted tuples P1-P2 and P4 is constrained by the same value range as the original values ​​of super-feature 132, and should have more or less the same value frequency probability distribution.

[0083] Since the sampling is random, some statistical distortion may occur. For example, for some or all of the super features 131-133, some of the original tuple 131 may not be sampled. For example, for the super feature 132, the original tuple T3 is not sampled.

[0084] In an embodiment, the random selection entails generating a real number between zero and one (including or excluding zero and one), and such real number can be scaled to fit within an integer range limited by the count of the original tuple 121. For example, for the original tuples T1-T4, the random number can be scaled to a range of 0-3, respectively. In various embodiments, the permuted tuple should not match: a) the tuple T5 to be interpreted, nor should it match b) any other permuted tuple. For example, if a randomly generated permuted value results in such a match, another permuted value can be randomly and repeatedly generated until a unique permuted tuple is generated.

[0085] For example, neighborhood 140 may be designed to generate only permuted tuples with different super-feature value combinations. Tuple T5 to be interpreted (and therefore value 142) may contain values ​​of the unpermuted super-feature that do not occur for that super-feature and / or any other super-feature in original value 122. For example, as shown, the values ​​of super-feature 133 containing features F6-F7 in tuple T5 to be interpreted are V14, V17, which is a combination that does not occur for super-feature 133 in original value 122.

[0086] Likewise, the tuple to be interpreted T5 (and therefore the value 142) may contain a value for a feature in the unpermuted superfeature that does not appear for that feature and / or any other feature in the original value 122. For example, as shown, the value V0 for feature F4 in the tuple to be interpreted T5 appears, but does not appear for any of the features F1-F7 in the original value 122. In these ways, the tuple to be interpreted T5 may not be present in the original tuple 121.

[0087] 1.14 Proxy Model

[0088] LIME and Kernel SHAP have (and embodiments herein may have) a proxy model 145, which is an additional ML model to the non-ML model 160. The ML model 160 is referred to herein as a black-box model, an opaque model, a target model, or a model to be explained. The proxy model 145 is referred to herein as an interpretable model due to human understandability.

[0089] In practice, ML model 160 learns a vast multidimensional space (e.g., at least corpus 110) and is more complex than surrogate model 145, which only needs to learn neighborhood 140 and tuples T5 to be explained. Surrogate model 145 can have a straightforward and simplified architecture, such as a decision tree or linear regression.

[0090] Less complex (i.e., easier to understand) MLX interpretations are the motivation for LIME and Kernel SHAP, but are only partially implemented in the prior art due to the lack of superfeatures. The dimensionality reduction achieved by superfeatures is an improvement in two important ways.

[0091] First, the MLX interpretation in this paper is based on super-features, not features, and presents super-features to users, not features. Due to the increased granularity of interpretation details, the complexity of this MLX interpretation is reduced. The reduced complexity of presentation means faster human comprehension.

[0092] For example, the prior art kernel SHAP can generate an explanation of a mixture of unrelated features, such as by ranking the top four features in descending order of importance, such as features F5, F7, F4, F3. A similar explanation herein can instead designate super-feature 132 as the most important feature, without the user having to disentangle features of different natural modalities, such as feature F7 that is not in super-feature 132.

[0093] Second, the proxy model 145 may or may not use feature vectors of tuples 141 and T5 that directly encode super features instead of features. In this context, super features may be encoded as row offsets in the original tuple 121, while the tuple T5 to be interpreted is an additional row. For example, the original tuples T1-T4 may be identified by corresponding integer offsets 1-4, and the tuple T5 to be interpreted may be identified by offset 5.

[0094] 1.11 Proxy feature vector

[0095] Thus, if there are three super features, then the value 142 can be encoded as an integer triplet for use by the proxy model 145, even though there may be dozens of features within each super feature. For example, the permutation tuple P2 shown as the triplet T5, T2, T5 in the value 142 can be encoded as the integer triplet 5, 2, 5. This dimensionality reduction means that the proxy model 145 can be trained in less time and less space without sacrificing accuracy.

[0096] Unlike the prior art LIME, dimensionality reduction through super-features is generally applicable to any problem domain. For example, LIME NPL proposes two exemplary problem domains, natural language processing (NLP) and computer vision, which are carefully expressed using Boolean encodings, which are actually n-hot encodings such that each Boolean value represents a feature. Although convenient, one Boolean value for each feature F1-F7 is only suitable for simple problems and lacks the general applicability of the feature engineering in this paper.

[0097] For example, feature F6 has four different values ​​V12-V15 in the original value 122, which cannot be encoded as a single Boolean value. For example, LIME's exemplary NLP is based on Boolean encoding, which is less informative than bag-of-words, which itself is known to have very low information content. Unlike LIME, our feature engineering provides dimensionality reduction without incurring inherent losses.

[0098] Super features are not the same as LIME's super pixels in several ways. Each tuple 121, 141, and T5 has the same count of super features. LIME images in the same corpus do not have a fixed count of super pixels, so that a first image may have more super pixels than a second image of the same size as the first image. LIME's super pixel count depends on the content of the image. The count of super features does not depend on the original value 122.

[0099] 1.12 Training the Agent Model

[0100] The inferences of ML models 145 and 160 for permuted tuple 141 are shown in permuted inferences 143. For example, for permuted tuple P1, ML models 145 and 160 consistently infer inference I3. However, ML models 145 and 160 function differently in three important ways, which may cause ML models 145 and 160 to generate different corresponding inferences I1 and I2 for the same perturbation tuple P4.

[0101] First, ML model 160 is more complex than proxy model 145, which may lead to different inferences. Second, ML models 145 and 160 have different corresponding training corpora, such that proxy model 145 is trained only with tuples 141 and T5, while ML model 160 is trained with at least original tuple 121. Third, ML models 145 and 160 are trained with corresponding corpora containing very different corresponding tuple counts.

[0102] Two columns are shown in permuted inference 143. Prior to supervised training of surrogate model 145, ML model 160 is applied to permuted tuples 141 to generate corresponding inferences in the left column of permuted inference 143.

[0103] Supervised training of the proxy model 145 uses the neighborhood 140 and the tuple to be explained T5 as the training corpus, and uses the left column of permuted inferences 143 and the inferences I2 of the tuple to be explained T5 as the training labels. During supervised training, the proxy model 145 is applied to the permuted tuple 141 to generate training inferences in the right column of permuted inferences 143. The training inferences may be more or less inaccurate.

[0104] That is, for some permuted tuples, the left and right columns of permuted inferences 143 may not agree. The fact that there is disagreement about the inferences for the permuted tuples represents a training loss, which is used by the proxy model 145 to reinforce learning. For example, the loss can be measured and used to adjust internal parameters (e.g., coefficients) in the proxy model 145, such as through backpropagation, to accomplish learning, as explained later herein. As a result of the training, the proxy model 145 should become able to generate the same inferences that the ML model 160 would generate for most permuted tuples 141.

[0105] Since the generation of permuted tuples 141 and the training of proxy models 145 are faster than the prior art LIME and kernel SHAP, MLX explanations can be generated more or less instantly on demand for any tuple to be explained T5. For example, MLX herein can be used by customer support personnel during a live phone call without delay.

[0106] 1.13 Importance score

[0107] The possibility of only one super-feature being replaced in a permutation tuple is discussed earlier herein. In an embodiment, several (i.e., multiple) super-features are replaced in each permutation tuple in some permutation tuples. For example, a first subset of permutation tuples 141 may replace one super-feature, a second subset may replace two super-features, and a third subset may replace three super-features. Similarly, super-features 131-132 may be replaced in some permutation tuples in permutation tuple 141, and super-features 132-133 may be replaced in other permutation tuples in permutation tuple 141. This increases the density of neighborhood 140, thereby improving the accuracy of proxy model 145.

[0108] Since the proxy model 145 is interpretable, the importance of the super features 131-133 can be more or less directly extracted from the proxy model 145. For example, if the proxy model 145 is a decision tree or a linear regression, then the coefficients of the linear regression or the number of levels of the decision tree can be more or less directly used as the importance scores of the super features 131-133.

[0109] The super features 131-133 may be ranked (i.e., sorted) by importance score to establish a relative order of the impact of the super features 131-133 on the inferred operation of the ML model 160. For example, super feature 132 may have a greater impact on the operation of the ML model 160 than super feature 131. Therefore, super feature 132 should have greater explanatory power for MLX than super feature 131. The local explanation of the ML model 160 will emphasize super feature 132 rather than super feature 131.

[0110] In the memory of the computer 100, the local explanation can be based on or include a data structure that ranks the super features 131-133 by importance score and / or excludes the least influential super features or a threshold count of super features with importance scores below a threshold. For example, the local explanation can be limited to the variable counts of the top two most influential super features or super features with importance scores of at least 0.4. Explanation generation will be discussed later in this document.

[0111] 1.14 Tuple Weighting

[0112] Earlier in this paper, we discussed the possibility that all permuted tuples 141 and their permuted inferences 143 have the same impact on the trained proxy model 145, and thus on feature importance, which may be inaccurate. For the same reason that local explanations may be more accurate than global explanations, if permuted tuple P4 is more similar to the tuple to be explained T5 than permuted tuple P1, then permuted tuple P4 may have greater explanatory power than permuted tuple P1.

[0113] To this end, the permuted tuples 141 and their training losses as discussed above may be weighted averaged according to the degree of similarity of each permuted tuple 141 to the tuple T5 to be explained. Likewise, the training of the proxy model 145 may be based on the weighting of the permuted tuples 141 .

[0114] In an embodiment, the tuple distance (i.e., the difference from the tuple T5 to be explained) can be measured to weight the permuted tuple based on the count of the permuted super-features. For example, the permuted tuple has only one permuted super-feature 132, and the distance between tuples T5 and P2 can be 1.

[0115] The tuple weight may be inversely proportional to the distance, and the smaller the distance, the larger the weight should be. In an embodiment, the tuple distance used for weighting may be based on the sum of the feature counts in the permuted super-features. For example, the permuted tuple P2 has only one permuted super-feature 132, which contains three features F3-F5, and the distance between the tuple T5 and P2 may be 3.

[0116] In an embodiment, regardless of which super-feature(s) are permuted, the tuple-weighted distance is measured as the sum of the differences in the values ​​of each of the features F1-F7 or the multi-dimensional Cartesian distance (i.e., less than the sum). Regardless of whether the distance is measured for a feature or a super-feature, the distance of the unpermuted feature or super-feature is always zero when contributing to the sum or count.

[0117] Feature values ​​can be normalized in various ways for feature encoding and / or distance calculations. For example, the difference / distance between adjacent months of the year may be smaller than the distance between adjacent days of the week. Mahalanobis distance is based on feature values ​​normalized to the standard deviation of the corresponding feature. For example, if most prices are in the range of $1 to $100, then the Mahalanobis distance between $30 and $90 may be smaller than the distance between $90 and $110.

[0118] 2.0 Example Importance Scoring Processing

[0119] Figure 2 1 is a flowchart depicting an example process that an embodiment of the computer 100 may perform to generate a permuted tuple 141 based on permuting the tuple to be explained T5 to provide machine learning (ML) explainability (MLX) for the black box ML model 160. Figure 1 discuss Figure 2 .

[0120] In this context, there are three phases in the MLX lifecycle of the ML model 160. Each MLX invocation requires a different tuple T5 to be interpreted. Regardless of how many tuples T5 to be interpreted there are, the preparation design phase occurs only once and requires step 201 of defining super-features 131-132, each of which contains a corresponding disjoint subset of features.

[0121] The experience phase generates one or more corresponding inferences of different tuples to be explained T5 and requires step 202. The order of steps 201-202 can be reversed. In step 202, ML model 160 infers inference 170 to explain tuples to be explained T5.

[0122] For example, the tuple T5 to be explained may be archived and historical, or may be real-time and streaming. Step 202 itself may be real-time or historical. For example, the inference I2 of the tuple T5 to be explained may or may not be archived.

[0123] The runtime stage calculates the importance of the super features 131-133 for a given tuple T5 to be interpreted, and may optionally perform MLX. The runtime stage needs to repeat steps 203-208 for each tuple T5 to be interpreted. Steps 203-207 calculate the importance of the super features 131-133. Repeating steps 203-204 for each super feature 131-133 may completely fill the left column of the value 142 and permutation tuple 141.

[0124] Step 203 randomly selects a replacement value from the original values ​​122 of the super feature in the original tuple 121. The random selection of step 203 only requires randomly generated offsets in the array of original tuples 121. The random selection of step 203 does not require identification of values ​​within the original values ​​122.

[0125] Step 204 generates permuted tuples, each of which is based on the tuple to be explained T5 and the corresponding permuted value of the super feature. Step 204 regenerates a different neighborhood 140 for each tuple to be explained T5.

[0126] Step 204 generates one or two feature vectors that redundantly represent the permuted tuples. The target feature vector has a format accepted by the ML model 160. In a first embodiment, the ML model 160 and the proxy model 145 accept the same target feature vector format.

[0127] In a second embodiment, step 204 generates another feature vector that has a format accepted by proxy model 145, but is different from the format of the target feature vector. The proxy feature vector contains only integer offsets to the tuples, such as offsets 1-4 for the original tuple 121 and offset 5 for the tuple T5 to be interpreted. For example, the permuted tuple P2 shown as the triple T5, T2, T5 in value 142 can be encoded as the integer triple 5, 2, 5. Unlike LIME and kernel SHAP, the proxy feature vector does not contain Boolean values.

[0128] The difference between these two feature vectors is as follows. The proxy feature vector is narrower because it contains fewer elements and bytes than the target feature vector. The proxy feature vector represents homogeneous data as tuples of integer offsets. The target feature vector instead represents heterogeneous data because features F1-F7 may have different data types.

[0129] Step 204 generates a target feature vector containing the values ​​of features F1-F7. For example, the permuted tuple P2 shown as the triple T5, T2, T5 in value 142 may be encoded as a tuple having seven values ​​V8, V8, V5, V6, V7, V14, V17, where only the bold values ​​V5-V7 are taken from the original tuple T2 of superfeature 132. The values ​​of features F1-F2 and F6-F7 of superfeatures 131 and 133 are instead taken from the tuple T5 to be interpreted.

[0130] Various embodiments of step 204 may populate the target feature vector by accessing the original values ​​122 in different ways. The original values ​​122 may be a two-dimensional array from which the feature values ​​may be accessed using the row offset of the original tuple and the column offset of the feature. The value of a super-feature may be accessed using the column offset range of the feature in the super-feature.

[0131] The raw values ​​122 can be sliced ​​vertically into a two-dimensional array containing feature values ​​for the corresponding super feature, or a one-dimensional array for each feature. The feature values ​​within the slice can be accessed as discussed above.

[0132] In step 205 , ML model 160 infers a corresponding permutation inference for each permutation tuple. Step 205 is repeated to completely fill in the left column of permutation inferences 143 based on permutation tuples 141 .

[0133] Step 206 generates and supervises training proxy model 145 using neighborhood 140 and tuple to be explained T5 as a training corpus encoded into a proxy feature vector and using the left column of inference 143 and inference I2 of tuple to be explained T5 as training labels for supervision.

[0134] Step 206 regenerates and retrains the proxy model 145 for each different tuple T5 to be explained. Each instance of the proxy model 145 is different due to the use of a correspondingly different neighborhood 140 for training. The supervised training of the proxy model 145 is discussed earlier in this document.

[0135] Step 207 calculates the respective importance of the super features 131-133 based on the proxy model 145. In an embodiment, steps 206-207 are combined so that the importance of the super features 131-133 is established by training the proxy model 145. Since the proxy model 145 is interpretable, step 207 can more or less directly extract the importance of the super features 131-133 from the proxy model 145. For example, if the proxy model 145 is a decision tree or a linear regression, then the coefficients of the linear regression or the number of levels of the decision tree can be more or less directly used as the importance scores of the super features 131-133.

[0136] For horizontal expansion, steps 203-205 can be executed concurrently by separate execution contexts for each hyper-feature 131-133. The execution contexts can be based on lightweight threads, operating system processes, hyperthreads, processing cores of a central processing unit (CPU), CPUs, coprocessors, and / or separate computers.

[0137] In an embodiment, the importance scores of super features 131-133 are used for feature selection for target ML model design and configuration, rather than for MLX. For example, if super feature 131 is the least important, then features F1-F2 of super feature 131 may be excluded from feature selection. Embodiments not used for MLX do not perform step 208.

[0138] Step 208 generates a local explanation of the ML model 160 based on the tuple to be explained T5. Step 208 may: a) rank the super features 131-133 and retain their corresponding importance scores, b) discard the neighborhood 140, and c) generate a local MLX explanation for the tuple to be explained T5, as discussed previously herein.

[0139] Hardware Overview

[0140] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. The special-purpose computing device can be hard-wired to perform the present technology, or can include a digital electronic device that is permanently programmed to perform the present technology, such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), or can include one or more general-purpose hardware processors that are programmed to perform the present technology according to program instructions in firmware, memory, other storage devices, or combinations. This special-purpose computing device can also combine customized hard-wired logic, ASICs or FPGAs with customized programming to implement the present technology. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hard-wired and / or program logic to implement the technology.

[0141] For example, Figure 3 3 is a block diagram illustrating a computer system 300 upon which embodiments of the present invention may be implemented. Computer system 300 includes a bus 302 or other communication mechanism for communicating information and a hardware processor 304 coupled with bus 302 for processing information. Hardware processor 304 may be, for example, a general purpose microprocessor.

[0142] Computer system 300 also includes a main memory 306, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 302 for storing information and instructions to be executed by processor 304. Main memory 306 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 304. Such instructions, when stored in a non-transitory storage medium accessible to processor 304, cause computer system 300 to become a special-purpose machine customized to perform the operations specified in the instructions.

[0143] Computer system 300 also includes a read only memory (ROM) 308 or other static storage device coupled to bus 302 for storing static information and instructions for processor 304. A storage device 310, such as a magnetic disk, optical disk, or solid state drive, is provided and coupled to bus 302 for storing information and instructions.

[0144] The computer system 300 may be coupled via the bus 302 to a display 312, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to the bus 302 for communicating information and command selections to the processor 304. Another type of user input device is a cursor control 316, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 304 and for controlling cursor movement on the display 312. Such input devices typically have two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), to allow the device to specify a position in a plane.

[0145] Computer system 300 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in conjunction with the computer system, makes computer system 300 a special purpose machine or programs computer system 300 to be a special purpose machine. According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. These instructions may be read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequences of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0146] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical disks, magnetic disks, or solid-state drives, such as storage device 310. Volatile media include dynamic memory, such as main memory 306. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a hole pattern, RAM, PROM and EPROM, FLASH-EPROM, NVRAM, any other memory chip, or cassette tapes.

[0147] Storage media are distinct from but may be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires comprising bus 302. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0148] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 304 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 300 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on bus 302. Bus 302 carries the data to main memory 306, from which processor 304 retrieves and executes the instructions. The instructions received by main memory 306 may optionally be stored on storage device 310 before or after execution by processor 304.

[0149] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides bidirectional data communication coupled to network link 320, wherein network link 320 is connected to local network 322. For example, communication interface 318 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides data communication connection to a telephone line of a corresponding type. As another example, communication interface 318 can be a LAN card that provides data communication connection to a compatible local area network (LAN). Wireless links can also be implemented. In any such implementation, communication interface 318 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

[0150] The network link 320 typically provides data communication to other data devices through one or more networks. For example, the network link 320 can provide a connection to a host computer 324 or to data equipment operated by an Internet Service Provider (ISP) 326 through a local network 322. The ISP 326, in turn, provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 328. Both the local network 322 and the Internet 328 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 320 and through the communication interface 318 are example forms of transmission media that carry the digital data to or from the computer system 300.

[0151] Computer system 300 can send messages and receive data, including program code, through the network(s), network link 320, and communication interface 318. In the Internet example, server 330 can transmit the requested code for an application program through Internet 328, ISP 326, local network 322, and communication interface 318.

[0152] The received code may be executed by processor 304 as it is received, and / or stored in storage device 310 or other non-volatile storage for later execution.

[0153] Software Overview

[0154] Figure 4 4 is a block diagram of a basic software system 400 that may be used to control the operation of computing system 300. Software system 400 and its components, including their connections, relationships, and functions, are intended to be exemplary only and are not meant to limit implementation of the example embodiment(s). Other software systems suitable for implementing the example embodiment(s) may have different components, including components with different connections, relationships, and functions.

[0155] Software system 400 is provided for directing the operation of computing system 300. Software system 400, which may be stored on system memory (RAM) 306 and fixed storage (eg, hard disk or flash memory) 310, includes a kernel or operating system (OS) 410.

[0156] OS 410 manages low-level aspects of computer operation, including managing the execution of processes, memory allocation, file input and output (I / O), and device I / O. One or more application programs, represented as 402A, 402B, 402C ... 402N, may be "loaded" (e.g., transferred from fixed storage 310 into memory 306) for execution by system 400. Applications or other software intended for use on computer system 300 may also be stored as a downloadable set of computer-executable instructions, for example, for download and installation from an Internet location (e.g., a Web server, app store, or other online service).

[0157] The software system 400 includes a graphical user interface (GUI) 415 for receiving user commands and data in a graphical (e.g., "click" or "touch gesture") manner. In turn, these inputs can be operated by the system 400 according to instructions from the operating system 410 and / or (one or more) applications 402. The GUI 415 is also used to display the results of the operations from the OS 410 and (one or more) applications 402, and the user can provide additional input or terminate the session (e.g., log out).

[0158] The OS 410 may execute directly on the bare hardware 420 (e.g., the processor(s) 304) of the computer system 300. Alternatively, a hypervisor or virtual machine monitor (VMM) 430 may be inserted between the bare hardware 420 and the OS 410. In this configuration, the VMM 430 acts as a software "buffer" or virtualization layer between the OS 410 and the bare hardware 420 of the computer system 300.

[0159] VMM 430 instantiates and runs one or more virtual machine instances ("clients"). Each client includes a "guest" operating system (such as OS 410), and one or more applications designed to execute on the guest operating system, such as application(s) 402. VMM 430 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0160] In some instances, VMM 430 can allow a guest operating system to run as if it were running directly on bare hardware 420 of computer system 300. In these instances, the same version of a guest operating system that is configured to execute directly on bare hardware 420 can also execute on VMM 430 without modification or reconfiguration. In other words, VMM 430 can, in some instances, provide full hardware and CPU virtualization to guest operating systems.

[0161] In other examples, the guest operating system may be specially designed or configured to execute on VMM 430 to improve efficiency. In these examples, the guest operating system is "aware" that it is executing on a virtual machine monitor. In other words, VMM 430 may provide paravirtualization to the guest operating system in some cases.

[0162] A computer system process includes an allocation of hardware processor time, and an allocation of (physical and / or virtual) memory, an allocation of memory for storing instructions executed by the hardware processor, for storing data generated by the execution of instructions by the hardware processor, and / or for storing hardware processor state (e.g., contents of registers) between allocations of hardware processor time when the computer system process is not running. A computer system process runs under the control of an operating system, and may run under the control of other programs executing on the computer system.

[0163] cloud computing

[0164] The term "cloud computing" is used generally herein to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows resources to be rapidly provisioned and released with minimal management effort or service provider interaction.

[0165] A cloud computing environment (sometimes referred to as a cloud environment or just a cloud) can be implemented in a variety of different ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or the general public. In contrast, a private cloud environment is generally intended for use by or within a single organization only. A community cloud is intended to be shared by several organizations within a community; while a hybrid cloud includes two or more types of clouds (e.g., private, community, or public) bound together by data and application portability.

[0166] In general, the cloud computing model enables some of those responsibilities that might previously have been provided by an organization's own information technology department to be delivered instead as a service layer within the cloud environment for use by consumers (internal or external to the organization, depending on the public / private nature of the cloud). Depending on the specific implementation, the precise definition of the components or features provided by or within each cloud service layer may vary, but common examples include: Software as a Service (SaaS), where consumers use software applications running on the cloud infrastructure, while the SaaS provider manages or controls the underlying cloud infrastructure and applications. Platform as a Service (PaaS), where consumers can develop, deploy, and otherwise control their own applications using software programming languages ​​and development tools supported by the PaaS provider, while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything under the runtime execution environment). Infrastructure as a Service (IaaS), where consumers can deploy and run arbitrary software applications, and / or provide processing, storage, networking, and other basic computing resources, while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS), where the consumer uses a database server or database management system running on a cloud infrastructure, while the DbaaS provider manages or controls the underlying cloud infrastructure and applications.

[0167] The above basic computer hardware and software and cloud computing environment are presented to illustrate the basic underlying computer components that can be used to implement the (one or more) example embodiments. However, the (one or more) example embodiments are not necessarily limited to any particular computing environment or computing device configuration. Instead, the (one or more) example embodiments can be implemented in any type of system architecture or processing environment that a person skilled in the art will understand in view of this disclosure as being able to support the features and functions of the (one or more) example embodiments presented herein.

[0168] Machine Learning Models

[0169] A machine learning model is trained using a specific machine learning algorithm. Once trained, the input is applied to the machine learning model to make a prediction, which may also be referred to as a predicate output or output in this article. The attributes of the input may be referred to as features, and the values ​​of the features may be referred to as feature values ​​in this article.

[0170] A machine learning model includes a model data representation or model artifact. The model artifact includes parameter values, which may be referred to herein as theta values, and is applied to the input by the machine learning algorithm to generate a predicted output. Training a machine learning model requires determining theta values ​​of the model artifact. The structure and organization of theta values ​​depends on the machine learning algorithm.

[0171] In supervised training, training data is used by a supervised training algorithm to train a machine learning model. The training data includes inputs and "known" outputs. In an embodiment, the supervised training algorithm is an iterative process. In each iteration, the machine learning algorithm applies the model artifacts and the inputs to generate a predicate output. The error or variance between the predicate output and the known output is calculated using an objective function. In practice, the output of the objective function indicates the accuracy of the machine learning model based on the specific state of the model artifacts in the iteration. By applying an optimization algorithm based on the objective function, the theta value of the model artifact can be adjusted. An example of an optimization algorithm is gradient descent. Iterations can be repeated until the desired accuracy is achieved or some other criterion is met.

[0172] In software embodiments, when a machine learning model is referred to as receiving inputs, executing, and / or generating outputs or predicates, a computer system process executing a machine learning algorithm applies a model artifact to an input to generate a predicted output. A computer system process executes a machine learning algorithm by executing software configured to cause the algorithm to be executed. When a machine learning model is referred to as performing an action, a computer system process executes a machine learning algorithm by executing software configured to cause the action to be performed.

[0173] Inference requires a computer to apply a machine learning model to inputs such as feature vectors to generate inferences by processing the inputs and content of the machine learning model in an integrated manner. Inference is data-driven based on data contained in the machine learning model, such as learning coefficients. In this article, this is referred to as inference of the machine learning model, and in practice, this is the execution of the machine learning algorithm that processes the machine learning model by the computer.

[0174] Classes of problems that machine learning (ML) excels at include clustering, classification, regression, anomaly detection, prediction, and dimensionality reduction (i.e., simplification). Examples of machine learning algorithms include decision trees, support vector machines (SVMs), Bayesian networks, random algorithms such as genetic algorithms (GAs), and connectionist topologies such as artificial neural networks (ANNs). Implementations of machine learning can rely on matrices, symbolic models, and hierarchical and / or associative data structures. Parameterized (i.e., configurable) implementations of best-of-breed machine learning algorithms can be found in open source libraries such as Google's TensorFlow for Python and C++ or Georgia Tech's MLPack for C++. Shogun is an open source C++ML library with adapters for several programming languages, including C#, Ruby, Lua, Java, Matlab, R, and Python.

[0175] Artificial Neural Networks

[0176] An artificial neural network (ANN) is a machine learning model that models at a high level a system of neurons interconnected by directed edges. An overview of neural networks is described in the context of hierarchical feedforward neural networks. Other types of neural networks share characteristics of the neural networks described below.

[0177] In a layered feedforward network such as a multilayer perceptron (MLP), each layer comprises a set of neurons. A layered neural network comprises an input layer, an output layer, and one or more intermediate layers known as hidden layers.

[0178] The neurons in the input layer and the output layer are referred to as input neurons and output neurons, respectively. Neurons in the hidden layer or the output layer may be referred to as activation neurons in this article. Activation neurons are associated with activation functions. The input layer does not contain any activation neurons.

[0179] From each neuron in the input layer and hidden layer, there may be one or more directed edges pointing to an activated neuron in a subsequent hidden layer or output layer. Each edge is associated with a weight. An edge from a neuron to an activated neuron represents the input from the neuron to the activated neuron, as adjusted by the weight.

[0180] For a given input to a neural network, each neuron in the neural network has an activation value. For an input neuron, the activation value is simply the input value for that input. For an activation neuron, the activation value is the output of the corresponding activation function that activated the neuron.

[0181] Each edge from a particular neuron to an activated neuron represents that the activation value of the particular neuron is the input to the activated neuron, i.e., the input to the activation function of the activated neuron, as adjusted by the weight of the edge. Thus, an activated neuron in a subsequent layer represents that the activation value of the particular neuron is the input to the activation function of the activated neuron, as adjusted by the weight of the edge. An activated neuron may have multiple edges pointing to the activated neuron, each edge representing (as adjusted by the weight of the edge) that the activation value from the starting neuron is the input to the activation function of the activated neuron.

[0182] Each activated neuron is associated with a bias. To generate an activation value for an activated neuron, the neuron's activation function is applied to the weighted activation value and the bias.

[0183] Illustrative Data Structures for Neural Networks

[0184] Artifacts of a neural network may include matrices of weights and biases. Training a neural network may iteratively adjust the matrices of weights and biases.

[0185] For layered feedforward networks, as well as other types of neural networks, the artifacts may include one or more matrices W of edges. Matrix W represents the edges from layer L-1 to layer L. Assuming the number of neurons in layers L-1 and L is N[L-1] and N[L] respectively, then the dimensions of matrix W are N[L-1] columns and N[L] rows.

[0186] The bias for a particular layer L may also be stored in a matrix B having N[L] rows and one column.

[0187] The matrices W and B may be stored in RAM memory as vectors or arrays, or as a comma-separated set of values ​​in memory. When the artifact is stored persistently in a persistent storage device, the matrices W and B may be stored as comma-separated values ​​in a compressed and / or serialized form or other suitable persistent form.

[0188] The specific input applied to the neural network includes the value of each input neuron. The specific input can be stored as a vector. The training data includes multiple inputs, each input is called a sample in the sample set. Each sample includes the value of each input neuron. The sample can be stored as a vector of input values, and the multiple samples can be stored as a matrix, where each row in the matrix is ​​a sample.

[0189] When input is applied to a neural network, activation values ​​are generated for the hidden and output layers. For each layer, the activation values ​​can be stored in a column of a matrix A that has a row for each neuron in the layer. In a vectorized approach for training, the activation values ​​can be stored in a matrix that has a column for each sample in the training data.

[0190] Training a neural network requires storing and processing additional matrices. The optimization algorithm generates matrices of derivative values ​​used to adjust the weights W and bias B matrices. Generating derivative values ​​can use and require matrices that store intermediate values ​​generated when calculating the activation values ​​of each layer.

[0191] The number of neurons and / or edges determines the size of the matrices required to implement the neural network. The fewer the number of neurons and edges in the neural network, the smaller the matrices and the amount of memory required to store the matrices. In addition, a smaller number of neurons and edges reduces the amount of computation required to apply or train the neural network. Fewer neurons means fewer activation values ​​that need to be calculated and / or fewer derivative values ​​that need to be calculated during training.

[0192] The properties of the matrix used to implement the neural network correspond to neurons and edges. A cell in the matrix W represents a particular edge from the L-1 layer to a neuron in the L layer. An activated neuron represents the activation function for that layer, which includes the activation function. The activated neurons in the L layer correspond to the rows of the matrix W for the weights of the edge between the L layer and the L-1 layer and the columns of the matrix W for the weights of the edge between the L layer and the L+1 layer. During the execution of the neural network, the neurons also correspond to one or more activation values ​​stored in the matrix A for that layer and generated by the activation function.

[0193] ANN is suitable for vectorization of data parallelism, which can utilize vector hardware, such as single instruction multiple data (SIMD), such as utilizing a graphics processing unit (GPU). Matrix partitioning can achieve horizontal scaling, for example, utilizing symmetric multiprocessing (SMP), such as utilizing a multi-core central processing unit (CPU) and / or multiple coprocessors (such as GPU). Feedforward calculations in ANN can occur with only one step in each neural layer. The activation values ​​in one layer are calculated based on the weighted propagation of the activation values ​​of the previous layer, so that the values ​​are calculated in turn for each subsequent layer, such as sequentially calculating the values ​​for each subsequent layer, such as utilizing the corresponding iterations of a for loop. Layering imposes the ordering of non-parallelizable calculations. Therefore, the network depth (i.e., the number of layers) can cause computational delays. Deep learning requires giving multilayer perceptrons (MLPs) many layers. Each layer implements data abstraction, while complex (i.e., multidimensional with several inputs) abstractions require multiple layers of cascade processing. Implementations of ANNs based on reusable matrices and matrix operations for feed-forward processing are readily available and parallelizable in neural network libraries such as Google's TensorFlow for Python and C++, OpenNN for C++, and Fast Artificial Neural Networks (FANN) from the University of Copenhagen. These libraries also provide model training algorithms such as backpropagation.

[0194] Back Propagation

[0195] The output of the ANN may be approximately correct. For example, an ANN that recognizes letters may mistake an I for an L because the letters have similar features. The correct output may have a particular value(s), while the actual output may have a slightly different value. The arithmetic or geometric difference between the correct output and the actual output may be measured as the error according to a loss function, such that zero represents error-free (i.e., perfectly accurate) behavior. For any edge in any layer, the difference between the correct output and the actual output is the delta value.

[0196] Backpropagation requires distributing the error backwards through the layers of the ANN to all connected edges within the ANN in varying amounts. The propagation of the error results in adjustments to the edge weights, which depend on the gradient of the error on each edge. The gradient of an edge is calculated by multiplying the error delta of the edge by the activation value of the upstream neuron. When the gradient is negative, the greater the magnitude of the error contributed by the edge to the network, the more the weight of the edge should be reduced, which is negative reinforcement. When the gradient is positive, positive reinforcement requires increasing the weight of the edge whose activation reduces the error. The edge weights are adjusted based on a percentage of the gradient of the edge. The steeper the gradient, the greater the adjustment. Not all edge weights are adjusted by the same amount. As model training continues with additional input samples, the error of the ANN should decrease. Training can stop when the error stabilizes (i.e., stops decreasing) or disappears below a threshold (i.e., approaches zero). Christopher M. Bishop teaches example mathematical formulas and techniques for feed-forward multilayer perceptrons (MLPs), including matrix operations and backpropagation, in the related reference "EXACT CALCULATION OF THE HESSIAN MATRIX FOR THE MULTI-LAYER PERCEPTRON".

[0197] Model training can be supervised, or unsupervised. For supervised training, the expected (i.e., correct) output is already known for each example in the training set. The training set is configured by pre-assigning classification labels to each example (e.g., by a human expert). For example, a training set for optical character recognition may have blurred photos of individual letters, and an expert may pre-label each photo according to which letter is shown. Error calculation and back-propagation occur as explained above.

[0198] Autoencoder

[0199] Unsupervised model training is more involved because the desired output needs to be discovered during training. Unsupervised training can be more easily adopted because no human experts are needed to pre-label training examples. Therefore, unsupervised training saves manpower. The natural way to implement unsupervised training is to use an autoencoder, which is a type of ANN. The autoencoder is used as an encoder / decoder (codec) with two sets of layers. The first set of layers encodes the input example into a condensed code that needs to be learned during model training. The second set of layers decodes the condensed code to regenerate the original input example. The two sets of layers are trained together as a combined ANN. The error is defined as the difference between the original input and the input regenerated after decoding. After sufficient training, the decoder outputs the original input almost exactly.

[0200] For each input example, the autoencoder relies on a condensed code as an intermediate format. Perhaps counterintuitively, the intermediate condensed codes do not initially exist, but emerge only through model training. Unsupervised training can implement a vocabulary of intermediate encodings based on features and distinctions of unexpected correlations. For example, which examples and which labels are used during supervised training can depend on a human expert's somewhat unscientific (e.g., anecdotal) or otherwise incomplete understanding of the problem space. Unsupervised training, on the other hand, discovers a suitable intermediate vocabulary more or less entirely based on statistical trends that reliably converge to optimality with enough training due to internal feedback generated by regenerating decodings. Techniques for unsupervised training of autoencoders for anomaly detection based on reconstruction error are taught in the non-patent literature (NPL) Jinwon An et al., "VARIATIONAL AUTOENCODER BASEDANOMALYDETECTION USING RECONSTRUCTION PROBABILITY", Special Lecture on IE, December 27, 2015;2(1):1-18.

[0201] Principal component analysis

[0202] Principal Component Analysis (PCA) provides dimensionality reduction by utilizing and organizing mathematical correlation techniques such as normalization, covariance, eigenvectors, and eigenvalues. PCA combines aspects of feature selection by eliminating redundant features. PCA can be used for prediction. PCA can be used in conjunction with other ML algorithms.

[0203] Random Forest

[0204] A random forest or random decision forest is a family of learning methods that construct a collection of randomly generated nodes and decision trees during a training phase. The different decision trees of the forest are constructed so that each is randomly restricted to only a specific subset of the feature dimensions of the dataset, such as by feature bootstrapping aggregation (bagging). Therefore, as the decision trees grow, the accuracy of the decision trees increases without being forced to overfit the training data as would occur if the decision trees were forced to learn all feature dimensions of the dataset. The predictions can be calculated based on the average (or other integral, such as soft max) of the predictions from the different decision trees.

[0205] Random forest hyperparameters can include: number-of-trees-in-the-forest (the number of trees in the forest), maximum-number-of-features-considered-for-splitting-a-node (the maximum number of features considered for splitting a node), number-of-levels-in-each-decision-tree (the number of levels in each decision tree), minimum-number-of-data-points-on-a-leaf-node (the minimum number of data points on a leaf node), method-for-sampling-data-points (the method for sampling data points), and so on.

[0206] In the foregoing specification, embodiments have been described with reference to numerous specific details that vary from implementation to implementation. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive. The sole and exclusive indicator of the scope of the invention, and what the applicants intend as the scope of the invention, is the literal and equivalent range of the set of claims issuing from this application, in the specific form in which such claims issue, including any subsequent corrections.

Claims

1. A computer-implemented method comprising: defining a plurality of super-features, each super-feature comprising corresponding non-overlapping feature subsets of the plurality of features; A machine learning (ML) model infers a specific inference for a specific tuple based on the plurality of features; For each superfeature in the plurality of superfeatures: Randomly select a plurality of replacement values ​​from the original values ​​of the super features in the plurality of original tuples based on the plurality of features, generating a plurality of permutation tuples, wherein each permutation tuple of the plurality of permutation tuples is based on the particular tuple and a corresponding permutation value of the plurality of permutation values, and The ML model infers a corresponding permutation inference for each permutation tuple in the plurality of permutation tuples; training a proxy model based on the permuted inference; For each super feature of the plurality of super features, importance of the super feature is calculated based on the proxy model.

2. The method of claim 1 , further comprising accessing a value of a super feature of an original tuple of the plurality of original tuples based on at least one item selected from the group consisting of: the offset of the primitive tuple in the array of primitive tuples, a series of offsets to the values ​​of the feature subsets of the superfeature stored contiguously in this original tuple, and offset into an array consisting of the values ​​of a subset of features of the superfeatures of the plurality of original tuples.

3. The method of claim 1, wherein at least one of the following is selected from the group consisting of: The plurality of super features correspond to a plurality of modalities respectively, and A first super-feature of the plurality of super-features includes more features than a second super-feature of the plurality of super-features.

4. The method of claim 1, further comprising generating a local explanation of the ML model based on the specific tuple.

5. The method of claim 4, wherein the generating the local explanation of the ML model is based on the importance of at least one super-feature of the plurality of super-features. 6 . The method of claim 5 , wherein the local interpretation comprises ranking at least two super features of the plurality of super features based on their importance.

7. The method of claim 1, wherein at least one of the group consisting of: The plurality of original tuples do not include the specific tuple, the value of a specific super feature in the plurality of super features of the plurality of original tuples does not include the value of the specific super feature in the specific tuple, and The values ​​of the plurality of features in the plurality of original tuples do not include a value of a specific feature in the plurality of features in the specific tuple.

8. The method of claim 1, wherein a specific super-feature of the plurality of super-features represents one selected from the group consisting of: a database connection, a database table, a query condition, a result of a database statement, and a kind of a database statement.

9. The method of claim 1, wherein the training the agent model comprises populating at least one item selected from the group consisting of: identifying a feature vector of at least one original tuple of the plurality of original tuples, a feature vector identifying said particular tuple, Characteristic vectors that do not contain Boolean values, a character vector containing at least one array offset, and A feature vector containing only integers.

10. The method of claim 1, wherein at least one of the following is selected from the group consisting of: ML models are unsupervised, and The plurality of original tuples are unlabeled.

11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause the method of any one of claims 1-10 to be performed.

12. A system comprising: a plurality of computing devices, each computing device in the plurality of computing devices having one or more processors; The plurality of computing devices have access to one or more non-transitory computer-readable media storing instructions that, when executed by the plurality of computing devices, cause performance of the method as set forth in any one of claims 1-10.

Images