Distribution-based risk management in classification models

Abstract

A system and method for risk management in expert systems are provided. The distribution of feature values ​​in the expert system's training dataset and each of several baseline datasets are determined, and a dataset of Kullback-Leibler divergence values ​​is provided by the Kullback-Leibler divergence between the baseline dataset and the training dataset. Measures of central tendency and statistical variance are determined for the Kullback-Leibler divergence value dataset, and a Kullback-Leibler divergence threshold is determined from these values ​​and a desired confidence interval for the new dataset. The Kullback-Leibler divergence value between the feature values ​​of the new dataset and the feature values ​​of the training dataset is determined, and if this Kullback-Leibler divergence value exceeds the threshold, it is determined that the second dataset represents an unacceptable deviation.

Description

Technical Field

[0001] The present invention relates to machine learning systems, and more particularly to distribution-based risk management in classification models. This application claims priority to U.S. Provisional Patent Application No. 63 / 521,134, filed on June 15, 2023, and U.S. Patent Application No. 18 / 393,263, entitled "DISTRIBUTION BASED RISK MANAGEMENT IN NATURAL LANGUAGE MODELS," filed on December 21, 2023, the entire contents of both of which are incorporated herein by reference.

Background Art

[0002] Over the past few years, machine learning (ML) has made significant progress, and the impact this new technology can have on industries as a whole has become clear. In this process, supervised ML and natural language processing (NLP) have been shown to be effective in automating tasks in research environments with problems such as multi-class text classification. A major prerequisite in supervised machine learning models is that the environment in which the model operates is relatively stable, i.e., the set of examples provided to the model for training and validation is similar to the set of examples that the system will classify. Supervised learning models function well in an environment similar to the one in which the model was trained and tested, but if the environment changes while the machine learning model is in operation, the model may provide incorrect predictions.

Summary of the Invention

[0003] According to one aspect of the present invention, the method includes determining the distribution of feature values ​​in a first dataset of a sample group used to train an expert system, and providing a dataset of Kullback-Leibler divergence values ​​by determining the Kullback-Leibler divergence between the feature values ​​of the first dataset and the feature values ​​of the baseline dataset for each of a plurality of baseline datasets received by the expert system. A measure of central tendency and a measure of statistical variance of the dataset of Kullback-Leibler divergence values ​​are determined, and a Kullback-Leibler divergence threshold representing an unacceptable deviation of the distribution of feature values ​​in a new dataset from the distribution of feature values ​​in the first dataset of the sample group is determined from a desired confidence interval for the new dataset, the measure of central tendency of the dataset of Kullback-Leibler divergence values, and the measure of statistical variance of the dataset of Kullback-Leibler divergence values. For the second dataset received by the expert system, a Kullback-Leibler divergence value is determined between the feature values ​​of the second dataset and the feature values ​​of the first dataset. If the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset exceeds the Kullback-Leibler divergence threshold, it is determined that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

[0004] According to another aspect of the present invention, the system includes a processor and a non-temporary computer-readable medium for storing computer-readable instructions executable by the processor, the non-temporary computer-readable medium for storing the computer-readable instructions for providing an expert system and a risk management component. The risk management component includes a training set characterization component that determines the distribution of feature values ​​in a first dataset of sample sets used to train the expert system, and a baseline characterization component. The baseline characterization component provides a dataset of Kullback-Leibler divergence values ​​by determining the Kullback-Leibler divergence between the feature values ​​of the first dataset and the feature values ​​of the baseline dataset for each of a plurality of baseline datasets received in the expert system; determines a measure of central tendency and a measure of statistical variance in the dataset of Kullback-Leibler divergence values; and determines a Kullback-Leibler divergence threshold representing an unacceptable deviation of the distribution of feature values ​​in the new dataset from the distribution of feature values ​​in the first dataset of the sample group, based on a desired confidence interval for the new dataset, the measure of central tendency in the dataset of Kullback-Leibler divergence values, and the measure of statistical variance in the dataset of Kullback-Leibler divergence values. The monitoring system determines the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset for the second dataset received by the expert system, and determines that the second dataset represents an unacceptable deviation of the sample group from the distribution of the feature values ​​of the first dataset if the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset exceeds the Kullback-Leibler divergence threshold.

[0005] A method for risk management in a document classification system is provided according to yet another aspect of the present invention. The distribution of feature values ​​in a first dataset of a sample group used to train an expert system is determined. Each sample in the first dataset of the sample group represents a document. For each of a plurality of baseline datasets received by the expert system, a dataset of Kullback-Leibler divergence values ​​is provided by determining the Kullback-Leibler divergence between the feature values ​​of the first dataset and the feature values ​​of the baseline dataset, and the mean and standard deviation of the dataset of Kullback-Leibler divergence values ​​are determined. A Kullback-Leibler divergence threshold, representing an unacceptable deviation of the distribution of feature values ​​in a new dataset from the distribution of feature values ​​in the first dataset of the sample group, is determined from a desired confidence interval for the new dataset, the mean and standard deviation of the dataset of Kullback-Leibler divergence values, and Chebyshev's inequality. For the second dataset received by the expert system, a Kullback-Leibler divergence value is determined between the feature values ​​of the second dataset and the feature values ​​of the first dataset. If the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset exceeds the Kullback-Leibler divergence threshold, it is determined that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 shows a classification system that utilizes distribution-based risk management. [Figure 2] Figure 2 shows an example of one implementation of a risk management system that can be used with the system in Figure 1. [Figure 3]Figure 3 shows a system for classifying documents that utilize distribution-based risk management. [Figure 4] Figure 4 shows a method for evaluating the input dataset in terms of its fit to the dataset used to train the expert system. [Figure 5] Figure 5 is a schematic block diagram showing an exemplary system of hardware components for implementing the systems and methods described herein. [Modes for carrying out the invention]

[0007] As used herein, the “measure of central tendency” of a dataset refers to a descriptive statistic that represents the central or representative value of the dataset. The measure of central tendency may include, but is not limited to, the mean (e.g., arithmetic mean, geometric mean, harmonic mean, or other mean), median, median range, mid-hinge, interquartile mean, and trimean of the dataset. The mean may be a weighted mean, truncated mean, or windsorized mean.

[0008] As used herein, “measure of statistical dispersion” for a dataset refers to a descriptive statistic that measures how much the data varies around the median or central tendency. Measures of statistical dispersion may include, but are not limited to, the variance, standard deviation, range, interquartile range, median absolute deviation, mean absolute deviation, average absolute deviation, entropy, Gini coefficient, relative mean difference, and coefficient of variation of a dataset.

[0009] As used herein, the "Kullback-Leibler divergence" or "KL divergence" (DKL) represents the divergence between a first discrete probability distribution P and a second discrete probability distribution Q defined in the sample space, and is defined by the following equation:

[0010]

number

[0011] The systems and methods provided herein offer a probabilistic approach to monitoring supervised machine learning (ML) models, such as ML models applying natural language processing (NLP). This incorporation achieves a novel probabilistic method for measuring the operating environment and providing signals to alert to potential environmental changes for the model. Figure 1 shows a classification system 100 that utilizes distribution-based risk management. System 100 includes a processor 102 and a non-temporary computer-readable medium 110 that stores computer-readable instructions executed by the processor 102. The executable instructions stored in the non-temporary computer-readable medium 110 include a network interface 112 for system 100 to communicate with other systems (not shown) via a network connection (e.g., an internet connection and / or a connection to an internal network). In the illustrated example, the other systems may include a database system that stores new samples acquired for classification. System 100 can also be implemented as a virtual or cloud server, in which case the processor 102 and the non-temporary computer-readable medium 110 may be shared by other applications.

[0012] The feature extractor 114 receives new samples, i.e., samples not presented in the model's training set, and extracts several features for use in the expert system 116. Features can be any numerical values ​​representing samples relevant to the classification process. The features may vary depending on the nature and format of the training samples (e.g., images, audio files, text, etc.), the subject matter of the training samples, and the classes relevant to the classification process. In the illustrated system, the expert system 116 uses the extracted features to classify new samples into one or more categories (or classes). The expert system 116 can utilize one or more pattern recognition algorithms, for example, implemented as classification and regression models, each classifying the report into one of the categories by analyzing the extracted features or a subset of the extracted features. The selected categories may be provided to the user, for example, as a record associated with the sample on a relevant display (not shown), or stored on a non-temporary computer-readable medium 110.

[0013] Expert system 116 and pattern recognition algorithms of any configuration can be trained on a set of training data, where each sample in the training data set contains values ​​for each feature in the feature set and its associated output class. Generally, training data can be generated by applying a feature extractor 114 to a set of samples having known output classes. To represent the initial distribution of the training set, the initial distribution of the values ​​for each feature and output class in the sample set can be obtained and stored.

[0014] In one example, each category is represented in a one-vs-all configuration by a separate machine learning model. In this example, each machine learning model is trained as a binary classifier that distinguishes the code category associated with that machine learning model from all other classes. In this example, the output of the machine learning models is a categorical or continuous parameter that reflects the likelihood that a sample will be appropriately classified using the code represented by the machine learning model. A mediation element may be used to provide consistent results from multiple machine learning models, for example, as the class with the highest continuous output value or the highest confidence level in the categorical output. In one example, the mediation element itself may be implemented as a classification model that takes the outputs of multiple models as features and generates one or more categories of a sample set.

[0015] Machine learning models can be trained with training data representing various classes of interest. In one implementation, a machine learning model may use a different model architecture, a different set of associated features, and different preprocessing methods (or no preprocessing). While the training process of a given model may vary depending on its implementation, training generally involves statistical aggregation of the training data to one or more parameters associated with the output classes. A variety of methods can be used in the model, including support vector machines (SVMs), regression models, self-organizing maps, fuzzy logic systems, data fusion processes, boosting and bagging methods, rule-based systems, or artificial neural networks (ANNs).

[0016] For example, an SVM classifier may utilize a set of functions called hyperplanes to conceptually divide the boundaries within an N-dimensional feature space, where each of the N dimensions represents one associated feature of the feature vector. The boundaries define the range of feature values ​​associated with each class. Thus, the output classes and associated confidence values ​​can be determined depending on where a given input feature vector is located in the feature space relative to the boundaries. The SVM classifier places the training data within the defined feature space by utilizing a user-specified kernel function. In the most basic implementation, the kernel function can be a radial basis function, but the systems and methods described herein can utilize any of several linear or nonlinear kernel functions.

[0017] An ANN classifier consists of multiple nodes with multiple interconnections. Values ​​from feature vectors are provided to multiple input nodes. Each input node provides these input values ​​to a layer of one or more intermediate nodes. A given intermediate node receives one or more output values ​​from the preceding node. The received values ​​are weighted according to a set of weight values ​​established during the classifier training. The intermediate node transforms its received values ​​into a single output according to the transfer function in the node. For example, an intermediate node may sum the received values ​​and provide the sum to a binary step function. The final layer of the node provides confidence values ​​for the ANN's output classes, where each node has an association value representing its confidence in one of the classifier's relevant output classes.

[0018] Regression models produce continuous results by applying a set of weight values ​​to various functions, most commonly linear functions, on extracted features. Generally, regression features can be categorical values ​​represented, for example, 0 or 1, or continuous values. In logistic regression, the model's output represents the log odds that the source of the extracted features belongs to a given class. In binary classification tasks, these log odds can be used directly as confidence values ​​for class belonging, or they can be transformed by a logistic function into the probability of class belonging given the extracted features.

[0019] Rule-based classifiers select an output class by applying a set of logical rules to extracted features. Generally, the rules are applied in an orderly manner, with the logical outcome at each stage influencing the analysis in subsequent steps. Specific rules and their order can be determined from one or all of the following: training data, analogies based on past examples, and existing domain knowledge. An example of a rule-based classifier is a decision tree algorithm, where the values ​​of features in a feature set are compared to corresponding thresholds in a hierarchical tree structure to select a class for a feature vector. Random forest classifiers are an improved version of the decision tree algorithm using bootstrap aggregation, or "bagging." In this method, multiple decision trees are trained on random samples of the training set, and the average (e.g., mean, median, or mode) result across the multiple decision trees is returned. For classification tasks, since the results from each decision tree are categorical values, the mode result may be used.

[0020] The risk management component 118 monitors the characteristics of the samples classified by the expert system 116 to confirm that the operating environment of the expert system 116 has not changed significantly. For this purpose, the risk management component continuously records the distribution of the feature values of the samples input for classification in the expert system 116, and compares a subset of the feature values with the baseline feature value distribution to determine whether the current distribution of the feature values of the input samples matches the baseline feature value distribution or indicates that the environment from which the feature values are extracted has changed. Such environmental changes may occur naturally due to changes in the characteristics of the samples to be classified, or may occur as a result of malicious acts that change the behavior of the data set received by the feature extractor 114 or the feature extractor.

[0021] The risk management component 118 first constructs a baseline data set sample distribution based on a comparison between the input data set and the baseline feature value distribution representing the data used for training and testing the expert system 116. Note that the size of the input data set, which includes both the baseline data set samples and the test target samples, can be standardized to include at least a certain number of samples in order to obtain feature values that provide a meaningful distribution. Next, a Kullback–Leibler divergence value data set for the baseline data set samples can be provided by calculating the Kullback–Leibler divergence for each data set. Next, descriptive statistical values including a measure of central tendency and a measure of statistical dispersion can be calculated for this data set of baseline data set samples to represent the distribution of the input data set. This may be a static distribution obtained by a single execution, or may be windowed to reflect the gradual changes in the environment in the distribution. In one implementation, the standard deviation and mean of the data set of Kullback–Leibler divergence values are used to represent the distribution.

[0022] The inventors have found that the distribution of input datasets does not necessarily satisfy the normality assumption that underlies many common methods in statistical analysis. Therefore, Chebyshev's inequality can be used to determine the Kullback-Leibler divergence threshold for rejecting datasets. For a broad class of probability distributions, Chebyshev's inequality is 1 / k of the distribution values. 2 The following describes a deviation from the mean by more than k times the standard deviation. Therefore, for any desired confidence interval, a threshold number of standard deviations representing sample datasets outside the confidence interval, i.e., the Kullback-Leibler divergence threshold, can be determined. Once this threshold is calculated from baseline dataset samples, the Kullback-Leibler divergence for newly received datasets can be determined and compared to the threshold to determine whether the newly received datasets deviate from the training datasets to an unacceptable degree. Therefore, if the Kullback-Leibler divergence value of the input dataset exceeds the determined threshold, an alert may be presented to the user, and, optionally, classification of further samples may be stopped until further input is received from the user. Furthermore, the expert system 116 can be retrained using either the original training set or a new training set representing the modified environment, the expert system can be taken offline, or, in response to the above decisions, the expert system can be restored from a saved backup.

[0023] FIG. 2 shows an implementation example of a risk management component 200 available in the system of FIG. 1. The illustrated risk management component 200 can be implemented as software instructions stored on a computer-readable medium and executed by an associated processor (not shown). The risk management component 200 includes a training set characterization component 202 that determines the distribution of feature values in a training data set of a sample group used to train an expert system. Specifically, the training set characterization component 202 generates a discrete probability distribution of feature values in the training data set. For this purpose, any continuous or discrete feature having a number of possible values can be binned within a selection range so that the generation of a discrete probability distribution becomes possible.

[0024] The baseline characterization component 204 determines the Kullback-Leibler divergence threshold, which represents the unacceptable deviation of the feature distribution in a new dataset from the feature distribution in the training dataset of the sample group. To this end, the baseline characterization component 204 evaluates multiple baseline datasets received by the expert system and generates a discrete probability distribution for each baseline dataset. The baseline datasets may be samples received during normal operation of the expert system, and groups of similar size are accumulated so that they can be used as various baseline datasets. For each of the multiple baseline datasets, the baseline characterization component 204 may provide a set of Kullback-Leibler divergence values ​​by calculating the Kullback-Leibler divergence values ​​between the feature values ​​of the baseline dataset and the feature values ​​of the training dataset. The baseline characterization component 204 then determines measures of central tendency and statistical variance, e.g., mean and standard deviation, for the set of Kullback-Leibler divergence values. From these values ​​and a desired confidence interval for the new dataset, the baseline characterization component 204 determines the Kullback-Leibler divergence threshold. In one example, the threshold is determined according to Chebyshev's inequality.

[0025] The monitoring system 206 determines the Kullback-Leibler divergence value between the feature values ​​of the new dataset and the feature values ​​of the training dataset for any new dataset received by the expert system, and determines whether the new dataset represents an unacceptable deviation from the distribution of feature values ​​in the training dataset for the sample group. Specifically, if the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset exceeds a threshold, the deviation is considered unacceptable. The retraining component 208 retrains the expert system in response to the monitoring system's determination that the second dataset represents an unacceptable deviation from the distribution of feature values ​​in the first dataset for the sample group. For example, the retraining component 208 may restore the expert system from a backup or retrain it using the original training set to address malicious changes to the expert system and its surrounding infrastructure, or it may retrain the expert system using a new training set based on new data to address changes in the environment. In one example, the retraining component 208 operates at the direction of a user or group of users. For example, the monitoring system 206 may review the deviation of the received dataset from the training dataset, determine the cause and severity of the problem causing the deviation, and alert the operations team to instruct the retraining component 208 to take appropriate action. In this case, the retraining process may be monitored by the operations team to ensure that the expert system is properly retrained. In another example, the retraining component 208 performs an analysis of the problem causing the deviation before taking action to correct the deviation.

[0026] Figure 3 shows a system 300 for classifying documents using distribution-based risk management. Documents classified by system 300 may include unstructured or semi-structured documents having one or more free text fields. System 300 may classify each document into a category representing the subject of the document. System 300 includes a processor 302 and a non-temporary computer-readable medium 310 that stores computer-readable instructions executed by the processor 302. The executable instructions stored in the non-temporary computer-readable medium 310 include a network interface 312 for system 300 to communicate with other systems (not shown) via a network connection (e.g., an internet connection and / or a connection to an internal network). In the illustrated example, other systems may include a database system that stores documents retrieved for classification. System 300 can also be implemented as a virtual or cloud server, in which case the processor 302 and the non-temporary computer-readable medium 310 may be shared by other applications.

[0027] In the illustrated example, a document may be extracted from a database (not shown) or a user terminal (not shown) via a network interface 312 and provided to a text preprocessor 313. The text preprocessor 313 may apply various techniques to prepare the text for analysis by an automated system, including, but not limited to, combining multiple free text fields within a semi-structured document, de-case-sensitive removal of individual characters, removal of spaces and punctuation between words, splitting the text into tokens such as words or phrases, removing stop words (i.e., common words that do little to distinguish between categories, such as articles, common verbs (e.g., various conjugations of "to be" and "to do"), common prepositions (e.g., "for" and "to")), and stemming words lacking common prefixes and suffixes to their base forms.

[0028] The feature extractor 314 receives a document from the text preprocessor 313 and extracts several features from the document for use in the expert system 316. For this purpose, the feature extractor 314 may calculate the frequency of occurrence of various terms in the extracted text. In one implementation, a simple count of each token may be used. In another example, tokens are normalized according to the total number of related tokens found in a given document, which is referred to herein as the "normalized count occurrence". In a third implementation, bag-of-words features may be weighted using token frequencies according to term frequency - inverse document frequency (TFIDF), such that terms that occur relatively infrequently throughout the report are given a greater weight per occurrence than common terms. In practice, in a given implementation of system 300, some of these methods may be used as options for each classification model.

[0029] The feature extractor 314 can then use the calculated frequencies as part of one or more natural language processing algorithms for extracting data from unstructured text. In some implementations, category codes assigned by the individual who created the document may be given some weight in the classification task. In other implementations, categories are assigned based on the content of the free text, regardless of the original classification. One example is the bag-of-words method, in which each report is represented as a feature vector generated according to the frequency of terms in the report, either through simple counts, normalized frequency, or term frequency-inverse document frequency. In one implementation, the bag-of-words can be implemented using N-gram tokens, such that the dictionary of tokens used in the bag-of-words analysis includes not only individual words but also phrases of two or more words.

[0030] In another example, topic modeling techniques may be used to provide data for classification by identifying latent topics within the document's free text. Topic modeling is an unsupervised method for detecting these latent topics, which can be used as additional information for classifying events. In one example, feature extractor 314 may generate a document-word matrix in which each column represents a document, each row represents a term of interest, and each element represents the frequency of a given term in a given report. By applying truncated singular value decomposition (tSVD) analysis to the document-word matrix, two additional matrices may be generated that associate terms and documents with latent topics, along with a set of singular values ​​representing the latent topics. Truncation means retaining only the highest set of singular values ​​from the set of singular values. This technique is called latent semantic analysis, and the topics are called latent topics. Once a suitable set of latent topics is identified during training of system 100, feature extractor 314 may transform each report into a topic representation formed from latent topics that are expected to generate the terms observed in the report.

[0031] In one example, the feature extractor 314 may utilize latent semantic indexing, a generative topic model that discovers topics within text documents. In latent semantic indexing, a vocabulary of terms is either pre-selected or generated as part of the indexing process. A matrix is ​​generated representing the frequency of occurrence of each term in the vocabulary of terms within each document, such that each row of the matrix represents a term and each column represents a document. The frequencies can be generated as normalized frequencies or using term-inverse-document frequencies (TFIDF). Next, a dimensionality reduction technique is applied to this matrix to project the terms into a lower-dimensional latent semantic space. In the illustrated example, the dimensionality reduction technique is truncation singular value decomposition. Each document is then represented by the projection values ​​in the appropriate columns of the reduced matrix.

[0032] In other examples, word embedding techniques such as Word2Vec or document embedding techniques such as Doc2Vec may be used. In Word2Vec, a neural network with an input layer where each node represents a term is trained on pairs of adjacent words in a document to provide a classifier that identifies words that are likely to appear in close proximity to each other. The weights of the links between the input nodes representing a given word and the hidden layer can then be used to characterize the document content, including the semantic and syntactic relationships between words.

[0033] Paragraph Vector Distributed Memory (PV-DM) is an extension of word embedding techniques. In PV-DM, the context from each paragraph (or appropriate text) is taken as input to the model, and the specific context of that paragraph is represented by link weights associated with these inputs, which are generated for each paragraph as part of the training process. Thus, the model is trained to generate paragraph vectors by predicting words that are likely to appear in close proximity to each other for a given paragraph in a document, where each column represents the trained context for each paragraph in the document. This is averaged or concatenated with the word vectors of the document to generate a set of document features that captures the averaged embedding representation across the words and word sequences that appear.

[0034] In the illustrated system, the expert system 316 uses extracted features to classify new documents, i.e., event reports not presented in the model's training set, into one or more categories. The expert system 316 may utilize one or more pattern recognition algorithms, for example, implemented as classification and regression models, each of which analyzes extracted features or a subset of extracted features to classify the report into one of the categories. The selected categories may be provided to the user, for example, as a record associated with the document, on a relevant display (not shown), or stored on a non-temporary computer-readable medium 310. An example of such an expert system can be found in U.S. Patent Application Publication 2021 / 0357766, which is incorporated herein by reference in its entirety.

[0035] An expert system 316 and any component pattern recognition algorithm can be trained on a set of training data. Each sample in the training data set contains values ​​for each feature in the feature set and its associated output class. Generally, training data can be generated by applying a feature extractor 114 to documents with known output classes. To represent the initial distribution of the training set, the initial distribution of values ​​for each feature and output class in the documents can be obtained and stored. The training process of a given model may vary depending on its implementation, but training generally involves a statistical aggregation of the training data to one or more parameters associated with the output classes. A variety of methods can be used in the model, including support vector machines (SVMs), regression models, self-organizing maps, fuzzy logic systems, data fusion processes, boosting and bagging methods, rule-based systems, or artificial neural networks (ANNs).

[0036] The risk management component 318 monitors the characteristics of documents classified by the expert system 316 to ensure that the operating environment of the expert system 316 has not changed significantly. To this end, the risk management component continuously records the distribution of feature values ​​of the input samples for classification in the expert system 316 and compares a subset of feature values ​​with the baseline feature value distribution to determine whether the current distribution of feature values ​​of the input samples matches the baseline feature value distribution or indicates a change in the environment in which the features are extracted. Such environmental changes may occur naturally due to changes in the characteristics of the documents being classified, but they may also result from malicious actions that alter the behavior of the feature extractor 314 or the dataset received by the feature extractor.

[0037] The risk management component 318 first constructs a baseline dataset sample distribution based on a comparison of the input dataset with a baseline feature value distribution representing the data used for training and testing the expert system 316. The size of the input dataset, which includes both the baseline dataset samples and the test samples, may be standardized to include at least a certain number of samples in order to obtain feature values ​​that provide a significant distribution. Next, a dataset of Kullback-Leibler divergence values ​​for the baseline dataset samples may be provided by calculating the Kullback-Leibler divergence for each dataset. Then, to represent the distribution of the input dataset, descriptive statistics, including measures of central tendency and variance, may be calculated for this baseline dataset sample dataset. This may be a static distribution obtained through a single run, or it may be windowed to reflect gradual changes in the environment. In one implementation, the standard deviation and mean of the Kullback-Leibler divergence value dataset are used to represent the distribution.

[0038] The inventors have found that the distribution of input datasets does not necessarily satisfy the normality assumption that underlies many common methods in statistical analysis. Therefore, Chebyshev's inequality can be used to determine the Kullback-Leibler divergence threshold for rejecting datasets. For a broad class of probability distributions, Chebyshev's inequality is 1 / k of the distribution values. 2 The following describes a deviation from the mean of more than k times the standard deviation. Therefore, for any desired confidence interval, a threshold number of standard deviations representing sample datasets outside the confidence interval, i.e., the Kullback-Leibler divergence threshold, can be determined. Once this threshold is calculated from baseline dataset samples, the Kullback-Leibler divergence for newly received datasets can be determined and compared to the threshold to determine whether the newly received dataset deviates from the training dataset to an unacceptable degree. Therefore, if the Kullback-Leibler divergence value of the input dataset exceeds the determined threshold, an alert may be presented to the user, and, optionally, further document classification may be stopped until further input is received from the user.

[0039] Methods according to various aspects of the present invention can be better understood by referring to Figure 4, taking into account the structural and functional features described above. For the sake of simplicity, the method in Figure 4 is shown and described as being performed sequentially, but according to the present invention, some aspects may be performed in a different order than those shown and described herein and / or simultaneously with other aspects. Therefore, the present invention is not limited by the order shown. Furthermore, not all illustrated features are required to carry out a method according to one aspect of the present invention.

[0040] Figure 4 shows a method 400 for evaluating an input dataset in terms of its fit to the dataset used to train an expert system. In 402, the distribution of feature values ​​in a first dataset of the sample set used to train the expert system is determined. Fewer features than all features used by the expert system may be used for this analysis. In 404, a dataset of Kullback-Leibler divergence values ​​is provided by determining the Kullback-Leibler divergence between the feature values ​​in the first dataset and the feature values ​​in the baseline dataset for each of the multiple baseline datasets received by the expert system. The Kullback-Leibler divergence is asymmetric, and the Kullback-Leibler divergence value representing each dataset may be the Kullback-Leibler divergence of the baseline dataset from the first dataset, or the Kullback-Leibler divergence of the first dataset from the baseline dataset, or it may be the maximum, minimum, or mean of these values.

[0041] In 406, measures of central tendency and statistical variance for the Kullback-Leibler divergence dataset are determined. In one example, the mean and standard deviation of the Kullback-Leibler divergence value dataset are used. In 408, the Kullback-Leibler divergence threshold, which represents the unacceptable deviation of the distribution of feature values ​​in the new dataset from the distribution of feature values ​​in the first dataset of the sample group, can be determined from the desired confidence interval for the new dataset and the measures of central tendency and statistical variance for the Kullback-Leibler divergence value dataset. In one example, the confidence interval is determined from the mean and standard deviation of the Kullback-Leibler divergence value dataset and Chebyshev's inequality. For example, if a 99 percent confidence interval is desired, a threshold equal to the following equation may be used: where δ is the standard deviation of the Kullback-Leibler divergence value dataset.

[0042]

number

[0043] In step 410, the Kullback-Leibler divergence value is determined between the feature values ​​of the second dataset received by the system and the feature values ​​of the first dataset. In step 412, it is determined whether the Kullback-Leibler divergence value between the feature values ​​of the second dataset received by the system and the feature values ​​of the first dataset is below a threshold. If it is below the threshold (Y), in step 414, the user is notified that the second dataset is acceptable for classification, and the method returns to step 410 to receive another new dataset. If it is not below the threshold (N), it is determined that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​in the first dataset, and in step 416, an alert is issued to the user.

[0044] Figure 5 is a schematic block diagram showing an exemplary system 500 of hardware components capable of implementing examples of systems and methods disclosed herein. System 500 may include various systems and subsystems. System 500 may be a personal computer, laptop computer, workstation, computer system, appliance, application-specific integrated circuit (ASIC), server, Server BladeCenter®, server farm, etc.

[0045] System 500 may include a system bus 502, a processing unit 504, system memory 506, memory devices 508, 510, a communication interface 512 (e.g., a network interface), a communication link 514, a display 516 (e.g., a video screen), and an input device 518 (e.g., a keyboard, touchscreen, and / or mouse). System bus 502 may be capable of communicating with the processing unit 504 and the system memory 506. Additional memory devices 508, 510, such as hard disk drives, servers, standalone databases, or other non-volatile memory, may also be capable of communicating with system bus 502. System bus 502 interconnects the processing unit 504, memory devices 506-510, communication interface 512, display 516, and input device 518. In some examples, system bus 502 also interconnects with additional ports (not shown), such as Universal Serial Bus (USB) ports.

[0046] The processing unit 504 may be an arithmetic unit and may include an application-specific integrated circuit (ASIC). The processing unit 504 executes a set of instructions for implementing the operation of the various examples disclosed herein. The processing unit may include a processing core.

[0047] Additional memory devices 506, 508, and 510 may store data, programs, instructions, database queries in text or compiled form, and any other information that may be required to operate the computer. Memory 506, 508, and 510 may be implemented as computer-readable media (integrated or removable), such as memory cards, disk drives, compact discs (CDs), or servers accessible over a network. In some examples, memory 506, 508, and 510 may contain text, images, videos, and / or audio, some of which may be available in a human-readable format.

[0048] Additionally or alternatively, the system 500 may access external data sources or query sources via a communication interface 512 that can communicate with the system bus 502 and the communication link 514.

[0049] During operation, system 500 may be used to implement one or more parts of the system for risk management in the document classification system according to the present invention, in particular the feature extractor 114, the expert system 116, and the risk management component 118. Computer-executable logic for implementing the system for evaluating management reports resides, according to some examples, in one or more of the system memory 506 and memory devices 508, 510. Processing unit 504 executes one or more computer-executable instructions arising from the system memory 506 and memory devices 508, 510. As used herein, the term “computer-readable medium” refers to a medium involved in providing instructions to processing unit 504 for execution. This medium may be distributed across multiple separate assemblies operably connected to a common processor or a set of associated processors.

[0050] The above description provides specific details to give a good understanding of the embodiments. However, it is understood that the embodiments can be carried out without these specific details. For example, physical components may be shown in block diagrams to avoid obscuring the embodiments with unnecessary details. In other examples, well-known circuits, processes, algorithms, structures, and methods may be shown without unnecessary details to avoid obscuring the embodiments.

[0051] The implementation forms of the techniques, blocks, steps, and means described above can be carried out in various ways. For example, these techniques, blocks, steps, and means can be implemented in hardware, software, or a combination thereof. In the case of hardware implementations, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing units (DSPDs), programmable logic units (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions described above, and / or a combination thereof.

[0052] Furthermore, embodiments may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process terminates when its operations are complete, but may have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination corresponds to the function returning to the calling function or main function.

[0053] Furthermore, embodiments may be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented by software, firmware, middleware, scripting languages, and / or microcode, program code or code segments for performing the required tasks may be stored in a machine-readable medium such as a storage medium. Code segments or machine-executable instructions may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, scripts, classes, or any combination of instructions, data structures, and / or program statements. Code segments may be coupled to other code segments or hardware circuits by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted via any suitable means, including memory sharing, message passing, ticket passing, network transmission, etc.

[0054] In the case of firmware and / or software implementations, the method may be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described herein. Any machine-readable medium that tangibly implements instructions may be used to implement the method described herein. For example, software code may be stored in memory. Memory may be implemented within or outside the processor. As used herein, the term “memory” refers to any type of long-term memory, short-term memory, volatile memory, non-volatile memory, or other storage medium, and should not be limited to any particular type of memory, number of memories, or type of medium in which the memory is stored.

[0055] Furthermore, the term “storage medium” as disclosed herein may mean one or more memories for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage medium, optical storage medium, flash memory device, and / or other machine-readable media for storing information. The term “machine-readable medium” includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage media capable of storing or carrying (one or more) instructions and / or data.

[0056] The above description is illustrative. It is not possible to describe all possible combinations of components and methods, and those skilled in the art will recognize that many further combinations and substitutions are possible. Accordingly, this disclosure is intended to encompass all such changes, modifications, and variations that fall within the scope of this application, including the appended claims. As used herein, the terms “includes” and “contains” mean “includes but not limited to.” The term “based on” means “based at least in part on.” Also, where this disclosure or the claims enumerate “one,” “first,” “other” element, or their equivalents, it should be interpreted as including one or more such elements, and not as requiring or excluding two or more such elements.

Claims

1. It is a method, To determine the distribution of feature values ​​in a first dataset of sample sets used to train an expert system. The expert system provides a dataset of Kullback-Leibler divergence values ​​by determining the Kullback-Leibler divergence between the feature values ​​of the first dataset and the feature values ​​of the baseline dataset for each of the multiple baseline datasets received by the expert system. To determine the measure of the central trend and the measure of the statistical variance of the Kullback-Leibler divergence value dataset, The Kullback-Leibler divergence threshold, which represents the unacceptable deviation of the distribution of feature values ​​in the new dataset from the distribution of feature values ​​in the first dataset of the sample group, is determined from a desired confidence interval for the new dataset, a measure of the central tendency of the dataset of Kullback-Leibler divergence values, and a measure of the statistical variance of the dataset of Kullback-Leibler divergence values. The expert system determines the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset for the second dataset received by the expert system, and If the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset exceeds the Kullback-Leibler divergence threshold, it is determined that the second dataset represents an unacceptable deviation of the sample group from the distribution of the feature values ​​of the first dataset. A method for providing this.

2. The method according to claim 1, further comprising retraining the expert system with a third dataset of the sample group in response to the determination that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

3. The method according to claim 1, further comprising taking the expert system offline in response to determining that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset, so as not to receive any further sample groups.

4. The method according to claim 1, further comprising retraining the expert system with the first dataset of the sample group in response to the determination that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

5. The method according to claim 1, wherein the measure of the central tendency of the Kullback-Leibler divergence value dataset is the mean of the Kullback-Leibler divergence value dataset, and the measure of the statistical variance of the Kullback-Leibler divergence value dataset is the standard deviation of the Kullback-Leibler divergence value dataset.

6. The method according to claim 1, wherein determining the Kullback-Leibler divergence threshold from the desired confidence interval for the new dataset, the measure of the central trend of the dataset of Kullback-Leibler divergence values, and the measure of the statistical variance of the dataset of Kullback-Leibler divergence values ​​is further comprising determining the Kullback-Leibler divergence threshold from the desired confidence interval for the new dataset, the measure of the central trend of the dataset of Kullback-Leibler divergence values, the measure of the statistical variance of the dataset of Kullback-Leibler divergence values, and Chebyshev's inequality.

7. The method according to claim 1, wherein the expert system is a document classification system, and each sample in the first dataset of the sample group represents a document.

8. It is a system, Processor and A non-temporary computer-readable medium for storing computer-readable instructions executable by the aforementioned processor, Expert systems and Risk management components and A non-temporary computer-readable medium storing the computer-readable instructions for providing, The risk management component is equipped with, A training set characterization component that determines the distribution of feature values ​​in a first dataset of sample sets used to train the expert system, A baseline characterization component that, for each of the multiple baseline datasets received by the expert system, determines the Kullback-Leibler divergence between the feature values ​​of the first dataset and the feature values ​​of the baseline dataset, thereby providing a dataset of Kullback-Leibler divergence values, determines a measure of central tendency and a measure of statistical variance in the dataset of Kullback-Leibler divergence values, and determines a Kullback-Leibler divergence threshold representing the unacceptable deviation of the distribution of feature values ​​in the new dataset from the distribution of feature values ​​in the first dataset of the sample group, based on a desired confidence interval for the new dataset, the measure of central tendency of the dataset of Kullback-Leibler divergence values, and the measure of statistical variance of the dataset of Kullback-Leibler divergence values. A monitoring system that, with respect to a second dataset received by the expert system, determines the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset, and determines that if the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset exceeds the Kullback-Leibler divergence threshold, the second dataset represents an unacceptable deviation of the sample group from the distribution of the feature values ​​of the first dataset, A system that includes this.

9. The system according to claim 8, further comprising a feature extractor that includes a document classification system and generates a first dataset, a second dataset, and a plurality of baseline datasets of the sample group from a corresponding set of documents.

10. The system according to claim 9, wherein the feature extractor generates a first dataset of the sample group, a second dataset, and a plurality of baseline datasets from a corresponding set of documents using a bag-of-words method.

11. The system according to claim 8, further comprising a retraining component for retraining the expert system in response to the monitoring system determining that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

12. The system according to claim 11, wherein the retraining component retrains the expert system using a third dataset of the sample group in response to the determination that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

13. The system according to claim 11, wherein the retraining component restores at least a portion of the expert system from a backup in response to the determination that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

14. The system according to claim 11, wherein the retraining component retrains the expert system using the first dataset of the sample group in response to the determination that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

15. The system according to claim 8, wherein the measure of the central tendency of the Kullback-Leibler divergence value dataset is the mean of the Kullback-Leibler divergence value dataset, and the measure of the statistical variance of the Kullback-Leibler divergence value dataset is the standard variance of the Kullback-Leibler divergence value dataset.

16. The system according to claim 15, wherein determining the Kullback-Leibler divergence threshold from the desired confidence interval for the new dataset, the measure of the central trend of the dataset of Kullback-Leibler divergence values, and the measure of the statistical variance of the dataset of Kullback-Leibler divergence values ​​comprises determining the Kullback-Leibler divergence threshold from the desired confidence interval for the new dataset, the measure of the central trend of the dataset of Kullback-Leibler divergence values, the measure of the statistical variance of the dataset of Kullback-Leibler divergence values, and Chebyshev's inequality.

17. A method for risk management in a document classification system, Determining the distribution of feature values ​​in a first dataset of a sample group used to train an expert system, wherein each sample in the first dataset of the sample group represents a document. The expert system provides a dataset of Kullback-Leibler divergence values ​​by determining the Kullback-Leibler divergence between the feature values ​​of the first dataset and the feature values ​​of the baseline dataset for each of the multiple baseline datasets received by the expert system. To determine the mean and standard deviation of the Kullback-Leibler divergence dataset, The Kullback-Leibler divergence threshold, which represents the unacceptable deviation of the distribution of feature values ​​in the new dataset from the distribution of feature values ​​in the first dataset of the sample group, is determined from a desired confidence interval for the new dataset, the mean and standard deviation of the dataset of the Kullback-Leibler divergence values, and Chebyshev's inequality. The expert system determines the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset for the second dataset received by the expert system, and If the Kullback-Leibler divergence value between the feature values ​​of the second dataset and the feature values ​​of the first dataset exceeds the Kullback-Leibler divergence threshold, it is determined that the second dataset represents an unacceptable deviation of the sample group from the distribution of the feature values ​​of the first dataset. A method for providing this.

18. The method according to claim 17, further comprising retraining the expert system with a third dataset of the sample group in response to the determination that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

19. The method according to claim 17, further comprising restoring at least a portion of the expert system from a backup in response to the determination that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

20. The method according to claim 17, further comprising retraining the expert system with the first dataset of the sample group in response to the determination that the second dataset represents an unacceptable deviation of the sample group from the distribution of feature values ​​of the first dataset.

Images