Method, computer system, and storage medium for generating performance predictions

Abstract

A technique for generating a performance prediction with uncertainty intervals for a machine learning model includes obtaining a first model configured to perform a task and a production dataset. At least one metric predicting performance of the first model in performing the task on the production dataset is generated using a second model. The second model is a meta-model associated with the first model. At least one value predicting uncertainty of the at least one metric predicting performance of the first model in performing the task on the production dataset is generated using a third model. The third model is a meta-meta-model associated with the second model. An indication of the at least one metric predicting performance of the first model and the at least one value predicting uncertainty of the at least one metric is provided.

Description

Background Technology

[0001] This invention generally relates to machine learning, and more specifically, to techniques for generating performance predictions for machine learning (ML) models with intervals of uncertainty.

[0002] Recently, ML models have been increasingly used for a wide range of tasks, including (to name just a few) image recognition, speech processing, language translation, and object classification. Typically, ML models for these tasks have become increasingly complex and expensive in terms of the computational resources and time required to train and maintain them. Furthermore, due to the dissimilar objectives of each task, the models themselves can differ significantly from one another, meaning that a model trained for one task or domain often cannot be used in other domains, even if these models are closely related. Because of this possibility, attempts have been made to predict the performance of a model on a dataset for a given task. However, performance predictions can suffer from multiple sources of uncertainty, affecting the accuracy of the predictions. Summary of the Invention

[0003] One embodiment presented herein includes a computer-implemented method for generating performance predictions for a machine learning (ML) model. This computer-implemented method generally includes obtaining a first model configured to perform a task and a production dataset comprising unlabeled data. The computer-implemented method further includes generating at least one metric using a second model to predict the performance of the first model when performing the task on the production dataset. The second model is a meta-model associated with the first model. The computer-implemented method further includes generating at least one value of the uncertainty predicting the at least one metric using one or more third models, the at least one metric predicting the performance of the first model when performing the task on the production dataset. Each of the one or more third models is a meta-meta-model associated with the second model. The computer-implemented method further includes providing an indication of the at least one metric predicting the performance of the first model and the at least one value of the uncertainty predicting the at least one metric.

[0004] Other embodiments include, but are not limited to, computer program products including storage media having computer-readable program code that enables a processing unit to implement one or more aspects of the disclosed methods, and systems having a processor, memory, and an application configured to implement one or more of the disclosed methods. Attached Figure Description

[0005] Figure 1 This is a block diagram illustrating a networked system for predicting the performance of a model at a task and the uncertainty of that model's performance, according to one embodiment.

[0006] Figure 2A and 2B A stacked meta-modeling workflow for generating performance predictions of a model and uncertainty intervals of the model's performance predictions, according to one embodiment, is shown.

[0007] Figure 3 The illustration shows an example process for generating one or more datasets for training a meta-meta model, according to one embodiment.

[0008] Figure 4 An example of generating features for training a meta-meta model is shown according to one embodiment.

[0009] Figure 5 This is a flowchart of a method for predicting the performance of a model at a task and the uncertainty of the model's performance at that task, according to one embodiment.

[0010] Figure 6 This is a flowchart of a method for training a stacked meta-model workflow and a meta-meta-model, according to one embodiment, to generate performance predictions of the model and uncertainty intervals of the model's performance predictions.

[0011] Figure 7 This is a flowchart of a method for training a meta-meta model according to one embodiment.

[0012] Figure 8 An exemplary simulation of performance prediction and uncertainty intervals according to one embodiment is shown. Detailed Implementation

[0013] ML tools are increasingly used for a wide range of tasks, including but not limited to regression and classification, optimization, and prediction. However, in some cases, the performance of a given model on a task can change over time. For example, a model's ability to make accurate predictions may deteriorate over time, depending on how different the underlying data is from the model's training data. Currently, many conventional techniques can be used to predict the performance of a model on a given task. For example, one conventional technique involves detecting the amount of drift in one or more features of the underlying data. Another traditional technique involves tracking the model's accuracy by measuring the proportion of correct predictions (e.g., the number of times any class is correctly predicted, the area under the receiver operating characteristic (ROC) curve, etc.) over time. However, these conventional techniques can only be used for performance prediction and do not capture the model's uncertainty.

[0014] In addition, there are many conventional techniques for predicting uncertainty in isolation (e.g., performance prediction tasks without considering underlying layers). Examples of such techniques include, but are not limited to, confidence intervals and probability (p) values. Furthermore, in Bayesian modeling, there are different types of model uncertainty, including, for example, random uncertainty and cognitive uncertainty. Random uncertainty captures the inherent uncertainty in the data, examples of which can include noise, various data gaps, confusion, etc. Cognitive uncertainty captures the uncertainty caused by the model (e.g., model architecture, model parameters, model assumptions, parameter estimation, undertraining, etc.).

[0015] One problem with the methods described above is that they typically assume the existence of choices in the architecture of the underlying model that enable predictions of model performance and / or model uncertainty. In cases where such choices are absent, methods such as ensemble techniques can be used to capture uncertainty; however, these methods assume the underlying model is white-box (e.g., a model whose internal architecture and parameters are visible or known). However, in many cases, these assumptions are unacceptable when using customer-provided models. Therefore, it may be desirable to provide a technique for generating both performance predictions and uncertainty predictions for the model.

[0016] Embodiments of this disclosure provide techniques for predicting the performance of an ML model on an underlying task, along with an uncertainty interval for the predicted performance of the ML model. More specifically, given a trained ML model and a set of (unlabeled) production data (which may or may not be significantly different from the model's training data), embodiments can predict the model's performance on that set of production data (e.g., accuracy or other performance or quality-related metrics) and an uncertainty interval (e.g., bands or error bars around the prediction) to account for the uncertainty associated with the model at a particular test instance or a batch of test instances.

[0017] In one embodiment described in more detail below, a stacked meta-modeling approach is employed to generate performance predictions and uncertainty predictions. For example, the embodiment generates a two-level meta-model: (1) a first-level meta-model that predicts the performance of the base model, and (2) a second-level meta-meta-model that predicts the uncertainty of the first-level meta-model (e.g., its prediction of the performance of the base model). Using the stacked meta-modeling approach enables the embodiment to work with models of any architecture (e.g., the stacked meta-modeling approach is agnostic to the model architecture and does not depend on access to the model's internal parameters) and captures multiple types of uncertainty (e.g., random uncertainty, cognitive uncertainty, etc.).

[0018] As used herein, "metamodel" generally refers to a model of models (lower-level models), while "meta-metamodel" generally refers to a model of metamodels. For example, a metamodel captures patterns observed on a lower-level model that interacts with data. Note that in various embodiments of this disclosure, visual tasks (e.g., processing images for action recognition, object detection, face recognition, digit or character classification, etc.) are used as examples to explain the functionality of ML models. However, embodiments of this disclosure can be readily applied to any number of domains using any input (e.g., visual, audio, text, etc.). Furthermore, as used herein, "performance" of a model can refer to one or more accuracy-related metrics of the model on unlabeled data. Similarly, "performance prediction" can refer to a model-based method that predicts one or more performance or quality-related metrics of the underlying underlying model on unlabeled data.

[0019] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or technical improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0020] The following reference is made to embodiments presented in this disclosure. However, the scope of this disclosure is not limited to the specifically described embodiments. Rather, any combination of the following features and elements is contemplated for implementing and practicing the intended embodiments, regardless of whether different embodiments are involved. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or prior art, whether a given embodiment achieves a particular advantage does not limit the scope of this disclosure. Therefore, the following aspects, features, embodiments, and advantages are merely illustrative and should not be considered elements or limitations of the appended claims unless expressly stated in the claims. Similarly, references to “the invention” should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered elements or limitations of the appended claims unless expressly stated in the claims.

[0021] Figure 1 This is a block diagram illustrating a networked system 100 for performance prediction and uncertainty prediction of a generative model according to one embodiment. System 100 includes a computing system 102. The computing system 102 may also be connected to other computers via a network 140. Network 140 may include one or more networks of various types, including local area networks (LANs), general wide area networks (WANs), telecommunications or cellular networks, and / or public networks (e.g., the Internet).

[0022] Computing system 102 typically includes one or more processors 104 connected via bus 150 to memory 106, storage device 108, network interface 110, input device 152, and output device 154. Computing system 102 is typically under the control of an operating system (not shown). Examples of operating systems include UNIX, various versions of Microsoft Windows, and various distributions of Linux. (Unix is ​​a registered trademark of The Open Group in the U.S. and other countries. Microsoft and Windows are trademarks of Microsoft Corporation in the U.S., other countries, or both. Linux is a registered trademark of Linus Torvalds in the U.S., other countries, or both.) More generally, any operating system supporting the functionality disclosed herein may be used. The included processor 104 represents a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc.

[0023] Memory 106 may include various computer-readable media selected for performance or other capabilities: volatile and / or non-volatile media, removable and / or non-removable media, etc. Memory 106 may include cache, random access memory (RAM), etc. Storage device 108 may be a disk drive or a flash memory device. Although shown as a single unit, storage device 108 may be a combination of fixed and / or removable storage devices, such as fixed disk drives, solid-state drives, removable memory cards, optical storage devices, network-attached storage devices (NAS), or storage area networks (SANs). Network interface 110 may be any type of network communication device that allows computing system 102 to communicate with other computing systems or devices via network 140.

[0024] Input device 152 provides input to computing system 102. For example, a keyboard and / or mouse may be used. Output device 154 may be any device used to provide output to a user of computing system 102. For example, output device 154 may be any conventional display screen. Although shown separately from input device 152, output device 154 and input device 152 may be combined. For example, a display screen with an integrated touchscreen may be used.

[0025] Here, storage device 108 includes production dataset 130, base model 132, test dataset 134, training dataset 136, and offline model and dataset 138. Base model 132 is a model used to solve a specific task (e.g., a client model for image classification). Generally, any suitable type of model can be used for base model 132, examples of which include, but are not limited to, artificial neural networks, decision trees, support vector machines, regression analysis, Bayesian networks, genetic algorithms, etc. Base model 132 can be a black-box model (e.g., a model where the architecture and / or parameters are unknown) or a white-box model (e.g., a model where the architecture and / or parameters are known).

[0026] A base model 132 can be trained / developed on a specific training dataset (e.g., training dataset 136), and the base model can be evaluated using a labeled test dataset (e.g., test dataset 134 including classification labels). The production dataset 130 is an unlabeled dataset for which no ground truth (e.g., classification labels) is available. The production dataset 130 can be collected from operational status after the base model 132 has been deployed. Offline models and / or datasets 138 include various models and datasets, some of which may be related to the original task of the base model 132, and some of which may be unrelated to the original task of the base model 132. Offline models and / or datasets 138 will be referenced below. Figure 2B and Figure 3 Let's discuss this in more detail.

[0027] Memory 106 includes a prediction engine 120 configured to implement one or more of the techniques described herein for generating performance predictions and uncertainty intervals for models. Prediction engine 120 includes a performance component 122 and an uncertainty component 124, each of which may include software, hardware, or a combination thereof. Prediction engine 120 is configured to generate performance predictions (or predicted performance) for a base ML model 132 (via performance component 122) and uncertainty intervals for the performance predictions (via uncertainty component 124), for example, using a stacked meta-modeling approach. For example, performance component 122 may use a meta-model (of base model 132) to predict the performance of base model 132 on an unlabeled production dataset 130 (e.g., one or more accuracy-related metrics). The meta-model may be trained using one or more of base model 132, test dataset 134, and training dataset 136.

[0028] Similarly, uncertainty component 124 can use a meta-meta-model (of the meta-model of base model 132) to predict the uncertainty of the meta-model of performance component 122. The meta-meta-model can be trained using one or more of a set of offline models / datasets 138, the meta-model of performance component 122, base model 132, production dataset 130, test dataset 134, and training dataset 136. In one embodiment, uncertainty component 124 can generate an uncertainty interval (also called an error bar) representing the uncertainty of the meta-model. The uncertainty interval depicts a band (or tolerance) around the output of performance component 122 (e.g., the predictive performance of base model 132). This band (or tolerance) around the prediction can account for the uncertainty associated with base model 132 at a particular test instance (e.g., on production dataset 130). Performance component 122 and uncertainty component 124 are referenced below. Figure 2A and 2B Let's discuss this in more detail.

[0029] Notice, Figure 1 Only one reference example of system 100 is shown, which can generate performance predictions and uncertainty intervals for a model, and other configurations of system 100 may be adapted to generate performance predictions and uncertainty intervals for a model. For example, in some embodiments, one or more contents of storage device 108 (e.g., production dataset 130, base model 132, test dataset 134, training dataset 136, and offline model / dataset 138) and / or one or more contents (e.g., performance component 122 and uncertainty component 124) may be distributed across one or more computing systems 102 in a network (e.g., a cloud computing environment). In such an embodiment, a first computing system 102 may retrieve one or more contents from one or more second computing systems 102 for generating performance predictions for base model 132. Similarly, a first computing system 102 may retrieve one or more contents from one or more second computing systems 102 for generating uncertainty intervals for the predicted performance of base model 132. In other embodiments, performance predictions and / or uncertainty intervals may be generated by a single computing system 102 or multiple computing systems 102.

[0030] Figure 2A-2B A stacked meta-modeling workflow 200 is illustrated according to one embodiment for generating performance predictions of a base model (e.g., base model 132) and uncertainty intervals of the performance predictions of the base model. Here, the stacked meta-modeling workflow 200 includes a training phase 210 and a usage (or deployment) phase 212. Figure 2A (as shown) and training phase 214 and usage phase 216 (as shown) Figure 2B (as shown in the image).

[0031] like Figure 2A As shown, performance component 122 includes meta-model 202, which typically learns to predict the probabilities of success and failure of base model 132 on an instance-by-instance basis. Workflow 200 illustrates several different pieces of information that can be considered during training phase 210. In the described embodiment, base model 132 and (labeled) test dataset 134 are processed and used as input to train meta-model 202. Also as shown, in some embodiments, training dataset 136 may also be used as additional input to train meta-model 202.

[0032] Training phase 210 describes a supervised learning method for training meta-model 202. Training phase 210 can be performed offline (e.g., when meta-model 202 is not in use or deployed) or online (e.g., when meta-model 202 is in use or deployed). However, more generally, any suitable machine learning training mechanism consistent with the functionality described herein can be used. Further note that although the base model 132, test dataset 134, and / or training dataset 136 are shown to be fed directly into meta-model 202 during training phase 210, those skilled in the art will understand that this information can be processed in various ways before any input data is fed into meta-model 202.

[0033] Once the meta-model 202 has been trained, the (unlabeled) production dataset 130 (e.g., without classification labels) can be fed into the meta-model 202 as input, and the meta-model 202 can output a performance prediction 204 (e.g., during the use (deployment) phase 212). In some cases, the output prediction scores (e.g., softmax, logits) of the base model 132 and / or the original input features of the base model 132 can also be fed into the meta-model 202 as input. Note that although the production dataset 130 is shown as being fed directly into the meta-model 202, those skilled in the art will understand that this information can be processed in various ways before any input data is fed into the meta-model 202.

[0034] Performance prediction 204 is a predictive metric of how well (or accurately) the base model 132 performs on the (unlabeled) production dataset 130 in its task (e.g., classifying images). For example, assuming the base model 132 is a model for object classification, performance prediction 204 could include a percentage value (e.g., 74%) indicating the probability of success / failure of the base model 132 in classifying objects from the production dataset 130. More generally, performance prediction 204 could include one or more accuracy-related metrics for the base model 132 on the production dataset 130. Examples of such accuracy-related metrics could include, but are not limited to, area under ROC score, true positive rate (TPR), F1 metric, false positive rate (FPR), R-squared score, accuracy score (e.g., the percentage of correct predictions made by the model), etc.

[0035] Performance prediction 204 can be a batch accuracy prediction or a point-by-point accuracy prediction. In some cases, performance prediction 204 can be used for pre-deployment quality checks of the base model 132 (e.g., determining whether the base model 132 should be deployed). In other cases, performance prediction 204 can be used for quality control / checks during the runtime or deployment of the base model 132 (e.g., filtering incorrect predictions, identifying degradations in production, etc.). However, as mentioned above, performance prediction 204 may suffer from multiple sources of uncertainty not captured in accuracy-related metrics (e.g., random uncertainty, cognitive uncertainty, etc.). To address this issue, embodiments use uncertainty component 124 to predict the uncertainty interval regarding performance prediction 204.

[0036] like Figure 2B As shown, the uncertainty component 124 includes a meta-model 206, which learns to be a function of the error in predicting the meta-model 202 and the actual observed performance (e.g., the absolute value of the predicted accuracy minus the actual accuracy). Workflow 200 illustrates several different pieces of information that can be considered during the training phase 214. In the described embodiment, the offline model / dataset 138 and the meta-model 202 are processed and used as inputs to train the meta-model 206. Also as shown, in some embodiments, the production dataset 130 may also be used as an additional or separate input to train the meta-model 206.

[0037] Training phase 214 describes the supervised learning method used to train the meta-meta-model 206. Training phase 214 can be performed offline once or multiple times. However, more generally, any suitable machine learning training mechanism consistent with the functionality described herein can be used. Further note that although the offline model / dataset 138, meta-model 202, and / or production dataset 130 are shown as directly fed into the meta-meta-model 206, those skilled in the art will understand that this information can be processed in various ways before any input data is fed into the meta-meta-model 206.

[0038] Meta-meta-model 206 is configured to learn from one or more offline runs of training phase 214 when meta-model 202 may err and to what extent. In one embodiment, meta-meta-model 206 may be trained on multiple background offline models / datasets 138 (excluding production dataset 130) to learn general features indicative of the expected uncertainty of meta-model 202. In another embodiment, meta-meta-model 206 may be trained on production dataset 130 (e.g., without offline model / dataset 138). In yet another embodiment, meta-meta-model 206 may be trained on both offline / model / dataset 138 and production dataset 130.

[0039] The offline model / dataset 138 may include a labeled dataset related to the task of the base model 132 (e.g., object classification) and / or a labeled dataset unrelated to the task of the base model 132 (e.g., audio-to-text conversion). In some embodiments, at least one (first) dataset of the offline model / dataset 138 used to train the meta-model 206 may be dynamically generated from one or more (second) datasets in the offline model / dataset 138. For example, one or more first datasets may be generated based on resampling one or more features from the second dataset.

[0040] Figure 3 An example of generating one or more datasets for training a meta-meta-model 206 is illustrated according to one embodiment. As shown, given a (base) labeled dataset 302 (which may be one of the offline models / datasets 138), prediction engine 120 can generate one or more training / test datasets 304-1 to 304-6 and one or more production datasets 306-1 to 306-6 for training the meta-meta-model 206. In one embodiment, prediction engine 120 can generate training / test datasets 304 and production datasets 306 by resampling the (base) labeled dataset 302 based on one or more features of the labeled dataset 302.

[0041] In one example, training / test dataset 304 and production dataset 306 can be generated by resampling the (base) labeled dataset 302 based on various features to introduce covariate shifts. In this example, the specific features of the (base) labeled dataset 302 used to resample the distributions of training / test dataset 304 and production dataset 306 can be dynamically selected by prediction engine 120 (e.g., based on feature importance). In another example, training / test dataset 304 and production dataset 306 can be generated by resampling the (base) labeled dataset 302 based on actual class labels to introduce prior probability shifts.

[0042] like Figure 3 As shown, assuming the feature is “feature A” (e.g., “dog”) (or “feature B”, such as “cat”), the ratio of “feature A” (or “feature B”) can be varied in each of the training / test datasets 304-1 to 304-6 and the production datasets 306-1 to 306-6 (e.g., by resampling the features in the labeled dataset 302). Here, for example, the training / test dataset 304-1 includes a statistical distribution of 100% (or 0% of “feature B”), and the corresponding production dataset 306-1 includes a statistical distribution of 0% (or 100% of “feature B”). Similarly, on the opposite end, the training / test dataset 304-6 includes a statistical distribution of 0% (or 100% of “feature B”), and the corresponding production dataset 306-6 includes a statistical distribution of 100% (or 0% of “feature B”).

[0043] Return to reference Figure 2B In some embodiments, training the meta-model 206 (e.g., during training phase 214) may involve generating one or more features based on metadata about the input dataset (e.g., offline model / dataset 138, which may include training / test dataset 304, production dataset 306, etc.) and / or other distributional feature spatial properties of the input dataset. In one embodiment, one or more features may be based on the base model 132 and the offline model / dataset 138 (including training and production datasets). For example, the prediction engine 120 may compute various distributions in the test and production datasets and compare the distances between the various distributions to create one or more features for training the meta-model 206.

[0044] like Figure 4As shown, for example, a first distribution (histogram 402) can be computed based on a test dataset (e.g., test dataset 304), and a second distribution (histogram 404) can be computed based on a production dataset (e.g., production dataset 306). The distance 406 (or more generally, divergence) between the distributions can be computed and used as a feature (also called an eigenvalue) 408. Generating features 408 in this way to train the meta-model 206 can take into account scenarios with significant noise (e.g., above a threshold) in performance predictions, scenarios where the average output from the meta-model 202 is stable but has high variance (e.g., above a threshold), scenarios where the per-sample uncertainty is too large for the batch error bars, etc.

[0045] In one embodiment, an example feature 408 may be based on one or more highest confidence scores from a base model 132. For example, the base model 132 may be used to score samples in a test dataset and a production dataset. Assuming the base model 132 is used for object classification (e.g., between class A and class B), a predicted score using the output of the base model 132 can be obtained for each sample (or data point) in the test dataset and each sample (or data point) in the production dataset. As an example, for the first sample in the test dataset, the base model 132 may output 95% for class A, and for the first sample in the production dataset, the base model 132 may output 90% for class A.

[0046] After all samples in each of the test and production datasets have been scored, histograms can be constructed using subsets of the scored samples from both datasets. For example, a (first) histogram can be generated from the highest confidence scores of the scored samples in the test dataset (e.g., within certain threshold ranges, such as 90-95%, 95-98%, 98-100%, etc.). Similarly, a (second) histogram can be generated from the highest confidence scores of the scored samples in the production dataset (e.g., within certain threshold ranges). The two histograms can then be compared with a divergence function (e.g., a similarity metric, a difference metric). In one embodiment, a distance 406 (e.g., Hellinger distance) between the two histograms can be calculated, and the value of distance 406 (e.g., in [0, 1]) can be used as a feature 408 of the meta-meta-model 206.

[0047] In one embodiment, an example feature 408 may be based on the highest confidence score of the shadow model. In this embodiment, another (surrogate) model is trained on the same training dataset (e.g., training dataset 136) used to train the base model 132, compared to the highest confidence score of the base model. The highest confidence score from both the test and production datasets is then computed using this surrogate model (e.g., relative to the base model 132). Histograms may be generated based on the highest confidence scores, and the distance between the histograms may be used as feature 408 of the meta-model 206.

[0048] In one embodiment, an example feature 408 may be based on class frequency distance. In this embodiment, for example, histograms of the percentage of samples in a test dataset and a production dataset, the production dataset being predicted by the base model 132 for each class, may be generated. The distance 406 between the histograms (e.g., Hellinger distance) may be calculated, and this distance value may be used as feature 408 of the meta-model 206.

[0049] In one embodiment, an example feature 408 can be based on the highest feature distance. In this embodiment, for example, a shaded random forest model can be trained and used to identify data features with the highest feature importance (e.g., satisfying predefined conditions). Once identified, histograms of the test and production datasets can be projected onto this dimension (e.g., a compressed one-dimensional feature space). The distance 406 between the histograms (e.g., Hellinger distance) can be calculated, and this distance value can be used as feature 408 of the meta-model 206.

[0050] In one embodiment, an example feature 408 may be based on a meta-model prediction (e.g., performance prediction 204). In this embodiment, the change in the accuracy of the underlying model between the test dataset and the production dataset predicted by the meta-model 202 may be used as feature 408 of the meta-meta-model 206. In one embodiment, an example feature 408 may be based on a statistical hypothesis test between a first statistical distribution (of the first dataset) and a second statistical distribution (of the second dataset).

[0051] Note that the features described above are provided only as reference examples of features that can be used to train meta-model 206, and other features or any combination of features based on metadata of the input dataset can be used, consistent with the functionality described herein. In some embodiments, for example, multiple meta-models can be generated / trained based on different features. In these embodiments, the specific meta-model used during runtime phase 216 can be dynamically selected at runtime based on the characteristics of the data (e.g., metadata about the dataset, the number of features, distribution feature space properties, etc.).

[0052] Once the meta-meta-model 206 has been trained, one or more of the following can be fed as input: base model 132, test dataset 134, training dataset 136, meta-model 202, production dataset 130, and performance prediction 204. The meta-meta-model 206 can then output an uncertainty prediction 208 (e.g., during use (deployment) phase 216). The uncertainty prediction 208 is a predictive measure of the uncertainty of the performance prediction 204 output from the meta-model 202. For example, assuming the base model 132 is a model for object classification, the uncertainty prediction 208 can indicate an interval (or tolerance) representing the amount of uncertainty in the performance prediction 204 (e.g., ±4, ±7, +2). In some embodiments, the uncertainty prediction 208 can be used for additional pre-deployment quality checks of the base model 132 and / or for quality control / checks during runtime or deployment of the base model 132.

[0053] Figure 5 This is a flowchart of a method 500 for predicting the performance of a model at a task and the uncertainty of the model's performance at the task, according to one embodiment. Method 500 may be performed by a prediction engine of a computing system (e.g., prediction engine 120 of computing system 102).

[0054] Method 500 can be entered at block 502, where the prediction engine obtains a base model (e.g., base model 132) and a set of production data (e.g., production dataset 130). At block 504, the prediction engine uses the meta-model (e.g., meta-model 202) to predict the performance of the base model on the production dataset (e.g., performance prediction 204). For example, the prediction engine may generate one or more accuracy-related metrics for the base model based on this set of production data. In a particular example, the accuracy-related metric may indicate the probability of success / failure of the base model at its task using this set of production data (e.g., X% success).

[0055] In block 506, the prediction engine uses a meta-meta-model (e.g., meta-meta-model 206) to predict one or more intervals of uncertainty (e.g., uncertainty prediction 208) for the performance of the meta-model against the set of production data. For example, the prediction engine can use the set of production data to generate a tolerance (or error band) (e.g., ±Y) that indicates the amount of uncertainty in the predictive performance of the underlying model at its task.

[0056] In one embodiment, the uncertainty interval (predicted in block 506) can be asymmetric. For example, the prediction engine can predict a signed value (e.g., + or -) of the error representing the uncertainty of the predicted performance. In another example, a first uncertainty interval can be generated for the upper band / range around the predicted performance, and a second uncertainty interval can be generated for the lower band / range around the predicted performance.

[0057] In some embodiments, multiple meta-meta-models can be used to generate multiple uncertainty intervals (e.g., a first meta-meta-model for a first (upper) uncertainty interval and a second meta-meta-model for a second (lower) uncertainty interval). In some embodiments, a single meta-meta-model can be used to generate multiple uncertainty intervals. For example, a single meta-meta-model may include two sub-modules (or components) for predicting the upper and lower uncertainty intervals.

[0058] In block 508, the prediction engine provides indications of the performance of the base model and one or more uncertainty intervals. In one embodiment, the prediction engine may provide these indications on a user interface of a display on a computing device (e.g., computing system 102). In one embodiment, the prediction engine may provide these indications in response to a request for performance and uncertainty predictions of the base model against a set of production data. For example, a user may use the prediction engine to determine which base model (among multiple base models) to use on that set of production data. In another example, the prediction engine may monitor the deployment / running of the base model and continuously provide indications to the computing device (e.g., at one or more predefined time intervals). These indications can then be used to determine when the base model should be refined and / or replaced. In some cases, the prediction engine may run a meta-meta-model at the same frequency as the meta-model.

[0059] Figure 6 This is a flowchart of a method 600 for training a meta-model and a meta-meta-model in a stacked meta-model workflow for generating performance predictions and uncertainty intervals of model performance predictions, according to one embodiment. Method 600 may be executed by a prediction engine of a computing system (e.g., prediction engine 120 of computing system 102).

[0060] Method 600 can be entered at block 602, where the prediction engine retrieves a base model (e.g., base model 132), a test dataset (e.g., test dataset 134), and one or more additional datasets (e.g., offline model / dataset 138). At block 604, the prediction engine trains a meta-model (e.g., meta-model 202) based on the base model and the test dataset to predict the performance of the base model on the task. At block 606, the prediction engine trains a meta-meta-model based on the meta-model and one or more additional datasets to predict the uncertainty of the base model's performance on its task.

[0061] Figure 7 This is a flowchart of a method 700 for training a meta-meta model according to one embodiment. Method 700 may be performed by a prediction engine of a computing system (e.g., prediction engine 120 of computing system 102).

[0062] Method 700 may enter at block 702, where the prediction engine obtains a labeled dataset (e.g., labeled dataset 302). At block 704, the prediction engine generates one or more supplementary datasets based on the labeled dataset (e.g., training / test dataset 304, production dataset 306). At block 706, the prediction engine determines (or computes) one or more features (e.g., feature 408) based on the evaluation of the one or more supplementary datasets. At block 708, the prediction engine trains one or more meta-meta-models (e.g., meta-meta-model 206), each meta-meta-model being configured to predict the uncertainty interval of the meta-model (e.g., meta-model 202) at least partially based on the one or more features.

[0063] Figure 8 Example simulations 802 and 804 are shown, illustrating the performance prediction and the uncertainty interval of the performance prediction based on a base model according to one embodiment.

[0064] More specifically, simulation 802 describes the performance prediction of the base model across different test / production datasets 1-K. As shown, the accuracy of the base model (e.g., the actual accuracy of the base model) is represented by line 808, the prediction accuracy of the base model is represented by line 808, and the error bars for the prediction accuracy are represented by 806. Each test / production dataset 1-K can have different statistical distributions for the features / labels. For example, test / production dataset 1 can include the statistical distributions of 100% "Feature A" in the test dataset and 0% "Feature A" in the production dataset. Similarly, on the opposite end, test / production dataset K can include the statistical distributions of 0% "Feature A" in the test dataset and 100% "Feature A" in the production dataset.

[0065] Simulation 804, corresponding to simulation 802, shows the meta-model error (represented by 840), the fixed error bar (or uncertainty value) 820, and the increment 830 between the meta-model error and the actual base model output. As shown, by using a stacked meta-model workflow to generate the uncertainty interval for the model's performance predictions, the embodiment can generate a more accurate uncertainty interval compared to using the fixed uncertainty value 820.

[0066] In the foregoing, reference has been made to the embodiments presented in this disclosure. However, the scope of this disclosure is not limited to the specifically described embodiments. Rather, any combination of features and elements is contemplated for implementing and practicing the intended embodiments, regardless of whether different embodiments are involved. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or prior art, whether a given embodiment achieves a particular advantage does not limit the scope of this disclosure. Therefore, the aspects, features, embodiments, and advantages discussed herein are merely illustrative and should not be considered as elements or limitations of the appended claims unless expressly stated in the claims. Similarly, references to “the invention” should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered as elements or limitations of the appended claims unless expressly stated in the claims.

[0067] Various aspects of the present invention may take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may be collectively referred to herein as a “circuit,” a “module,” or a “system.”

[0068] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0069] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0070] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0071] Computer-readable program instructions for performing the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from the computer-readable program instructions.

[0072] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0073] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0074] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0075] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the blocks may occur in a non-linear order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0076] Embodiments of the present invention can be provided to end users through cloud computing infrastructure. Cloud computing generally refers to providing scalable computing resources as a service over a network. More formally, cloud computing can be defined as providing computing power that abstracts between computing resources and their underlying technical architecture (e.g., servers, storage devices, networks), thereby enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Therefore, cloud computing allows users to access virtual computing resources (e.g., storage, data, applications, or even complete virtualized computing systems) in the “cloud,” regardless of the underlying physical systems (or the location of those systems) used to provide the computing resources.

[0077] Typically, cloud computing resources are provided to users on a pay-as-you-go basis, where users are charged only for the computing resources they actually use (e.g., the amount of storage space consumed or the number of virtualized systems instantiated by the user). Users can access any resource residing in the cloud at any time and from anywhere on the Internet. In the context of this invention, users can access applications (e.g., prediction engine 120) or related data (e.g., base model 132, test dataset 134, training dataset 36, offline model / dataset 138, production dataset 130, etc.) available in the cloud. For example, prediction engine 120 can execute on a computing system in the cloud and predict the performance of the base model and the uncertainty range of the predicted performance of the base model. In this case, the application can retrieve one or more of the input information used to generate performance predictions and / or uncertainty predictions from a storage location in the cloud and store the performance predictions and / or uncertainty predictions at the storage location in the cloud.

[0078] While the foregoing relates to embodiments of the present invention, other and further embodiments of the present invention may be designed without departing from the basic scope of the present invention, and the scope of the present invention is defined by the appended claims.

Claims

1. A computer-implemented method for generating a performance prediction of a machine learning model, comprising: obtaining a first model configured to perform a task and a production dataset comprising unlabeled data, the unlabeled data comprising visual data, audio data, or textual data, the task comprising action recognition, object detection, facial recognition, digital or character classification, image classification, image recognition, language translation, or object classification; generating, using a second model, a prediction of at least one measure of performance of the first model in performing the task on the production dataset, wherein the second model is a meta-model associated with the first model; generating, using one or more third models, at least one value predicting an uncertainty of the at least one measure of performance of the first model in performing the task on the production dataset, wherein each of the one or more third models is a meta-meta-model associated with the second model; and providing an indication of the at least one measure of performance of the first model and the at least one value predicting the uncertainty of the at least one measure.

2. The computer-implemented method of claim 1, wherein the first model is a black box model.

3. The computer-implemented method of claim 1, wherein the first model is a white box model.

4. The computer-implemented method of claim 1, wherein at least one of the one or more third models is trained on a training dataset of the first model and the second model.

5. The computer-implemented method of claim 1, wherein at least one of the one or more third models is trained on a training dataset of the first model, one or more additional datasets, and the second model.

6. The computer-implemented method of claim 1, wherein at least one of the one or more third models is trained on one or more additional datasets and the second model.

7. The computer-implemented method of claim 6, wherein the at least one of the one or more third models is trained offline on the one or more additional datasets and the second model.

8. The computer-implemented method of claim 6, wherein: the one or more additional datasets comprise at least one dataset comprising a first set of labels for classifying data in the at least one dataset; and the first set of labels is different from a second set of labels for classifying data in a dataset used to train the second model.

9. The computer-implemented method of claim 6, wherein the one or more additional datasets comprise at least one labeled dataset, the computer-implemented method further comprising: based on the labeled dataset, generating a plurality of test datasets and a plurality of production datasets; and including the plurality of test datasets and the plurality of production datasets in the one or more additional datasets. ​ 10. The computer-implemented method of claim 9, wherein generating the plurality of test data sets and the plurality of production data sets comprises resampling one or more features of the labeled data set such that each test data set and corresponding production data set comprises a different proportion of the one or more features relative to the labeled data set.

11. The computer-implemented method of claim 6, further comprising generating at least one feature value for training the one or more third models based at least in part on a first statistical distribution of a first data set of the one or more additional data sets and a second statistical distribution of a second data set of the one or more additional data sets, wherein the at least one of the one or more third models is also trained with the one or more feature values.

12. The computer-implemented method of claim 11, wherein the at least one feature value is further generated based on a divergence function between the first statistical distribution and the second statistical distribution.

13. The computer-implemented method of claim 11, wherein the at least one feature value is further generated based on a statistical hypothesis test between the first statistical distribution and the second statistical distribution.

14. The computer-implemented method of claim 6, wherein the one or more third models is a subset of a plurality of third models, the computer-implemented method further comprising selecting the one or more third models from the plurality of third models based at least in part on one or more features of the one or more additional data sets.

15. The computer-implemented method of claim 1, wherein predicting the at least one value of the uncertainty of the at least one metric comprises an interval range.

16. The computer-implemented method of claim 15, wherein the interval range is an asymmetric interval range comprising a first upper limit and a second lower limit.

17. The computer-implemented method of claim 16, wherein the first upper limit and the second lower limit are generated via one of the one or more third models.

18. The computer-implemented method of claim 16, wherein: the one or more third models comprise a plurality of third models; the first upper limit is generated via a first third model of the plurality of third models; and the second lower limit is generated via a second third model of the plurality of third models.

19. The computer-implemented method of claim 16, wherein: the first upper limit is generated via a first component of one of the one or more third models; and the second lower limit is generated via a second component of the one of the one or more third models.

20. A computer system comprising: one or more computer processors; and memory containing a program which, when executed by the one or more computer processors, performs operations for generating a performance prediction of a machine learning model, the operations comprising: ​ ​ obtaining a first model configured to perform a task and a production dataset comprising unlabeled data, the unlabeled data comprising visual data, audio data, or textual data, the task comprising action recognition, object detection, facial recognition, digit or character classification, image classification, image recognition, language translation, or object classification; generating, using a second model, at least one metric predicting a performance of the first model in performing the task on the production dataset, wherein the second model is a meta-model associated with the first model; generating, using a third model, at least one value predicting an uncertainty of the at least one metric predicting the performance of the first model in performing the task on the production dataset, wherein the third model is a meta-meta-model associated with the second model; and providing an indication of the at least one metric predicting the performance of the first model and the at least one value predicting the uncertainty of the at least one metric.

21. A computer-readable storage medium having computer-readable program code executable by one or more computer processors to perform operations for generating a performance prediction of a machine learning model, the operations comprising: obtaining a first model configured to perform a task and a production dataset comprising unlabeled data, the unlabeled data comprising visual data, audio data, or textual data, the task comprising action recognition, object detection, facial recognition, digit or character classification, image classification, image recognition, language translation, or object classification; generating, using a second model, at least one metric predicting a performance of the first model in performing the task on the production dataset, wherein the second model is a meta-model associated with the first model; generating, using a third model, at least one value predicting an uncertainty of the at least one metric predicting the performance of the first model in performing the task on the production dataset, wherein the third model is a meta-meta-model associated with the second model; and providing an indication of the at least one metric predicting the performance of the first model and the at least one value predicting the uncertainty of the at least one metric.

Images