Generating performance predictions with uncertainty intervals

The stacked meta-modeling approach addresses the challenge of uncertainty in ML model performance predictions by using a meta-model to predict performance and a meta-meta model to estimate uncertainty intervals, ensuring accurate and reliable assessments across diverse domains.

JP7588653B2Active Publication Date: 2024-11-22INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2022555680
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-04-07
Filing Date
2021-02-16
Publication Date
2024-11-22
Estimated Expiration
2041-02-16

AI Technical Summary

Technical Problem

Existing machine learning (ML) model performance prediction techniques fail to accurately account for uncertainty, particularly when models are complex and domain-specific, leading to inaccuracies in performance assessments.

Method used

A stacked meta-modeling approach involving a first-level meta-model to predict performance and a second-level meta-meta model to predict uncertainty, allowing for the generation of performance predictions with uncertainty intervals, applicable to ML models of arbitrary architectures.

Benefits of technology

This method effectively captures multiple types of uncertainty, providing accurate performance predictions with associated uncertainty intervals, enhancing the reliability of ML model assessments across different domains.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007588653000001
    Figure 0007588653000001
  • Figure 0007588653000002
    Figure 0007588653000002
  • Figure 0007588653000003
    Figure 0007588653000003
Patent Text Reader

Abstract

Generating a performance prediction with an uncertainty interval. A technique for generating a performance prediction of a machine learning model with an uncertainty interval includes obtaining a first model configured to perform a task and a manufacturing dataset. At least one metric predicting the performance of the first model when performing the task on the manufacturing dataset is generated using a second model. The second model is a meta-model associated with the first model. At least one value predicting the uncertainty of the at least one metric predicting the performance of the first model when performing the task on the manufacturing dataset is generated using a third model. The third model is a meta-meta-model associated with the second model. An indication of the at least one metric predicting the performance of the first model and the at least one value predicting the uncertainty of the at least one metric is provided.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to machine learning, and more particularly to techniques for generating performance predictions for machine learning (ML) models with uncertainty intervals. [Background technology]

[0002] In recent years, ML models have been increasingly used for a variety of tasks, including image recognition, speech processing, language translation, and object classification, to name a few. Generally, ML models used for these tasks have become increasingly complex and expensive in terms of the computational resources and time required to train and maintain the models. Furthermore, due to the different goals of each task, the models themselves may differ dramatically from each other, such that models trained for one task or domain are typically unusable for other domains, even when the models are closely related. Due to this possibility, attempts have been made to predict the performance of models on a set of data for a given task. However, performance predictions suffer from multiple sources of uncertainty, which may affect the accuracy of the performance predictions. Summary of the Invention

[0003] An embodiment presented herein includes a computer-implemented method for generating a performance prediction of a machine learning (ML) model. The computer-implemented method generally includes obtaining a manufacturing dataset including a first model configured to perform a task and unlabeled data. The computer-implemented method also includes generating at least one metric predicting the performance of the first model when performing the task on the manufacturing dataset using a second model. The second model is a meta-model associated with the first model. The computer-implemented method further includes generating at least one value predicting an uncertainty of the at least one metric predicting the performance of the first model when performing the task on the manufacturing dataset using one or more third models. Each of the one or more third models is a meta-meta-model associated with the second model. The computer-implemented method further includes providing an indication of the at least one metric predicting the performance of the first model and the at least one value predicting the uncertainty of the at least one metric.

[0004] Other embodiments include, but are not limited to, a computer program product including a storage medium having computer readable program code that enables a processing unit to perform one or more aspects of the disclosed methods, and a system having a processor, memory and application program configured to perform one or more of the disclosed methods. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram illustrating a networked system used to predict the performance of a model on a task and the uncertainty of the model's performance on a task, according to one embodiment. [Figure 2A] 1 illustrates a stacked meta-modeling workflow for generating model performance predictions and uncertainty intervals for model performance predictions, according to one embodiment. [Figure 2B] 1 illustrates a stacked meta-modeling workflow for generating model performance predictions and uncertainty intervals for model performance predictions, according to one embodiment. [Diagram 3] 1 illustrates an example procedure for generating one or more datasets for training a meta-meta-model, according to one embodiment. [Figure 4] 1 illustrates an example of generating features for training a meta-meta model according to one embodiment. [Diagram 5] 1 is a flowchart of a method for predicting a model's performance on a task and uncertainty for the model's performance on the task, according to one embodiment. [Figure 6] 1 is a flowchart of a method for training a meta-model and a meta-meta-model of a stacked meta-model workflow for generating model performance predictions and uncertainty intervals for model performance predictions according to one embodiment. [Figure 7] 1 is a flowchart of a method for training a meta-meta model according to one embodiment. [Figure 8] 1 illustrates a simulation of an example performance prediction and uncertainty interval, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] ML tools are increasingly used for a variety of tasks, including but not limited to regression and classification, optimization, prediction, and the like. However, in some cases, the performance of a given model in a task may change over time. For example, the model's ability to predict accurately may deteriorate over time depending on how much the underlying data differs from the model's training data. Currently, many conventional techniques can be used to predict the performance of a model in a given task. For example, one conventional technique involves detecting the amount of movement of one or more features of the underlying data. Other conventional techniques involve tracking the accuracy of a model by measuring the proportion of correct predictions (e.g., the number of times any class was correctly predicted, the area under the receiver operating characteristic (ROC) curve, etc.) over time. However, these conventional techniques are used only for performance prediction and do not capture the uncertainty of the model.

[0007] In addition, many conventional techniques exist to predict only uncertainty (e.g., without considering the underlying performance prediction task). Examples of such techniques include, but are not limited to, confidence intervals and probability (p) values. In addition, in Bayesian modeling, there are different types of model uncertainty, including, for example, aleatoric uncertainty and epistemic uncertainty. Aleatoric uncertainty captures the inherent uncertainty of the data, examples of which can include noise, various data missingness, confusion, etc. Epistemic uncertainty captures the uncertainty caused by the model (e.g., model architecture, model parameters, model assumptions, parameter estimation, insufficient training, etc.).

[0008] One problem with the methods described above is that they generally assume that a choice exists over the architecture of the base model, allowing predictions of model performance or model uncertainty, or both. When this choice does not exist, some methods, such as ensemble techniques, can be used to capture uncertainty, but these methods assume that the base model is a white box (e.g., a model whose internal architecture and parameters are visible or known). However, in many situations, when customer-provided models are used, none of the above-mentioned assumptions are acceptable. Therefore, it may be desirable to provide techniques for generating both model performance predictions and model uncertainty predictions.

[0009] Embodiments of the present disclosure provide techniques for predicting the performance of an ML model on an underlying task along with an uncertainty interval for the predicted performance of the ML model. More specifically, given a trained ML model and a set of (unlabeled) manufacturing data (which may or may not differ significantly from the model's training data), embodiments can predict the model's performance (e.g., accuracy or other performance or quality related metric) on the set of manufacturing data and an uncertainty interval (e.g., bands or error bars around the prediction) to account for the uncertainty associated with the model on a particular test instance or batch of test instances.

[0010] In one embodiment, described in further detail below, a stacked meta-modeling method is used to generate performance predictions and uncertainty predictions. For example, an embodiment generates two levels of meta-models: (1) a first level meta-model that predicts the performance of a base model, and (2) a second level meta-model that predicts the uncertainty of the first level meta-model (e.g., with respect to its prediction of the performance of the base model). By using the stacked meta-modeling method, an embodiment can work with models of any architecture (e.g., the stacked meta-modeling method is agnostic to the model architecture and does not depend on having access to the model's inner parameters) and capture multiple types of uncertainty (e.g., aleatory uncertainty, epistemic uncertainty, etc.).

[0011] As used herein, a "meta-model" generally means a model of a (lower-level) model, and a "meta-meta-model" generally means a model of a meta-model. A meta-model captures patterns observed in lower-level models interacting with data, for example. Note that in various embodiments of the present disclosure, a vision task (e.g., processing images for action recognition, object detection, face recognition, digit or letter classification, etc.) is used as an example to illustrate the functionality of an ML model. However, embodiments of the present disclosure are readily applicable to any number of domains, using any input (e.g., video, audio, text, etc.). Additionally, as used herein, the "performance" of a model may refer to one or more accuracy-related metrics of the model on unlabeled data. Similarly, "performance prediction" may refer to a model-based method that predicts one or more performance or quality-related metrics of an underlying base model on unlabeled data.

[0012] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be comprehensive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements over the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed in this specification.

[0013] Reference will now be made to the embodiments shown in this disclosure. However, the scope of the disclosure is not limited to the specific described embodiments. Instead, any combination of the following features and elements, whether related to different embodiments or not, is contemplated for implementing and practicing the discussed embodiments. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or prior art, whether or not a particular advantage is achieved by a given embodiment is not a limitation of the scope of the disclosure. Thus, the following aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the appended claims, unless expressly recited in the claims. Similarly, references to the "invention" should not be construed as a generalization of any inventive subject matter disclosed herein, and should not be considered elements or limitations of the appended claims, unless expressly recited in the claims.

[0014] 1 is a block diagram illustrating a networked system 100 that may be used to generate performance predictions and model uncertainty predictions, according to one embodiment. System 100 includes a computing system 102. Computing system 102 may also be connected to other computers via a network 140. Network 140 may include one or more networks of various types, including a local area or local area network (LAN), a general wide area network (WAN), a telecommunications or cellular network, or a public network (e.g., the Internet), or any combination thereof.

[0015] The computing system 102 generally includes one or more processors 104, storage 108, network interface 110, input devices 152, and output devices 154 connected to memory 106 via bus 150. The computing system 102 is generally under the control of an operating system (not shown). Examples of operating systems include the UNIX® operating system, versions of the Microsoft Windows® operating system, and distributions of the Linux® operating system (UNIX is a registered trademark of The Open Group in the United States and other countries, Microsoft and Windows are trademarks of Microsoft Corporation in the United States and / or other countries, and Linux is a registered trademark of Linus Torvalds in the United States and / or other countries). More generally, any operating system that supports the functionality disclosed herein may be used. Processor 104 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, and the like.

[0016] Memory 106 may include a variety of computer readable media selected for performance or other capabilities, i.e., volatile and / or non-volatile media, removable and / or non-removable media, etc. Memory 106 may include cache, random access memory (RAM), etc. Storage 108 may be a disk drive or flash storage device. Although shown as a single unit, storage 108 may be a combination of fixed and / or removable storage devices, e.g., fixed disk drives, solid state drives, removable memory cards, optical storage devices, network attached storage (NAS) or storage area networks (SAN). Network interface 110 may be any type of network communication device that allows computing system 102 to communicate with other computing systems or devices over network 140.

[0017] The input device(s) 152 provide input to the computing system 102. For example, a keyboard or a mouse or both may be used. The output device(s) 154 may be any device for providing output to a user of the computing system 102. For example, the output device 154 may be any conventional display screen. Although shown separately from the input device 152, the output device 154 and the input device 152 may be combined. For example, a display screen with an integrated touch screen may be used.

[0018] Here, the storage 108 includes a manufacturing dataset 130, a base model 132, a test dataset 134, a training dataset 136, and an offline model and dataset 138. The base model 132 is a model used to analyze a specific task (e.g., a customer model for image classification). In general, any suitable type of model can be used for the base model 132, examples of which include, but are not limited to, artificial neural networks, decision trees, support vector machines, regression analysis, Bayesian networks, genetic algorithms, etc. The base model 132 can be a black box model (e.g., a model whose architecture and / or parameters are not known) or a white box model (e.g., a model whose architecture and / or parameters are known).

[0019] The base model 132 may be trained / developed on a particular training dataset (e.g., training dataset 136) and may be evaluated using a labeled test dataset (e.g., test dataset 134 including classification labels). The production dataset 130 is an unlabeled dataset where ground truth (e.g., classification labels) are not available. The production dataset 130 may be collected from operating conditions after the base model 132 is deployed. The offline models and / or datasets 138 include various models and datasets, some of which may be relevant to the original task of the base model 132 and some of which may not be relevant to the original task of the base model 132. The offline models and / or datasets 138 are described in further detail below with respect to FIGS. 2B and 3.

[0020] The memory 106 includes a prediction engine 120 configured to perform one or more of the techniques described herein to generate performance predictions and uncertainty intervals for the models. The prediction engine 120 includes a performance component 122 and an uncertainty component 124, each of which may include software, hardware, or a combination thereof. The prediction engine 120 is configured to generate a performance prediction (or predict the performance) of the base ML model 132 (via the performance component 122) and generate an uncertainty interval for the performance prediction (via the uncertainty component 124), for example, using a stacked meta-modeling method. For example, the performance component 122 can use a meta-model (of the base model 132) to predict the performance (e.g., one or more accuracy-related metrics) of the base model 132 on the unlabeled production dataset 130. The meta-model may be trained using one or more of the base model 132, the test dataset 134, and the training dataset 136.

[0021] Similarly, the uncertainty component 124 can predict the uncertainty of the meta-model of the performance component 122 using the meta-meta-model (of the meta-model of the base model 132). The meta-meta-model can be trained using one or more of the collections of offline models / datasets 138, the meta-model of the performance component 122, the base model 132, the manufacturing dataset 130, the test dataset 134, and the training dataset 136. In one embodiment, the uncertainty component 124 can generate an uncertainty interval (also known as an error bar) that represents the uncertainty of the meta-model. The uncertainty interval delineates a band (or tolerance) around the output of the performance component 122 (e.g., the predicted performance of the base model 132). This band (or tolerance) around the prediction can account for the uncertainty associated with the base model 132 in a particular test instance (e.g., of the manufacturing dataset 130). Performance component 122 and uncertainty component 124 are described in further detail below with respect to Figures 2A and 2B.

[0022] It should be noted that Fig. 1 merely illustrates one illustrative example of a system 100 capable of generating performance predictions with model uncertainty intervals, and other configurations of the system 100 may be adapted to generate performance predictions with model uncertainty intervals. For example, in some embodiments, one or more contents (e.g., manufacturing data set 130, base model 132, test data set 134, training data set 136, and offline model / data set 138) of storage 108 and / or one or more contents (e.g., performance component 122 and uncertainty component 124) may be distributed across one or more computing systems 102 in a network (e.g., a cloud computing environment). In such an embodiment, a first computing system 102 may retrieve one or more of the contents used to generate a performance prediction of the base model 132 from one or more second computing systems 102. Similarly, the first computing system 102 may retrieve one or more of the contents used to generate the uncertainty interval for the predicted performance of the base model 132 from one or more second computing systems 102. In yet other embodiments, the performance predictions and / or uncertainty intervals can be generated by a single computing system 102 or multiple computing systems 102.

[0023] 2A-2B illustrate a stacked meta-modeling workflow 200 for generating performance predictions of a base model (e.g., base model 132) and uncertainty intervals for the performance predictions of the base model, according to one embodiment, where stacked meta-modeling workflow 200 includes a training phase 210 and a use (or deployment) phase 212 (shown in FIG. 2A) and a training phase 214 and a use phase 216 (shown in FIG. 2B).

[0024] 2A, the performance component 122 includes a meta-model 202 that generally learns to predict the probability of success and failure of the base model 132 on an instance by instance basis. The workflow 200 illustrates many different pieces of information that can be considered during the training phase 210. In the depicted embodiment, the base model 132 and a (labeled) test dataset 134 are processed and used as inputs to train the meta-model 202. As further shown, in some embodiments, a training dataset 136 can also be used as an additional input to train the meta-model 202.

[0025] The training phase 210 depicts a supervised learning method for training the meta-model 202. The training phase 210 can be performed offline (e.g., when the meta-model 202 is not in use or deployed) or online (e.g., when the meta-model 202 is in use or deployed). More generally, however, any suitable machine learning training mechanism consistent with the functionality described herein can be used. It is further noted that while the base model 132, the test dataset 134, and / or the training dataset 136 are shown as directly entering the meta-model 202 during the training phase 210, those skilled in the art will recognize that before any input data is provided to the meta-model 202, this information can be processed in a variety of ways.

[0026] Once the meta-model 202 is trained, the (unlabeled) production dataset 130 (e.g., without classification labels) can be provided as input to the meta-model 202, and the meta-model 202 can output a performance prediction 204 (e.g., during a use (deployment) phase 212). In some cases, the output prediction scores (e.g., softmax, logit) of the base model 132 and / or the original input features of the base model 132 can also be provided as input to the meta-model 202. Note that while the production dataset 130 is shown as going directly into the meta-model 202, one skilled in the art will recognize that this information can be processed in a variety of ways before any input data is provided to the meta-model 202.

[0027] The performance prediction 204 is a predicted measurement of how well (or accurately) the base model 132 will perform at its task (e.g., image classification) on the (unlabeled) manufacturing dataset 130. For example, assuming the base model 132 is a model for object classification, the performance prediction 204 may include a percentage value (e.g., 74%) indicating the probability of success / failure of the base model 132 in classifying objects from the manufacturing dataset 130. More generally, the performance prediction 204 may include one or more accuracy-related metrics of the base model 132 on the manufacturing dataset 130. Examples of such accuracy-related metrics may include, but are not limited to, the area under the ROC score, true positive rate (TPR), F1 value, false positive rate (FPR), R-squared score, precision score (e.g., the percentage of predictions that the model will be accurate), etc.

[0028] The performance prediction 204 can be a batch accuracy prediction or a point accuracy prediction. In some cases, the performance prediction 204 can be used for pre-deployment quality checking of the base model 132 (e.g., to determine if the base model 132 should be deployed). In some cases, the performance prediction 204 can be used for quality control / checking (e.g., filtering inaccurate predictions, identifying production losses, etc.) during run-time or deployment of the base model 132. However, as discussed above, the performance prediction 204 can suffer from multiple sources of uncertainty (e.g., aleatory uncertainty, epistemic uncertainty, etc.) that are not captured in accuracy-related metrics. To address this, embodiments use the uncertainty component 124 to predict an uncertainty interval for the performance prediction 204.

[0029] As shown in FIG. 2B , the uncertainty component 124 includes a meta-meta model 206 that learns to predict the error of the meta model 202 as a function of the actual observed performance (e.g., the absolute value of the predicted accuracy minus the actual accuracy). The workflow 200 illustrates many different pieces of information that can be considered during the training phase 214. In the depicted embodiment, the offline model / dataset 138 and the meta model 202 are processed and used as inputs to train the meta-meta model 206. As further shown, in some embodiments, the manufacturing dataset 130 can also be used as an additional input or the only input to train the meta-meta model 206.

[0030] The training stage 214 depicts a supervised learning method for training the meta-meta model 206. The training stage 214 may be performed offline for one or more runs. However, more generally, any suitable machine learning training mechanism consistent with the functionality described herein may be used. It is further noted that while the offline models / datasets 138, meta-model 202 and / or production datasets 130 are shown as going directly into the meta-meta model 206, those skilled in the art will recognize that before any input data is provided to the meta-meta model 206, this information may be processed in a variety of ways.

[0031] The meta-meta model 206 is configured to learn when the meta model 202 is likely to be wrong and by how much from one or more offline runs of the training phase 214. In one embodiment, the meta-meta model 206 is trained on multiple background offline models / datasets 138 (excluding the manufacturing dataset 130) to learn general features that represent the expected uncertainty of the meta model 202. In other embodiments, the meta-meta model 206 can be trained on the manufacturing dataset 130 (e.g., without the offline models / datasets 138). In yet other embodiments, the meta-meta model 206 can be trained on the offline / models / datasets 138 and the manufacturing dataset 130.

[0032] The offline models / datasets 138 may include labeled datasets that are relevant to the task of the base model 132 (e.g., object classification) and / or labeled datasets that are not relevant to the task of the base model 132 (e.g., speech-to-text conversion). In some embodiments, at least one (first) dataset of the offline models / datasets 138 used to train the meta-meta-model 206 can be dynamically generated from one or more (second) datasets in the offline models / datasets 138. For example, the first dataset can be generated based on resampling one or more features in the second dataset.

[0033] 3 illustrates an example of generating one or more datasets for training the meta-meta model 206, according to one embodiment. As shown, given a (base) labeled dataset 302 (which may be one of the offline models / datasets 138), the prediction engine 120 can generate one or more training / test datasets 304-1 to 304-6 and one or more production datasets 306-1 to 306-6 for training the meta-meta model 206. In one embodiment, the prediction engine 120 can generate the training / test datasets 304 and the production datasets 306 by resampling the (base) labeled dataset 302 based on one or more features of the labeled dataset 302.

[0034] In one example, the training / testing data set 304 and the production data set 306 can be generated by resampling the (base) labeled data set 302 based on different features to introduce a covariate shift. In this example, the particular features of the (base) labeled data set 302 used to resample the distributions of the training / testing data set 304 and the production data set 306 can be dynamically selected by the prediction engine 120 (e.g., based on feature importance). In another example, the training / testing data set 304 and the production data set 306 can be generated by resampling the (base) labeled data set 302 based on actual class labels to introduce a conventional probability shift.

[0035] As shown in FIG. 3, assuming a feature is “Feature A” (e.g., “Dog”) (or “Feature B” (e.g., “Cat”), the proportion of “Feature A” (or “Feature B”) may be varied (e.g., by resampling the feature in the labeled dataset 302) in each of the training / testing datasets 304-1 through 304-6 and the manufacturing datasets 306-1 through 306-6. Here, for example, the training / testing dataset 304-1 may include a statistical distribution of 100% of “Feature A” (or 0% of “Feature B”) and the corresponding manufacturing dataset 306-1 may include a statistical distribution of 0% of “Feature A” (or 100% of “Feature B”). Similarly, at the opposite end, the training / testing dataset 304-6 may include a statistical distribution of 0% of “Feature A” (or 100% of “Feature B”) and the corresponding manufacturing dataset 306-6 may include a statistical distribution of 100% of “Feature A” (or 0% of “Feature B”).

[0036] 2B , in some embodiments, training the meta-meta model 206 (e.g., during the training phase 214) may include generating one or more features based on metadata about the input datasets (e.g., the offline model / dataset 138, which may include the training / test dataset 304, the production dataset 306, etc.) and / or other distribution feature space characteristics of the input datasets. In one embodiment, the one or more features may be based on the base model 132 and the offline model / dataset 138 (which may include the training and production datasets). For example, the prediction engine 120 may calculate various distributions in the test dataset and the production dataset and compare the distances between the various distributions to create one or more features used to train the meta-meta model 206.

[0037] As shown in FIG. 4, for example, a first distribution (histogram 402) can be calculated based on a test data set (e.g., test data set 304) and a second distribution (histogram 404) can be calculated based on a production data set (e.g., production data set 306). The distance 406 (or more generally, the disparity) between the distributions can be calculated and used as features (also called feature values) 408. Generating features 408 to train the meta-meta model 206 in this manner may explain performance predictions that are significantly noisy (e.g., above a threshold) in scenarios where the mean output from the meta model 202 is stable, but the variance is high (e.g., above a threshold) in scenarios where the sample-based uncertainty is too large for the batch-based error bars, etc.

[0038] In one embodiment, the example features 408 may be based on the highest confidence score from the base model 132. For example, the base model 132 may be used to score samples in the test dataset and the production dataset. Assuming that the base model 132 is used for object classification (e.g., between class A and class B), an output prediction score using the base model 132 may be obtained for each sample (or data point) in the test dataset and each sample (or data point) in the production dataset. As an example, for the first sample in the test dataset, the base model 132 may output a class A of 95%, and for the first sample in the production dataset, the base model 132 may output a class A of 90%.

[0039] After all samples in each of the test and production datasets have been scored, a subset of the scored samples in both the test and production datasets can be used to create histograms. For example, a (first) histogram of the highest confidence scores (e.g., between a certain threshold range, e.g., between 90-95%, 95-98%, 98-100%, etc.) from the scored samples in the test dataset can be generated. Similarly, a (second) histogram of the highest confidence scores (e.g., between a certain threshold range) from the scored samples in the production dataset can be generated. The two histograms can then be compared with a divergence function (e.g., a similarity metric, a dissimilarity metric). In one embodiment, a distance 406 (e.g., Hellinger distance) between the two histograms can be calculated, and the distance 406 value (e.g., in [0, 1]) can be used as a feature 408 for the meta-meta-model 206.

[0040] In one embodiment, an example feature 408 can be based on the highest confidence score of the shadow model compared to the highest confidence score of the base model. In this embodiment, the other (proxy) model is trained on the same training dataset (e.g., training dataset 136) used to train the base model 132. The highest confidence scores from the test dataset and the production dataset are then calculated using the proxy model (e.g., as opposed to the base model 132). Histograms can be generated based on the highest confidence scores, and the distance between the histograms can be used as features 408 for the meta-meta model 206.

[0041] In one embodiment, an example feature 408 can be based on class frequency distance. In this embodiment, for example, histograms can be created of the percentage of samples in the test and production data sets that are predicted to be in each class by the base model 132. The distance 406 between the histograms (e.g., Hellinger distance) can be calculated and the distance values ​​can be used as features 408 for the meta-meta-model 206.

[0042] In one embodiment, the example features 408 can be based on highest feature distance. In this embodiment, for example, a shadow random forest model can be trained and used to identify data features with highest feature importance (e.g., meeting a predefined condition). Once identified, the histograms of the test data set and the production data set can be projected into this dimension (e.g., a compressed one-dimensional feature space). The distance 406 between the histograms (e.g., Hellinger distance) can be calculated, and the distance value can be used as the feature 408 for the meta-meta model 206.

[0043] In one embodiment, the example features 408 can be based on a meta-model prediction (e.g., performance prediction 204). In this embodiment, the change in accuracy of the base model between the test dataset and the production dataset as predicted by the meta-model 202 can be used as the features 408 for the meta-meta-model 206. In one embodiment, the example features 408 can be based on a statistical hypothesis test between a first statistical distribution (of a first dataset) and a second statistical distribution (of a second dataset).

[0044] It should be noted that the above features are provided as merely examples of features that may be used to train the meta-meta model 206, which may use other features consistent with the functionality described herein or any combination of features based on the meta data of the input dataset. In some embodiments, for example, multiple meta-meta models may be generated / trained based on different features. In these embodiments, the particular meta-meta model used during the execution phase 216 may be dynamically selected at run-time based on characteristics of the data (e.g., metadata about the dataset, number of features, distribution feature space characteristics, etc.).

[0045] Once the meta-meta model 206 is trained, one or more of the base model 132, the test dataset 134, the training dataset 136, the meta model 202, the production dataset 130, and the performance prediction 204 can be provided as inputs to the meta-meta model 206, which can output (e.g., during a deployment phase 216) an uncertainty prediction 208. The uncertainty prediction 208 is a predicted amount of uncertainty in the performance prediction 204 output from the meta model 202. For example, assuming the base model 132 is a model for object classification, the uncertainty prediction 208 can indicate an interval (or tolerance) (e.g., ±4, ±7, +2) that expresses the amount of uncertainty in the performance prediction 204. In some embodiments, the uncertainty prediction 208 can be used for additional pre-deployment quality checking of the base model 132 or for quality control / checking at run-time or during deployment of the base model 132, or both.

[0046] 5 is a flowchart of a method 500 for predicting a model's performance on a task and an uncertainty for the model's performance on the task, according to one embodiment. The method 500 may be performed by a prediction engine of a computing system (e.g., prediction engine 120 of computing system 102).

[0047] The method 500 may begin at block 502 where a prediction engine obtains a base model (e.g., base model 132) and a set of manufacturing data (e.g., manufacturing dataset 130). At block 504, the prediction engine uses a meta-model (e.g., meta-model 202) to predict the performance (e.g., performance prediction 204) of the base model on the set of manufacturing data. For example, the prediction engine may generate one or more accuracy-related metrics of the base model on the set of manufacturing data. In one particular example, the accuracy-related metric may indicate the success / failure probability (e.g., X% success) of the base model at its task using the set of manufacturing data.

[0048] In block 506, the prediction engine uses the meta-meta model (e.g., the meta-meta model 206) to predict one or more uncertainty intervals (e.g., the uncertainty prediction 208) of the meta model's performance for the set of manufacturing data. For example, the prediction engine can use the set of manufacturing data to generate a tolerance (or error band) (e.g., ±Y) that indicates the amount of uncertainty in the base model's predicted performance at the task.

[0049] In one embodiment, the uncertainty intervals (predicted in block 506) may be asymmetric. For example, the prediction engine may predict a signed value of error (e.g., + or -) that represents the uncertainty of the predicted performance. In another example, a first uncertainty interval can be generated for an upper band / range around the predicted performance and a second uncertainty interval can be generated for a lower band / range around the predicted performance.

[0050] In some embodiments, multiple meta-meta models can be used to generate multiple uncertainty intervals (e.g., a first meta-meta model for a first (upper) uncertainty interval and a second meta-meta model for a second (lower) uncertainty interval). In some embodiments, multiple uncertainty intervals can be generated by a single meta-meta model. For example, a single meta-meta model can include two sub-modules (or components) for predicting upper and lower uncertainty intervals.

[0051] At block 508, the prediction engine provides an indication of the performance of the base model and one or more uncertainty intervals. In one embodiment, the prediction engine can provide an indication on a user interface of a display of a computing device (e.g., computing system 102). In one embodiment, the prediction engine can provide the indication in response to a request for a performance prediction and an uncertainty prediction of a base model for a set of manufacturing data. For example, a user may use the prediction engine to determine which base model (of multiple base models) should be used for a set of manufacturing data. In another example, the prediction engine may monitor the deployment / runtime of the base model and provide an indication to the computing device continuously (e.g., at one or more predetermined time intervals). The indication can then be used to determine when the base model should be improved and / or replaced. In some cases, the prediction engine may run a meta-meta model as frequently as a meta-model.

[0052] 6 is a flowchart of a method 600 for training a meta-model and a meta-meta-model of a stacked meta-model workflow to generate model performance predictions and uncertainty intervals for model performance predictions according to one embodiment. Method 600 may be performed by a prediction engine of a computing system (e.g., prediction engine 120 of computing system 102).

[0053] The method 600 may begin at block 602 where the prediction engine retrieves a base model (e.g., base model 132), a test dataset (e.g., test dataset 134), and one or more additional datasets (e.g., offline model / dataset 138). At block 604, the prediction engine trains a meta-model (e.g., meta-model 202) to predict the performance of the base model at a task based on the base model and the test dataset. At block 606, the prediction engine trains a meta-meta-model to predict the uncertainty of the performance of the base model at the task based on the meta-model and the one or more additional datasets.

[0054] 7 is a flowchart of a method 700 for training a meta-meta model according to one embodiment. The method 700 may be performed by a prediction engine of a computing system (e.g., prediction engine 120 of computing system 102).

[0055] The method 700 may begin at block 702, where a prediction engine obtains a labeled dataset (e.g., labeled dataset 302). At block 704, the prediction engine generates one or more additional datasets (e.g., training / test dataset 304, production dataset 306) based on the labeled dataset. At block 706, the prediction engine determines (or calculates) one or more features (e.g., features 408) based on an evaluation of the one or more additional datasets. At block 708, the prediction engine trains one or more meta-meta models (e.g., meta-meta model 206), each configured to predict an uncertainty interval of a meta model (e.g., meta model 202) based at least in part on the one or more features.

[0056] FIG. 8 illustrates example simulations 802 and 804 of a base model performance prediction and an uncertainty interval for the performance prediction, according to one embodiment.

[0057] More specifically, simulation 802 depicts performance predictions of the base model across different test / production data sets 1-K. As shown, the base model accuracy (e.g., actual accuracy of the base model) is represented by line 808, the predicted accuracy of the base model is represented by line 810, and error bars for the predicted accuracy are represented by 806. Each test / production data set 1-K may have a different statistical distribution of features / labels. For example, test / production data set 1 may include a statistical distribution of 100% "Feature A" in the test data set and 0% "Feature A" in the production data set. Similarly, at the opposite end, test / production data set K may include a statistical distribution of 0% "Feature A" in the test data set and 100% "Feature A" in the production data set.

[0058] Simulation 804, which corresponds to simulation 802, shows the meta-model error (represented by 840), a fixed error bar (or uncertainty value) 820, and the delta between the meta-model error and the actual base model output 830. As shown, by using a stacked meta-model workflow to generate uncertainty intervals for model performance predictions, embodiments can generate more accurate uncertainty intervals as compared to using a fixed uncertainty value 820.

[0059] The above refers to the embodiments shown in this disclosure. However, the scope of the disclosure is not limited to the specific described embodiments. Instead, any combination of features and elements, whether related to different embodiments or not, is contemplated for implementing and practicing the discussed embodiments. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or prior art, whether or not a particular advantage is achieved by a given embodiment is not a limitation of the scope of the disclosure. Thus, the aspects, features, embodiments and advantages described herein are merely illustrative and are not considered elements or limitations of the appended claims, unless expressly recited in the claims. Similarly, references to the "invention" should not be construed as a generalization of any inventive subject matter disclosed herein, and should not be considered elements or limitations of the appended claims, unless expressly recited in the claims.

[0060] Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.), or an embodiment combining software and hardware aspects, which may be generally referred to herein as "circuits," "modules," or "systems."

[0061] The present invention may be a system, a method, and / or a computer program product, which may include a computer readable storage medium having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0062] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, without limitation, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards or mechanically encoded devices such as ridge structures in grooves with instructions recorded thereon, and any suitable combination of the foregoing. A computer-readable storage medium as used herein should not be construed as a signal that is itself ephemeral, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable) or an electrical signal transmitted through a wire.

[0063] The computer readable program instructions described herein can be downloaded from the computer readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.

[0064] The computer readable program instructions for carrying out the operations of the present invention may be either source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or conventional procedural programming languages, such as object oriented programming languages, such as Smalltalk®, C++, and the like, as well as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0065] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer readable program instructions.

[0066] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing device to produce a machine such that the instructions, executed by the processor of the computer or other programmable data processing device, create means for performing the functions / acts specified in the blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium capable of directing a computer, programmable data processing device or other device or combination thereof to function in a particular manner such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions implementing aspects of the functions / acts specified in the blocks of the flowcharts and / or block diagrams.

[0067] Also, the computer readable program instructions may be loaded into a computer, other programmable data processing device or other device such that the instructions, which execute on a computer, other programmable device or other device, perform the functions / acts specified in the blocks of the flowcharts and / or block diagrams, and cause a series of operational steps to be performed on the computer, other programmable device or other device to generate a computer-implemented process.

[0068] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0069] An embodiment of the present invention may be provided to end users through a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as computing power that provides an abstraction between the computing resources and their underlying technology architecture (e.g., servers, storage, network), enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal administrative effort or service provider interaction. Thus, cloud computing allows users to access virtual computing resources (e.g., storage, data, applications, and even fully virtualized computing systems) in the "cloud," regardless of the underlying physical systems used to provide the computing resources (or the location of those systems).

[0070] Typically, cloud computing resources are provided to users on a pay-per-use basis, and users are charged only for the computational resources actually used (e.g., the amount of storage space consumed by the user or the number of virtualization systems instantiated by the user). Users can access any of the resources in the cloud at any time and from anywhere across the Internet. In the context of the present invention, users can access the available applications (e.g., prediction engine 120) or related data (e.g., base model 132, test dataset 134, training dataset 136, offline model / dataset 138, production dataset 130, etc.) in the cloud. For example, prediction engine 120 can run on a computing system in the cloud and predict the performance of the base model along with an uncertainty interval for the predicted performance of the base model. In such a case, the application can retrieve one or more of the input information used to generate the performance prediction and / or uncertainty prediction from a storage location in the cloud and store the performance prediction and / or uncertainty prediction in a storage location in the cloud.

[0071] While the forgoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, which scope is determined by the following claims.

Claims

1. 1. A computer-implemented method for generating a performance prediction for a machine learning (ML) model, comprising: obtaining a manufacturing dataset including a first model configured to perform a task and unlabeled data; generating at least one metric that predicts performance of the first model when performing the task on the manufacturing dataset using a second model, the second model being a meta-model associated with the first model; generating at least one value predicting an uncertainty of the at least one metric that predicts the performance of the first model when performing the task on the manufacturing dataset using one or more third models, each of the one or more third models being a meta-meta model associated with the second model; providing an indication of the at least one metric predicting the performance of the first model and the at least one value predicting the uncertainty of the at least one metric; 23. A computer-implemented method comprising:

2. the first model is a black box model; 2. The computer-implemented method of claim 1.

3. the first model is a white box model; 2. The computer-implemented method of claim 1.

4. At least one of the one or more third models is trained on (i) a training dataset for the first model and (ii) the second model.

2. The computer-implemented method of claim 1.

5. At least one of the one or more third models is trained on (i) a training dataset for the first model, (ii) one or more additional datasets, and (iii) the second model.

2. The computer-implemented method of claim 1.

6. At least one of the one or more third models is trained on (i) one or more additional datasets and (ii) the second model.

2. The computer-implemented method of claim 1.

7. said at least one of said one or more third models is trained offline on said one or more additional datasets and said second model; 7. The computer-implemented method of claim 6.

8. the one or more additional datasets comprising at least one dataset including a first set of labels for classifying data in the at least one dataset; the first set of labels is different from a second set of labels for classifying data in a dataset used to train the second model; 7. The computer-implemented method of claim 6.

9. The one or more additional datasets comprise at least one labeled dataset, and the computer-implemented method further comprises: generating a plurality of test data sets and a plurality of production data sets based on the labeled data sets; including the plurality of test data sets and the plurality of production data sets in the one or more additional data sets.

7. The computer-implemented method of claim 6.

10. generating the plurality of test data sets and the plurality of production data sets includes resampling one or more features of the labeled data set such that each test data set and corresponding production data set includes a different ratio of the one or more features relative to the labeled data set.

10. The computer-implemented method of claim 9.

11. The computer-implemented method further includes generating at least one feature value for training the one or more third models based at least in part on a first statistical distribution of a first dataset in the one or more additional datasets and a second statistical distribution of a second dataset in the one or more additional datasets, wherein the at least one of the one or more third models is further trained with the one or more feature values.

7. The computer-implemented method of claim 6.

12. the at least one feature value is further generated based on a divergence function between the first statistical distribution and the second statistical distribution.

12. The computer-implemented method of claim 11.

13. the at least one feature value is further generated based on a statistical hypothesis test between the first statistical distribution and the second statistical distribution.

12. The computer-implemented method of claim 11.

14. the one or more third models are a subset of a plurality of third models, the computer-implemented method further comprising selecting the one or more third models from the plurality of third models based at least in part on one or more features of the one or more additional datasets.

2. The computer-implemented method of claim 1.

15. the at least one value predicting the uncertainty of the at least one metric comprises an interval range.

2. The computer-implemented method of claim 1.

16. the interval range is an asymmetric interval range comprising a first upper range and a second lower range; 16. The computer-implemented method of claim 15.

17. the first upper range and the second lower range are generated via one of the one or more third models.

17. The computer-implemented method of claim 16.

18. Through the plurality of third models, a third model for an upper uncertainty interval and a third model for a lower uncertainty interval are generated.

17. The computer-implemented method of claim 16.

19. 1. A system comprising: one or more computer processors; a memory containing a program, The program, when executed by the one or more computer processors, performs operations for generating a performance prediction of a machine learning (ML) model, the operations including: obtaining a manufacturing dataset including a first model configured to perform a task and unlabeled data; generating at least one metric that predicts performance of the first model when performing the task on the manufacturing dataset using a second model, the second model being a meta-model associated with the first model; generating at least one value predicting an uncertainty of the at least one metric that predicts the performance of the first model when performing the task on the manufacturing dataset using a third model, the third model being a meta-meta model associated with the second model; providing an indication of the at least one metric predicting the performance of the first model and the at least one value predicting the uncertainty of the at least one metric; A system including:

20. A computer-readable storage medium having a computer program recorded thereon, the computer program being for causing a computer to execute each step of the method according to any one of claims 1 to 18.

Citation Information

Patent Citations

  • Data analysis system and data analysis method

    JP2016004525A

  • Imbalance price prediction device, method, program and electric power trading system

    JP2018045615A

  • Estimation method, estimation device, and estimation program

    JP2019113915A

  • Binary prediction tree modeling with many predictors and its uses in clinical and genomic applications

    US20090319244A1

  • Accuracy-estimating-model generating system and accuracy estimating system

    WO2016152053A1