Method and apparatus for reliable prediction based on sample difficulty

Abstract

A computer-implemented method for training a machine learning model is disclosed. The method comprises the following steps: obtaining a training set comprising a plurality of training samples; punishing the training loss of the machine learning model by using a regularization term weighted by the learning difficulty of each sample; and training a machine learning model based on the penalty training loss.

Description

Technical Field

[0001] Various aspects of this disclosure generally relate to artificial intelligence, and more particularly to methods and apparatus for providing reliable predictions based on uncertainty regularization of known sample difficulty. Background Technology

[0002] Model training often encounters blurry or even distorted samples. Learning from these samples is difficult—directly forcing the model to fit them can lead to undesirable memory and overconfidence. Modern neural networks have been found to be poorly calibrated and to make overconfidence predictions regardless of the inherent sample difficulty and data uncertainty.

[0003] The surge in existing techniques focuses on uncertainty regularization, mitigating overfitting and overconfidence in deep neural networks by adding implicit or explicit regularization terms to the training loss. For example, L p Norm and entropy regularization (ER) can be used to explicitly force small norm log-odds (logit) or high predictive entropy. However, L p Norms and ER do not adjust prediction confidence based on sample difficulty. As another example, Correctness Ranking Loss (CRL) can be used to regularize confidence based on the frequency of correct predictions during training loss. However, CRL only involves pairwise rankings within the same batch.

[0004] In other words, it is necessary to cleverly adjust the training objective for each sample based on the sample difficulty of each sample in order to achieve better generalization and uncertainty quantification. Summary of the Invention

[0005] The following is a simplified overview of one or more aspects to provide a basic understanding of these aspects. This overview is not a comprehensive overview of all anticipated aspects, and is neither intended to identify key or important elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0006] While deep neural networks (DNNs) outperform humans on many tasks, they still miss the point of understanding what they don't know. When faced with ambiguous or even unseen input, they often make incorrect but confident predictions. This hinders their deployment in safety-critical applications.

[0007] Data ambiguity and uncertainty significantly affect the difficulty of predicting a given label, or in other words, the difficulty of learning a sample, i.e., sample difficulty. Training a model to predict a given label for each training sample regardless of their inherent uncertainty is suboptimal, leading to undesirable memorization and poor calibration at test time. Therefore, disclosed herein is a sample difficulty-aware instance-wise uncertainty regularization term added to the training loss to penalize overconfident predictions based on the learning difficulty of each sample.

[0008] Such an instance-wise training loss formulation adjusts the learning objective in a fine-grained but simple manner, especially remedying the unjustified training loss formulation for difficult samples (inputs with incorrect labels or without discriminative cues). It leads to improved generalization performance and the quality of uncertainty estimates.

[0009] In a further step, the model trained with the disclosed sample difficulty-aware instance-wise uncertainty regularization term can be used with various uncertainty measures for downstream usage, such as active learning, anomaly detection, and selective classification, etc.

[0010] In an aspect, a computer-implemented method for training a machine learning model is disclosed. The computer-implemented method comprises: obtaining a training set comprising a plurality of training samples; penalizing a training loss of the machine learning model with a regularization term weighted by a learning difficulty of each sample; and training the machine learning model based on the penalized training loss.

[0011] In a further aspect, the learning difficulty of each sample is normalized into a range of (0, 1) as a sample-specific weighting for the regularization term.

[0012] In a further aspect, the sample-specific weighting is multiplied with a coefficient to control the global strength of the regularization term.

[0013] In a further aspect, the training loss is an entropy loss.

[0014] In a further aspect, the regularization term is an entropy regularization term or L p at least one of the norms.

[0015] In a further aspect, the plurality of training samples are at least one of digital images or audio signals.

[0016] In an aspect, a computer-implemented method for performing a classification task using a machine learning model trained with one or more methods disclosed herein is disclosed. The computer-implemented method includes feeding an input to the trained machine learning model; estimating a prediction uncertainty based on a task-relevant prediction for the input; and determining how to handle the input or the task-relevant prediction for the input based on a comparison of the prediction uncertainty to a threshold.

[0017] In a further aspect, estimating a prediction uncertainty based on a task-relevant prediction for an input includes taking a maximum logit output of logit outputs corresponding to each task-relevant class for the input as the prediction uncertainty; or taking a maximum softmax layer output of softmax layer outputs corresponding to each task-relevant class for the input as the prediction uncertainty; or taking an entropy of all softmax layer outputs corresponding to each task-relevant class for the input as the prediction uncertainty.

[0018] In a further aspect, an input to the trained machine learning model is captured by at least one of a sensor, a camera, or a scanner.

[0019] In a further aspect, determining how to handle the input based on a comparison of the prediction uncertainty to a threshold includes selecting the input to be labeled for use as training data for further learning based on the prediction uncertainty being above the threshold.

[0020] In a further aspect, determining how to handle the input based on a comparison of the prediction uncertainty to a threshold includes determining the input as an anomaly based on the prediction uncertainty being above the threshold.

[0021] In a further aspect, determining how to handle the task-relevant prediction based on a comparison of the prediction uncertainty to a threshold includes determining to not accept the task-relevant prediction based on the prediction uncertainty being above the threshold.

[0022] In an aspect, a computer system is disclosed, the computer system comprising one or more processors; and one or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform operations of one or more of the methods disclosed herein.

[0023] In an aspect, one or more computer-readable storage media storing computer-executable instructions that, when executed, cause one or more processors to perform operations of one or more methods disclosed herein are disclosed.

[0024] In an aspect, a computer program product comprising computer-executable instructions that, when executed, cause one or more processors to perform the operations of one or more methods disclosed herein is disclosed. BRIEF DESCRIPTION OF DRAWINGS

[0025] The disclosed aspects will be described with respect to the enclosed figures, which are provided to illustrate and not to limit the disclosed aspects.

[0026] Figure 1 Samples that are difficult to learn and easy to learn are shown in accordance with aspects of the disclosure.

[0027] Figure 2 An exemplary flowchart for training a machine learning model is shown in accordance with aspects of the disclosure.

[0028] Figure 3 An exemplary flowchart for a classification task using a machine learning model trained with one of the disclosed methods is shown in accordance with aspects of the disclosure.

[0029] Figure 4 An exemplary computer system in accordance with various aspects of the disclosure is shown. DETAILED DESCRIPTION

[0030] The present disclosure will now be discussed with reference to several example implementations. It should be understood that the discussion of these implementations is merely meant to provide a better understanding of and to therefore enable the implementation of the present disclosure to those of ordinary skill in the art, and is not intended to suggest any limitation as to the scope of the present disclosure.

[0031] Various implementations will be described in detail with reference to the drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References to examples and implementations are for illustrative purposes and are not intended to limit the scope of the present disclosure.

[0032] Data ambiguity and uncertainty significantly affect the difficulty of predicting a given label, or in other words, the difficulty of learning a sample, i.e., sample difficulty. It can be clearly observed that a dataset is typically composed of samples with different levels of difficulty, e.g., as shown in Figure 1 where (a) is illustrated as a difficult sample for containing multiple salient or ambiguous objects, and (b) is illustrated as an easy sample for containing a distinct cue for classification.

[0033] There is no strict definition of the notion of sample difficulty. Intuitively, easy-to-learn samples will repeat in a form that shows similar patterns. The repeating patterns specific to each class are valuable cues for classification. Moreover, they neither contain confusing information nor conflicting information. In contrast, hard-to-learn samples tend to be challenging for classification because they lose distinguishing cues or contain ambiguous information. For example, a single-labeled image that contains multiple salient objects belonging to different classes or has a wrong label will be a difficult sample.

[0034] From another perspective, easy-to-learn samples are generally less important for training because they usually have easy-to-learn class distinction features, so pruning a training dataset with a portion of easy-to-learn samples will not bring a huge sacrifice of test accuracy. While hard-to-learn samples will be more important for training because they have much more impact on test accuracy when removed.

[0035] The standard training loss formula is agnostic to sample difficulty. Training a model to predict a given label for each training sample regardless of their inherent uncertainty is suboptimal, leading to undesirable memorization and poor calibration at test time. Although different regularization-based methods have been proposed to address this issue, they all ignore the difference between simple and hard samples and can assign inappropriate regularization to some samples. Therefore, this disclosure adds an instance-wise uncertainty regularization term that is aware of sample difficulty to the training loss to penalize overconfident predictions based on the learning difficulty of each sample.

[0036] Take the classification task as an example. Let denote a classifier parameterized by a deep neural network, whose output is a conditional probability distribution . Typically, the deep neural network is trained by minimizing the training loss on the training set . As an example but not limitation, the training loss can be the cross-entropy loss as follows: (1) where denotes the predicted probability for the ground truth label .

[0037] As previously discussed, the deep neural network tends to make overconfident predictions. In addition to overfitting reasons, this is also because the ground truth label is usually a one-hot vector, which represents the highest confidence regardless of sample difficulty. In view of this, a regularization term that is aware of sample difficulty is proposed to regularize the training loss. Still taking the classification task as an example, the training loss can be modified as follows: (2) where can be any suitable regularization term, e.g., an entropy regularization term or L p norm, but any suitable regularization term is possible, and in particular relates to the instance-wise sample difficulty scores presented herein. To penalize overconfident predictions, harder samples will have higher , in contrast, easier samples will have lower .

[0038] In an example, the sample-specific weighting can be a normalization of the sample difficulty score, so as to map each sample difficulty score to the range Examples are given below but are not limited to: (3)

[0039] The skilled person anticipates that any normalization measure is possible, such as min-max normalization, Z-score normalization, etc.

[0040] Since some regularizations encourage uncertain predictions, the sample adaptive weighting is proportional to the sample difficulty score , where the parameter equals a small constant (e.g., 1e-3) that ensures the value range . The parameter is adjustable to control the relative importance between all training data. As , the sample-specific weighting tends to be equal, i.e., converges to a constant weighting, regardless of the actual sample difficulty score. In contrast, produces very small weights for most samples except for very difficult samples, i.e., only the confidence on the most difficult samples is penalized.

[0041] In another example, the sample-specific weighting can be multiplied by a coefficient that controls the global strength of the regularization, as follows: (4) where can be a hyperparameter, and the value of .

[0042] To assess the sample difficulty or importance of a sample used for training, the sample difficulty score can be computed in any suitable way. Several methods are presented herein as examples and not limitation.

[0043] As a first embodiment, by treating the importance of a sample as the learning difficulty of the sample, a sample difficulty score can be calculated along with the training process, since easy-to-learn samples are generally less important for training, while difficult-to-learn samples are the opposite.

[0044] Let us consider supervised classification, where Represents the training set, which has input vectors. One-hot vector of the encoded tag For a fixed neural network architecture, let It is a neural network's response to input With weight The logit output. Let This represents the output of a neural network in the form of a probability vector. For any probability vector... ,make This represents the cross-entropy loss, similar to equation (1).

[0045] make It is an iteration of stochastic gradient descent (SGD), which, at any training step, is given the current position. Training example The contribution to loss reduction on any other example is determined by the gradient norm. Defining. Therefore, the expected value of the gradient norm. This can be considered a score indicating the difficulty of the sample. In other words, it is expected that examples with low scores will have a limited impact on learning how to classify the rest of the training data at a given time, making them easier to learn.

[0046] As another example, for ease of calculation, the expected value of the gradient norm can be approximated by the norm of the error vector after several training epochs, where the error vector is the predicted class probability minus the one-hot label encoding, yielding a score indicating the sample difficulty. .

[0047] Based on experience, the above method will be effective even when computed early in training, and its performance is significantly better than a random baseline, even when computed at initialization. Furthermore, the scoring mechanism works better when computed with weights early in training for multiple trajectories; therefore, computed scores through several independently run ensembles, such as averaging these norms over multiple weight initializations, would be a preferred implementation.

[0048] As a second embodiment, sample difficulty scores can be computed based on pre-trained models. Large-scale pre-trained models learn to encode rich data semantics, which is generally considered beneficial for a wide range of applications. Instead of using them as backbone networks for downstream tasks, they can also help score sample difficulty, where a simple distance metric is sufficient to represent similarity.

[0049] In the case of pre-training performed in a self-supervised manner, clustering can be performed before calculating the learning difficulty of each sample because there are no ground truth labels. As an example, k-means clustering can be performed in the embedding space of the pre-trained self-supervised model, and the difficulty of each data point is defined by the distance to its nearest cluster centroid or prototype (such as cosine distance, Euclidean distance, Manhattan distance, Hamming distance, etc.). Therefore, the easy (hard) examples are the closest (least) to the prototype.

[0050] As a third embodiment, sample difficulty estimation can be performed in the feature space of the pre-trained model and projected as a density estimation problem, since samples with typical distinguishing features are easier to learn and typical features will recur.

[0051] Taking supervised learning as an example, consider the downstream training set. It can be a set of image-label pairs, where as well as These are respectively used as images and their labels. The feature distribution will be modeled with and without class information as a condition.

[0052] In the example, for a single object detection task, the output of the intermediate layer of the pre-trained model would be preferred to be the penultimate layer output, because the output features can be learned well for a particular class. In another example, for a multi-object detection task, the output of the intermediate layer of the pre-trained model would be preferred to be the earlier layer output, because the output features have not yet been focused on a specific class.

[0053] This can be achieved, for example, by fitting any suitable model to a class belonging to the same category. eigenvectors To model distributions conditioned on classes, such as, but not limited to, Gaussian and Bernoulli distributions. β Distributions include gamma distribution, chi-square distribution, or the distribution of training data can be learned from training deep probabilistic models.

[0054] As another example, in the case of semi-supervised learning, since not all samples in the training set have ground truth labels, features can be modeled based on samples with corresponding labels, and for unlabeled samples, the ground truth labels can be replaced by the closest class labels.

[0055] As another example, in fully unsupervised learning, since all samples are unlabeled, features can be pre-clustered, and ground truth labels can be replaced with the indices of the closest clusters. Then, a distribution conditioned on clusters, rather than on classes, can be derived.

[0056] In addition to class-conditioned distributions, class-agnostic distributions can be obtained by fitting all feature vectors regardless of their class.

[0057] To score the sample difficulty, the learning difficulty of each sample in the training set can be quantified based on the training data distribution, based on the difference between the distance of a sample’s features to other samples from the same class and the distance of the sample’s features to all other samples in the training set. Any suitable method can be used, such as Mahalanobis distance, Euclidean distance, Manhattan distance, cosine distance, Hamming distance, etc.

[0058] As an example, each training sample is fed into a single pre-trained model, such as CLIP, or more than one pre-trained model, which can be of the same type or different types, and the output, which can be a feature map or a vector, can be collected and used to evaluate the sample difficulty. The sample difficulty derived from more than one pre-trained model can be integrated to obtain a final sample difficulty score. It is contemplated that any suitable integration method can be used for those skilled in the art.

[0059] Figure 2 An exemplary flowchart for training a machine learning model is shown in accordance with aspects of the present disclosure. As described below, some or all of the features shown can be omitted in certain implementations within the scope of the present disclosure, and some of the features shown can not be required for implementation of all embodiments. In addition, some blocks can be performed in parallel or in a different order. In some examples, the method can be performed by any suitable means or unit for performing the functionality or the algorithm described below.

[0060] The disclosed method of training a machine learning model can be used for tasks such as, but not limited to, classification tasks.

[0061] The method begins at block 201, where a training set comprising a plurality of training samples is obtained.

[0062] The training set for the machine learning model will contain samples, such as digital images or audio signals, which in some cases can be pre-processed into any suitable form. In an example, the plurality of training samples are one of the following types of digital images: e.g., videos, radar images, lidar images, ultrasound images, motion images, and thermal images, or audio signals, or any other type of data or signal acquired by at least one of a sensor, a camera, or a scanner, etc.

[0063] The method then proceeds to block 202, where the training loss of the machine learning model is penalized with a regularization term weighted by the learning difficulty of each sample.

[0064] In an example, the learning difficulty of each sample is normalized to the range of (0, 1) as a sample-specific weighting for the regularization term.

[0065] In another example, the sample-specific weighting is multiplied with the coefficient to control the global strength of the regularization term.

[0066] In an example, the training loss is an entropy loss.

[0067] In an example, the regularization term is an entropy regularization term or L p one of the norms.

[0068] In an example, the learning difficulty of each sample is quantified by its corresponding expected value of the loss gradient norm term based on an ensemble of multiple early-trained machine learning models trained in a supervised manner, where the loss gradient norm bounds the contribution of the sample to the change of the training loss of the machine learning model.

[0069] In another example, the learning difficulty of each sample is quantified by its corresponding expected value of the norm term of the error vector based on an ensemble of multiple early-trained machine learning models as previously described, where the error vector is the L2 distance between the predicted class probability and the one-hot label.

[0070] In another example, another machine learning model is pre-trained in a self-supervised manner, and k-means clustering is performed in the embedding space of the pre-trained machine learning model. The learning difficulty of each sample is quantified by the cosine distance to its nearest cluster centroid or prototype. In other examples, the learning difficulty of each sample is quantified by any suitable distance to its nearest cluster centroid or prototype, such as Euclidean distance, Manhattan distance, Hamming distance, etc.

[0071] In another example, at least one machine learning model is pre-trained, and the training data distribution is modeled in the feature space of the at least one pre-trained model with and without conditioning on the class-related information. Based on the training data distribution, the learning difficulty of each sample is quantified by the difference between the distance of the sample’s feature to other samples from the same class and the distance of the sample’s feature to all other samples in the training set.

[0072] In another example, the training data distribution can be modeled by one of Gaussian distribution, Bernoulli distribution, Beta distribution, Gamma distribution, Chi-square distribution, etc. In another example, the training data distribution is learned from a training deep probabilistic model.

[0073] In another example, the distance of a sample's features to other samples from the same class and the distance of the sample's features to all other samples in the training set are evaluated by any suitable method, such as Mahalanobis distance, Euclidean distance, Manhattan distance, Cosine distance, Hamming distance, etc.

[0074] In an example, the sample difficulty of each sample is quantified based on a single pre-trained model, and in another example, the sample difficulty of each sample is quantified based on an ensemble of more than one pre-trained model.

[0075] The method then proceeds to block 203, where the machine learning model is trained based on the penalized training loss.

[0076] The trained machine learning model can be used in conjunction with uncertainty measurements to make more reliable predictions. Several real-world applications will be active learning, anomaly detection, or selective classification, among others.

[0077] Figure 3 An exemplary flowchart for a classification task using a machine learning model trained with one of the disclosed methods is shown in accordance with aspects of the present disclosure. As described below, some or all of the features shown can be omitted in certain implementations within the scope of the present disclosure, and some of the features shown can not be required for implementation of all embodiments. Moreover, some blocks can be performed in parallel or in a different order. In some examples, the method can be performed by any suitable means or unit for performing the functionality or algorithm described below.

[0078] The method begins at block 301, where an input is fed to the trained machine learning model. The input to the machine learning model will be digital images or audio signals, among others, which can be pre-processed into any suitable form in some cases. In an example, the input is one of the following types of digital images: such as videos, radar images, lidar images, ultrasound images, motion and thermal images, or audio signals, or any other type of data or signal obtained by at least one of a sensor, a camera, or a scanner, among others.

[0079] The method then proceeds to block 302, where a prediction uncertainty is estimated based on a task-related prediction for the input.

[0080] In an example, the task-related prediction can be a probability corresponding to each of all classes. In another example, the task-related prediction can be a probability corresponding to the most likely class. In another example, the task-related prediction can be the most likely class.

[0081] In an example, the maximum logit output among the logit outputs corresponding to each task-relevant class for the input is taken as the prediction uncertainty. In another example, the maximum softmax layer output of the softmax layer outputs corresponding to each task-relevant class for the input is taken as the prediction uncertainty. In another example, the entropy of all softmax layer outputs corresponding to each task-relevant class for the input is taken as the prediction uncertainty.

[0082] The method then proceeds to block 303, where it is determined how to handle the input or the task-relevant prediction for the input based on a comparison of the prediction uncertainty to a threshold. It is natural to the skilled person that while we do not have ground truth prediction uncertainty annotations for each sample, it is expected that more uncertain predictions should be more likely to be wrong.

[0083] In the active learning scenario, it iterates between training the model and selecting new inputs to be labeled. Data with uncertainty measure above the threshold will be considered unknown data, and thus preferably labeled. In different computational rules, uncertainty measure below the threshold can also indicate that the input should be labeled for further training.

[0084] In the anomaly detection scenario, uncertainty measure above the threshold can indicate that the input is unknown to the model, and thus the input is determined to be an anomaly based on the uncertainty measure above the threshold. In different computational rules, uncertainty measure below the threshold can also indicate that the input can be an anomaly input to the model.

[0085] In the selective classification scenario, a key real-world application of calibrated predictions is to make the model aware of what it does not know. Thus, rejecting the task-relevant prediction when the uncertainty measure is above the threshold indicates that the input can be an unknown input to the model. In different computational rules, uncertainty measure below the threshold can also indicate that the input can be an unknown input to the model.

[0086] The disclosed method adjusts the learning objective in a fine-grained but simple way via the per-instance training loss formula, especially to remedy the unreasonable training loss formula for difficult samples (inputs with wrong labels or without discriminative cues). It leads to improved generalization performance and the quality of uncertainty estimation.

[0087] Figure 4 An example computer system in accordance with various aspects of the present disclosure is shown. The computer system can include at least one processor 410. The computer system can also include at least one storage device 420. It should be appreciated that the storage device 420 can store computer-executable instructions that, when executed, cause the processor 410 to perform the operations in accordance with the methods described in connection with Figures 1-3Any of the operations described with respect to the embodiments of the present disclosure.

[0088] Embodiments of the present disclosure can be embodied in one or more computer readable media, such as non-transitory computer readable media. The non-transitory computer readable medium can store instructions that, when executed, cause one or more processors to perform operations according to embodiments of the present disclosure. Figures 1-3 Any of the operations described with respect to the embodiments of the present disclosure.

[0089] Embodiments of the present disclosure can be embodied in a computer program product including computer executable instructions that, when executed, cause one or more processors to perform operations according to embodiments of the present disclosure. Figures 1-3 Any of the operations described with respect to the embodiments of the present disclosure.

[0090] Embodiments of the present disclosure can be embodied in a vehicle including one or more processors configured to perform operations according to embodiments of the present disclosure. Figures 1-3 Any of the operations described with respect to the embodiments of the present disclosure. In an embodiment, the one or more devices can be an electronic control unit (ECU), an electronic management unit (EMU), a parking assist system, an autonomous driving system, etc. In another embodiment, the one or more devices can be any computable unit in a vehicle.

[0091] It should be understood that all of the operations of the above-described methods are merely exemplary and the present disclosure is not limited to any of the operations in the methods or the order of the operations, and should cover all other equivalents under the same or similar concepts.

[0092] It should also be understood that all of the modules in the above-described devices can be implemented in various methods. The modules can be implemented as hardware, software, or a combination thereof. In addition, any of the modules can be further divided into sub-modules or combined together in function.

[0093] The preceding description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein. All elements described throughout this disclosure are meant to be examples of elements that can be used in the various aspects of the disclosure, and are not meant to be limiting. Any element described throughout this disclosure can be used in any other aspect of the disclosure in place of, or in addition to, any element described in that aspect.

Claims

1. A computer-implemented method for training a machine learning model, comprising: obtaining a training set comprising a plurality of training samples; penalizing a training loss of the machine learning model with a regularization term weighted by a learning difficulty of each sample; and training the machine learning model based on the penalized training loss.

2. The computer-implemented method of claim 1, further comprising: normalizing the learning difficulty of each sample to a range of (0, 1) as a sample-specific weighting for the regularization term.

3. The computer-implemented method of claim 2, further comprising: multiplying the sample-specific weighting with a coefficient to control a global strength of the regularization term. the training loss is an entropy loss.

4. The computer-implemented method of claim 1, wherein, the plurality of training samples are at least one of digital images or audio signals.

5. The computer-implemented method of claim 1, wherein, The regularization term is an entropy regularization term or L p at least one of the norms.

6. The computer-implemented method of claim 1, wherein, 7. A computer-implemented method for using a machine learning model trained with one of claims 1-6 for a classification task, comprising: feeding an input to the trained machine learning model; estimating a prediction uncertainty based on a task-related prediction for the input; and determining how to handle the input or the task-related prediction for the input based on a comparison of the prediction uncertainty to a threshold.

8. The computer-implemented method of claim 7, estimating the prediction uncertainty based on the task-related prediction for the input, further comprising: taking a maximum logit output of logit outputs corresponding to each task-related class for the input as the prediction uncertainty; or taking a maximum softmax layer output of softmax layer outputs corresponding to each task-related class for the input as the prediction uncertainty; or taking an entropy of all softmax layer outputs corresponding to each task-related class for the input as the prediction uncertainty. the input to the trained machine learning model is captured by at least one of a sensor, a camera, or a scanner.

10. The computer-implemented method of claim 7, determining how to handle the input based on the comparison of the prediction uncertainty to the threshold, further comprising: selecting the input to be labeled for use as training data for further learning based on the prediction uncertainty being higher than the threshold.

9. The computer-implemented method of claim 7, wherein, 11. The computer-implemented method of claim 7, determining how to handle the input based on the comparison of the prediction uncertainty to the threshold, further comprising: determining the input as an anomaly based on the prediction uncertainty being higher than the threshold.

12. The computer-implemented method of claim 7, determining how to handle the task-related prediction based on the comparison of the prediction uncertainty to the threshold, further comprising: determining to not accept the task-related prediction based on the prediction uncertainty being higher than the threshold.

13. A computer system, comprising: one or more processors; and a memory coupled to the one or more processors and having stored therein instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1-12. ​ ​ ​ one or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform the operations of the method of one of claims 1-12.

14. One or more computer-readable storage media storing computer-executable instructions that, when executed, cause one or more processors to perform the operations of the method of one of claims 1-12.

15. A computer program product comprising computer-executable instructions that, when executed, cause one or more processors to perform the operations of the method of one of claims 1-12.

Images