Learning reliability model based on distance

Abstract

To provide a method of and a system for quantizing reliability of a trained model.SOLUTION: A system (100) for training both of a classification model (210) and a reliability model (220) executes, on a data processing hardware (104), operations including steps of: receiving a training data set (110) including a plurality of training data subsets (112); selecting a support set of training examples and a query set of training examples; updating parameters of the classification model based on a class distance scale and a correct answer distance associated with query encoding generated with respect to each of training examples in the query set for the training examples; sampling a new query encoding for each training example discriminated as erroneous classification; and updating parameters of the reliability model based on the new query encoding.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a learning reliability model based on distance.

Background Art

[0002] Background A machine learning model receives an input and generates an output, e.g., a predicted output, based on the received input. The machine learning model is trained on data. However, quantifying the trust of a model trained for prediction (also known as confidence calibration) is a challenge. In the case of a "well-calibrated" model, predictions with higher confidence should be more likely to be accurate. However, the prediction probabilities (softmax outputs) obtained at the end of the pipeline, which are misinterpreted as model confidence, are poorly calibrated for the decision quality of the model, and the confidence values tend to be high even when the classification is inaccurate. The confidence values tend to be high even when the classification is inaccurate.

Summary of the Invention

Means for Solving the Problems

[0003] Summary One aspect of the present disclosure provides a method for jointly training a classification model and a confidence model. The method includes receiving, in data processing hardware, a training data set including a plurality of training data subsets. Each training data subset is associated with a different respective class and has a plurality of corresponding training examples belonging to each class. From two or more training data subsets within the training data set, the method also includes the data processing hardware selecting a support set of training examples and a query set of training examples. The support set of training examples includes K training examples sampled from each of two or more training data subsets, and the query set of training examples includes training examples sampled from each of two or more training data subsets that are not included in the support set of training examples. For each respective class associated with two or more training data subsets, the method further includes the data processing hardware determining a centroid value by averaging K support encodings associated with the K training examples within the support set of training examples belonging to each class, using the classification model. For each training example in the query set of training examples, the method also includes the data processing hardware generating a query encoding using the classification model, the data processing hardware determining a class distance metric representing each distance between the query encoding and the centroid value determined for each respective class, the data processing hardware determining a correct distance between the query encoding and the correct label associated with the corresponding training example within the query set of training examples, and the data processing hardware updating the parameters of the classification model based on the class distance metric and the correct distance.For each training example in the query set of training examples identified as misclassified, the method further includes the data processing hardware generating, using a confidence model, a standard deviation value of the query encoding generated by the classification model for the corresponding misclassified training example; the data processing hardware sampling a new query encoding for the corresponding misclassified training example using the standard deviation value and the query encoding; and the data processing hardware updating the parameters of the confidence model based on the new query encoding.

[0004] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the correct label includes a correct centroid value within a distance-based representation space. In some examples, updating the parameters of the classification model based on the class distance metric and the correct distance trains the classification model to minimize the within-class distance and maximize the between-class distance.

[0005] In some embodiments, the confidence model is trained to maximize the standard deviation value for larger correct distances and sample new query encodings closer to each correct centroid value. In some examples, any training example within the query set of training examples identified as misclassified includes any training example within the query set of training examples that includes a correct distance that does not meet the distance threshold. In some embodiments, the confidence model is not trained on training examples within the query set of training examples that include a correct distance that meets the distance threshold. In some examples, updating the parameters of the confidence model includes updating the parameters of the confidence model to encourage the confidence model to output a larger standard deviation value for query encodings associated with a larger class distance metric. In some embodiments, the training examples include image data. The classification model may include a deep neural network (DNN). In some examples, the confidence model includes a deep neural network (DNN).

[0006] Another aspect of the present disclosure provides a system for jointly training a classification model and a confidence model. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving a training data set that includes a plurality of training data subsets. The operations also include selecting a support set of training examples and a query set of training examples from two or more training data subsets within the training data set. The support set of training examples includes K training examples sampled from each of the two or more training data subsets, and the query set of training examples includes training examples sampled from each of the two or more training data subsets that are not included in the support set of training examples. The operations further include, for each class associated with the two or more training data subsets, the data processing hardware determining a centroid value by averaging K support encodings associated with the K training examples within the support set of training examples that belong to each class, using the classification model. For each training example in the query set of training examples, the operations also include generating a query encoding using the classification model, determining a class distance metric representing the respective distance between the query encoding and the centroid value determined for each class, determining a correct distance between the query encoding and the correct label associated with the corresponding training example within the query set of training examples, and updating the parameters of the classification model based on the class distance metric and the correct distance.The above operation further includes, for each training example in the query set of training examples identified as misclassified, using a confidence model to generate a standard deviation value of the query encoding generated by the classification model for the corresponding misclassified training example, sampling a new query encoding for the corresponding misclassified training example using the standard deviation value and the query encoding, and updating the parameters of the confidence model based on the new query encoding.

[0007] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the correct label includes a correct centroid value in a distance-based representation space. In some examples, updating the parameters of the classification model based on the class distance metric and the correct distance trains the classification model to minimize the within-class distance and maximize the between-class distance.

[0008] In some implementations, the confidence model is trained to maximize the standard deviation value for larger correct distances and sample new query encodings as close as possible to the correct centroid value. In some examples, any training example in the query set of training examples identified as misclassified includes any training example in the query set of training examples that includes a correct distance that does not meet the distance threshold. In some implementations, the confidence model is not trained on training examples in the query set of training examples that include a correct distance that meets the distance threshold. In some examples, updating the parameters of the confidence model includes updating the parameters of the confidence model to prompt the confidence model to output a larger standard deviation value for query encodings associated with a larger class distance metric. In some implementations, the training examples include image data. The classification model may include a deep neural network (DNN). In some examples, the confidence model includes a deep neural network (DNN).

[0009] Details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Best Mode for Carrying Out the Invention

[0011] Like reference numerals in the various drawings indicate like elements. Detailed Description Conventional techniques for training a deep neural network (DNN) typically result in a poorly calibrated DNN. Since DNNs are deployed in many important decision-making scenarios, poor calibration can lead to costly misjudgments. To prevent acting based on incorrect judgments made by a DNN, it is desirable for the DNN to output a confidence estimate on the decisions output by the DNN. For this purpose, a system can refrain from acting based on decisions output by the DNN with low confidence, avoid acting based on such decisions, and / or consult a human expert, so that harmful results can be avoided if these low-confidence decisions are trusted and acted upon. Unfortunately, accurate confidence estimation is difficult for DNNs, especially for poorly calibrated DNNs.

[0012] Examples implemented herein are directed to a training framework that implements learning from distance-based errors (DBLE) to yield a well-calibrated neural network. In DBLE, a system has a classification model (also referred to as a "prediction model") It trains the classification model and the confidence model in parallel. Training the classification model using DBLE learns a distance-based representation space, whereby the distance-based representation space defines the L2 distance of a test sample to the center of the correct class of the test sample in order to calibrate the performance of the classification model for a given test sample. Thus, unlike vanilla training (conventional training for optimizing the maximum likelihood), training the classification model with DBLE has features for use as the best confidence measure for calibrating its decision quality. However, since calculating this distance for a test sample requires the label for the center of the correct class, it cannot be directly obtained during inference. Thus, training the confidence model with DBLE is configured to estimate this distance as the confidence score during inference. To train the confidence model, DBLE utilizes misclassified training samples (learning from training errors) during the training of the classification model.

[0013] Referring to FIG. 1, in some implementations, system 100 includes a computing environment 130 that includes a resource 102. The resource 102 includes data processing hardware 104 (e.g., a server or CPU) and / or remote memory hardware 106 that stores instructions, which, when executed on the data processing hardware 104, cause the data processing hardware 104 to perform operations. A distance-based learning from errors (DBLE) architecture 200 that trains a classification model 210 and a confidence model 220 in parallel may be present on the resource 102. In the illustrated example, the DBLE architecture 200 trains the classification model 210 on a training data set 110 that includes a plurality of training data subsets 112, 112a-n, each of which includes a plurality of training examples 114 associated with different respective classes. Each training example 114 includes a corresponding ground truth label indicating the respective class to which the training example 114 belongs. Here, the ground truth label may include a ground truth centroid value 212G in a distance-based representation space. In some examples, the training examples correspond to images or image data.

[0014] As will become apparent, the DBLE architecture 200 is configured to learn a distance-based representation space through the classification model 210 and utilize the distances within this space to yield a well-calibrated classification. The DBLE architecture 200 relies on the fact that the correlation between the position of a test sample in the representation space and the distance of the test sample to the training samples in the same class contains useful information for inducing confidence estimation. That is, the DBLE architecture adapts prototype learning for training and inference such that the distance of a test sample to the correct class center can calibrate the performance of the classification model 210, and is configured to learn a distance-based representation space through classification. As used herein, prototype learning refers to both training and prediction that depends only on the distances of samples in the representation space to their corresponding class centers (also referred to as "prototypes"), thereby minimizing the within-class distance and maximizing the between-class distance so as to optimize the training of the classification model 210 such that related samples are clustered together in the representation space. During inference, since the correct label of the correct class center is unknown, the DBLE architecture 200 trains a separate confidence model 220 in conjunction with the classification model 210, thereby enabling the estimation of the distance of a test sample to its correct class center. Specifically, an embodiment targets training the confidence model only with the training samples misclassified by the classification model 210 during training. The models 210, 220 may each include a deep neural network (DNN).

[0015] Figure 2 provides an exemplary DBLE architecture 200 for training the classification model 210 and the confidence model 220 in parallel and together, and trains the confidence model 220 using the training example 114 identified as misclassified by the classification model 210 By looping, the reliability model 220 enables the distance of a test sample to its correct center in the distance-based representation space to be estimated during inference when the correct center is unknown. In contrast to the vanilla training technique for classification based on a variant of mini-batch gradient descent, the DBLE architecture uses episodic training to train the classification model 210. DBLE creates an episode by randomly sampling training examples 114 from the training dataset 110 to select two sets of training examples, namely, (1) the support set 114S of training examples, and (2) the query set 114Q of training examples. More specifically, DBLE creates each episode by first randomly sampling / selecting N training data subsets 112 from a plurality of data subsets 112. Then, DBLE selects the support set 114S of training samples by sampling K training examples 114Sa-k from each of the N training data subsets 112, and selects the query set 114Q of training examples by sampling training examples 114 not included in the support set 114S of training samples from each of the N training data subsets 112. In some examples, the N training data subsets include two or more training data subsets 112 within the training dataset 110. The N training data subsets can include the entire plurality of training data subsets, but DBLE does not need to use the entire training data subset because it can be difficult to adapt training examples from the support set of training examples to the processor memory in batches when the number of different classes is very large.

[0016] For each of the classes associated with the N training data subsets 112, DBLE determines the centroid value 214 by averaging the K support encodings 212S, 212Sa~k associated with the K training examples 114Sa~k within the support set 114S of training examples belonging to each class, using the classification model 210. That is, for a given class, the classification model 210 receives, as input, each of the K training examples 114 within the support set 114S of training examples, and generates, as output, the corresponding support encoding 212S for each training example in the support set. For that given class, DBLE averages the K support encodings 212S to calculate / determine the respective centroid value 214 for each given class. Thus, DBLE is repeated for the remaining N training subsets 212, and N centroid values 214 are calculated such that each centroid value 214 represents each of the N classes.

[0017] The classification model 210 further generates, for each training example in the query set 114Q of training examples, the respective query encoding 212Q, h iGenerate, and DBLE obtains a class distance measure representing the distance between the query encoding 212Q and the centroid value 214 obtained for each respective class. DBLE also obtains the correct distance between the query encoding 22Q and the correct centroid value 212G associated with the corresponding training example within the query set 114Q of training examples, and updates the parameters of the classification model 210 based on the class distance measure and the correct distance. Specifically, DBLE receives the query encoding 212Q and the centroid values 214, 214a - n obtained for each of the N respective classes, uses the proto-loss 215 of classification to obtain / calculate each class distance measure, receives the correct centroid value 212G, and also obtains / calculates the correct distance between the query encoding 212Q and the correct centroid value 212G. Therefore, the classification model 210 is a function parameterized by the trainable parameter Q of the training example, and uses the loss associated with the negative log-likelihood of the correct centroid value 212G of each training example in the query set 114Q of training examples, assuming the support set 114S of training examples, as represented by the following equation.

[0018]

Number

[0019] Where Se is the support set 114S of training examples, Qe is the query set 114Q of training examples, y i is the correct centroid value 212G, and x i is the query set training example 114Q of training examples input to the classification model, and θ represents the trainable parameters of the classification model 210. The prototype loss 215 of classification is configured to calculate the predicted label distribution of each training example x i within the query set 114Q of training examples based on each class distance measure for each respective class of the N classes using the following equation.

[0020]

Number

[0021] Here, h i is the corresponding query encoding 214Q that represents the corresponding training example x i in the representation space based on distance. Therefore, DBLE uses Equation 2 to calculate the predicted label distribution i for each training example x

[0022] [Number]

[0023] for the loss calculated by Equation 1 using

[0024] [Number]

[0025] By minimizing, the trainable parameters Q of the classification model 210 are updated. Therefore, in the representation space of the query encoding 212Q and the centroid values 214 obtained for N classes, the training of the classification model 210 maximizes the distance between classes and minimizes the distance within classes. As a result, training examples belonging to the same class are clustered together, and clusters representing different classes are pushed apart within the representation space.

[0026] While classifying each training example in the query set 114Q of training examples using the classification model 210, the DBLE 200 identifies any training example misclassified by the classification model 210. The DBLE 200 may identify a training example as a misrecognition if the classification predicted by the classification model 210 does not match the corresponding correct label 212G of the training example. In some examples, the DBLE 200 identifies a training example as a misrecognition if the respective correct distance between the query encoding 212Q and the correct centroid value 212G associated with the corresponding training example in the query set 114Q of training examples does not satisfy a distance threshold within the representation space. Otherwise, the DBLE 200 identifies any training example in the query set 114Q of training examples that has a respective correct distance between the query encoding 212Q and the correct centroid value 212G, which satisfies the distance threshold (e.g., is less than or equal to the distance threshold), as being correctly classified by the classification model 210.

[0027] In some implementations, the confidence model 220 trains on training examples in the query set 114Q of training examples that are identified as being misclassified by the classification model 210. Generally, correctly classified training examples constitute the majority of the training examples encountered during the training of the classification model. Based on this concept, using all of the training examples 114Q would result in a small / short class distance measure associated with the correctly classified training examples in the query set 114Q of training examples that dominate the training of the confidence model 220, thereby making it more difficult for the confidence model 220 to capture the larger class distance measures associated with the misclassified training examples that constitute a minority of all the training examples 114Q.

[0028] In the exemplary DBLE architecture 200 of FIG. 2, the dashed box around the confidence model 220, as well as the sampling operation 225 and the calibration proto-loss 250 associated with the confidence model 220, indicate that the confidence model 220 is trained in parallel with the classification model 210 using only each training example in the query set 114Q of training examples identified as misclassified. Thus, for each training example in the query set 114Q of training examples identified as misclassified, the DBLE executed on the data processing hardware 104 uses the confidence model 220 to generate a standard deviation value 222, σ, for the query encoding 212Q generated by the classification model 210 for the corresponding misclassified training example, samples a new query encoding 224 for the corresponding misclassified training example using the standard deviation value 222 and the query encoding 212Q, and updates the parameter φ of the confidence model 220 based on the new query encoding 224.

[0029] The confidence model 220 is trained to output a larger standard deviation value 222, σ, for the query encoding 212Q associated with a larger class distance metric. The new query encoding 224, z s To sample, the confidence model 220 uses a sampling operation 225 that samples from an isotropic Gaussian distribution parameterized by the corresponding query encoding 212Q, h s , and the corresponding standard deviation value 222, σ s . The calibration proto-loss 250 is configured to calculate the prototype loss using the predicted label distribution of the newly sampled query encoding 224z s for each misclassified training example x s .

[0030]

Equation

[0031] Therefore, DBLE updates the trainable parameter φ of the confidence model 220 so as to prompt the confidence model 220 to output a larger standard deviation value 222, σ for the query encoding 212Q associated with a larger class distance metric. In particular, by fixing the query encoding 212Q for each training example misclassified in the representation space, when maximizing Equation 3, the new query encoding 224 will be as close as possible to each correct centroid value 212G. Since the misclassified training examples include query encodings 212Q that are farther from the correct centroid value 212G, prompting the confidence model 220 to output a larger corresponding standard deviation value 222 forces the new query encoding 224 to be closer to the correct centroid value 212G. Figure 4 shows the use of the DBLE 200 described in Figure 2 provides an exemplary algorithm 400 for training the classification model and calibration model 210, 220.

[0032] Figures 3A and 3B show exemplary plots 300a, 300b of dots representing training examples from the query set 114Q of training examples in the representation space. The dotted vertical lines in each of the plots 300a, 300b indicate the decision boundaries where the left and right training examples belong to different classes. Further, the dotted circles represent the standard deviation values 222, σ for the corresponding query encodings 212Q, h a -h c and h a 、h b is associated with the misclassified query encoding 222Q corresponding to the misclassified training example 114Q, and h c is associated with the correctly classified query encoding 222Q corresponding to the correctly classified training example 114Q. Plot 300a in Figure 3A shows the misclassified query encodings h a 、h b and the correctly classified query encoding h cshows the short standard deviation values 222 for both. The plot 300b in FIG. 3B shows that after updating the trainable parameter φ of the reliability model 220, the calibration proto-loss samples new query encodings z a from misclassified training examples, and moves them as close as possible to the correct class-associated ground-truth centroid value 212G, resulting in misclassified query encodings h b that show a larger standard deviation value 222 for h a compared to h b that is closer to the correct class.

[0033] Referring again to FIG. 2, during inference, the DBLE 200 executed on the data processing hardware 104 calculates the class centroids 214 for all classes c in the training set by averaging the representations 212S of all corresponding training examples using the following equation.

[0034]

Equation

[0035] where

[0036]

Equation

[0037] is the set of all training examples belonging to class k. Then, when a test sample x t is given, the respective class distance metrics of the corresponding query encodings 212Q to each class centroid 214 are determined. The prediction of the label of x t is based on the class distance metric such that x t is assigned to the class with the closest center in the representation space. Thus, if the query encoding is too far from its correct center 214G, it may be misclassified. The ground-truth centroid value 212G is the test sample x tis unknown, i.e., since the labels are not available, DBLE estimates each class distance measure using the trained confidence model 220 to assist the classification model 210 in predicting the labels. That is, the classification model 210 uses the corresponding query encoding h t 212Q to predict the labels of all test samples x t . Then, the confidence model 220 outputs the standard deviation value σ t 222 of the query encoding h t , and the sampling operation 225 samples a new query encoding 224. Then, DBLE averages the predicted label distribution as a confidence estimate using the following equation.

[0038]

Number

[0039] Here, U is the total number of new query encodings z t 224, and

[0040]

Number

[0041] is used as the confidence score to calibrate the prediction

[0042]

Number

[0043] of the classification model 210. Therefore, due to the greater estimated variability from the confidence model, DBLE adds more randomness to the representative sampling for test examples that are further away from the correct class center (more likely to be misclassified).

[0044] FIG. 5 is a flowchart of an exemplary configuration of operations for a method 500 of training classification model 210 in parallel with reliability model 220. Method 500 can be executed on data processing hardware 104 of FIG. 1 based on instructions stored in memory hardware 106 of FIG. 1. In operation 502, method 500 includes receiving, at data processing hardware 104, a training data set 110 that includes a plurality of training data subsets 112. Each training data subset 112 is associated with a different respective class and has a plurality of corresponding training examples 114 belonging to each class.

[0045] In operation 504, for two or more training data subsets 112 within training data set 110, method 500 includes selecting, by data processing hardware 104, a support set 114S of training examples and a query set 114Q of training examples. The support set 114S of training examples includes K training examples 114 sampled from each of two or more training data subsets 112. The query set 114Q of training examples includes training examples 114 sampled from each of two or more training data subsets 112 that are not included in the support set 114S of training examples.

[0046] Method 500 includes, in operation 506, obtaining a centroid value 214 by the data processing hardware 104 averaging K support encodings 215 associated with K training examples 114 within a support set 114S of training examples belonging to respective classes, using the classification model 210. In operation 508, for each training example within a query set 114Q of training examples, method 500 includes the data processing hardware 104 generating a query encoding 212Q using the classification model 210, the data processing hardware 104 obtaining a class distance measure representing the respective distance between the query encoding 212Q and the centroid value 214 determined for each class, the data processing hardware 104 obtaining a correct distance between the query encoding 212Q and the correct label 214G associated with the corresponding training example 114 within the query set 114Q of training examples, and the data processing hardware 104 updating the parameters of the classification model 210 based on the class distance measure and also the correct distance.

[0047] In operation 510, for each training example 114 within a query set 114Q of training examples identified as misclassified, method 500 includes the data processing hardware 104 generating a standard deviation value 222 of the query encoding 212Q generated by the classification model 210 for the corresponding misclassified training example, using the reliability model 220, the data processing hardware 104 sampling a new query encoding 224 for the corresponding misclassified training example, using the standard deviation value 222 and the query encoding 212Q, and the data processing hardware 104 updating the parameters of the reliability model 220 based on the new query encoding 224.

[0048] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application", an "app", or a "program". Examples of applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0049] A non-transitory memory may be a physical device used to temporarily or permanently store a program (e.g., a sequence of instructions) or data (e.g., program state information) for use by a computing device. The non-transitory memory may be a volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as a boot program). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks or tapes.

[0050] FIG. 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as a laptop, desktop, workstation, personal digital assistant, server, blade server, mainframe, and other appropriate computers. The components shown in this specification, their connections and relationships, and their functions are exemplary only and are not intended to limit the examples of the invention described and / or claimed in this document.

[0051] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to memory 620 and high-speed expansion port 650, and a low-speed interface / controller 660 connected to low-speed bus 670 and storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 may be interconnected using various buses and may be implemented on a common motherboard or in other manners as required. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or storage device 630 in order to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 680 coupled to the high-speed interface 640. In other embodiments, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and multiple types of memory. Also, multiple computing devices 600 may be connected, and each device may provide a portion of the necessary operations (e.g., as a server bank, as a group of blade servers, or as a multiprocessor system).

[0052] Memory 620 stores information non-temporarily within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 620 can be a physical device used to store a program (e.g., a sequence of instructions) or data (e.g., program state information) temporarily or persistently for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as a boot program). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks or tapes.

[0053] Storage device 630 can provide mass storage for computing device 600. In some implementations, storage device 630 is a computer-readable medium. In various different implementations, storage device 630 can be an array of devices including a floppy (registered trademark) disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In a further implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods such as the above-described methods. The information carrier is a computer-readable medium or a machine-readable medium such as memory 620, storage device 630, or memory on processor 610.

[0054] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, and the low-speed controller 660 manages low-bandwidth-intensive operations. Such role assignments are merely exemplary. In some implementations, the high-speed controller 640 is coupled to a high-speed expansion port 650 that can accept a memory 620, a display 680 (e.g., through a graphics processor or accelerator), and various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, Wi-Fi®), can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, via a network adapter. The low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, Wi-Fi®), can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, via a network adapter.

[0055] As shown in the figure, the computing device 600 can be implemented in several different forms. For example, it can be implemented as a standard server 600a, or multiple times as a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0056] Various implementations of the systems and techniques described herein may be implemented in digital electronics and / or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0057] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0058] The processes and logical flows described in this specification can be executed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs so as to perform functions by operating on input data and generating output. The processes and logical flows can also be executed by dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). Processors suitable for the execution of a computer program may include, by way of example, both general and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. In general, a processor will receive instructions and data from a read only memory or a random access memory or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer will also include, or be operatively coupled to receive data from or transfer data to, one or more mass storage devices for storing data, such as, for example, magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices, such as, EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0059] To provide an interaction with a user, one or more aspects of the present disclosure include, for example, a display device for presenting information to a user, such as a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen, and optionally a keyboard and a pointing device, such as a mouse or a trackball, by which a user can provide an input to a computer. It can be implemented on a computer having the same. Other types of devices can also be used to provide an interaction with a user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input received from the user can be received in any form, including acoustic input, voice input, or tactile input. Additionally, the computer can interact with the user by sending a document to the device used by the user and receiving a document from the device used by the user, such as by sending a web page to a web browser on the user's client device in response to a request received from the web browser on the user's client device.

[0060] Some implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the claims.

Claims

1. A method executed by a computer, which is executed by data processing hardware and causes the data processing hardware to execute an operation, the operation including: obtaining a set of training samples, each training sample in the set of training samples including a correct label, the method further comprising: training a classification model using the set of training samples; and during training the classification model, determining, based on the correct label of each training sample, that the classification model misclassified each training sample in the set of training samples; and in response to determining that the classification model misclassified each training sample, training a confidence model using each training sample, wherein the set of training samples includes a plurality of training data subsets, each training data subset being associated with a different respective class and having a plurality of corresponding training examples belonging to the respective class, wherein the confidence model is a method executed by a computer that predicts, as a confidence score for each prediction of the classification model, the distance from each training sample to the center of the correct class of the training sample.

2. Training the classification model includes: selecting a first subset of training samples from the set of training samples; and selecting a second subset of training samples from the set of training samples, the second subset of training samples including training samples not included in the first subset of training samples, and training the classification model further includes: training the classification model using the first subset of training samples and the second subset of training samples. The method according to claim 1.

3. Determining that the classification model misclassified each training sample includes determining that the correct distance between the prediction of the classification model and the correct label does not satisfy a threshold distance. The method according to claim 1 or 2.

4. The method according to any one of claims 1 to 3, wherein the correct label includes a correct centroid value in a representation space based on distance. **Claim 5** A method executed by a computer that is executed by data processing hardware and causes the data processing hardware to execute operations, the operations including: obtaining a set of training samples, each training sample of the set of training samples including a correct label, the method further comprising: training a classification model using the set of training samples; while training the classification model, determining, based on the correct label of each training sample, that the classification model has misclassified each training sample of the set of training samples; in response to determining that the classification model has misclassified each training sample, training a confidence model using each training sample, the set of training samples includes a plurality of training data subsets, each training data subset is associated with a different respective class and has a plurality of corresponding training examples belonging to each respective class, The confidence model is a method executed by a computer that is trained only on training samples misclassified by the classification model during the training of the classification model. **Claim 6** A method executed by a computer that is executed by data processing hardware and causes the data processing hardware to execute operations, the operations including: obtaining a set of training samples, each training sample of the set of training samples including a correct label, the method further comprising: training a classification model using the set of training samples; while training the classification model, determining, based on the correct label of each training sample, that the classification model has misclassified each training sample of the set of training samples; In response to determining that the classification model misclassifies each of the training samples, training a confidence model using each of the training samples, The set of training samples includes a plurality of training data subsets, each training data subset is associated with a different respective class and has a plurality of corresponding training examples belonging to each respective class, The confidence model is a computer-implemented method that predicts a confidence score for each prediction of the classification model. **Claim 7** The method according to any one of claims 1 to 6, wherein the set of training samples includes image data. **Claim 8** The method according to any one of claims 1 to 7, wherein the classification model includes a deep neural network (DNN). **Claim 9** The method according to any one of claims 1 to 8, wherein the confidence model includes a deep neural network (DNN). **Claim 10** Data processing hardware, A system comprising memory hardware communicating with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to execute the method according to any one of claims 1 to 9.

Images