Image classification model calibration

Abstract

Classification model calibration. A computer-implemented method of calibrating a trained classification model is described. The trained classification model is trained to classify input samples according to a plurality of classes and provide associated prediction probabilities, and includes a plurality of hidden layers and at least one activation layer. The method includes accessing the trained classification model and accessing a plurality of validation samples, each validation sample having a ground truth label indicating a ground truth class. The method further includes applying the trained classification model to the plurality of validation samples, for each validation sample, obtaining an output logit vector from a layer of the trained classification model prior to the last activation layer, and training a calibration module. The calibration module is trained to adjust prediction probabilities derived from the output logit vector. The calibration module includes at least one of a fine-tuning submodule and a hierarchical submodule. The method includes appending the trained calibration module to the trained classification model to obtain a calibrated classification model.

Description

Technical Field

[0001] The present invention relates to a method for calibrating a training classification model, a system for calibrating a training classification model, a method for classifying an input image, a system for classifying an input image, and a computer-readable medium. Background Technology

[0002] Machine learning (“trained”) models are widely used in many real-world applications such as autonomous driving, robotics, manufacturing, and building control. For example, machine-learnable models, such as neural networks, can be trained to infer the state of physical systems, such as autonomous vehicles or robots, or the environment of systems, such as roads on which vehicles travel or workspaces of robots, based on sensor data acquired by one or more sensors. Once the state is inferred, the physical system can be controlled, for example, using one or more actuators, or its operation can be monitored.

[0003] Typically, as is known in machine learning, machine-learnable models, such as neural networks, can be trained on training data to provide outputs such as predictions or decisions. An example of a decision is classification. By training a model, it can provide such predictions or decisions without being explicitly programmed to do so. For example, to classify images, a model can be trained on a training dataset that includes images whose classification the model already knows. Thus, a trained model can be trained to determine the classification of an input—such as an input image, sound, or text. For example, the model can be trained as a multi-class classifier, by which an image can be classified as, for example, "cat," "dog," or "tree." Another example is that a trained model can be trained as a binary classifier (e.g., classifying an input image into only one of two complementary categories, "normal (OK)" or "abnormal (NOK)").

[0004] Generally, such models can include deep neural networks (DNNs), which are defined as having multiple layers between the input layer and the output layer.

[0005] Classification using machine learning models (such as trained neural networks) can be used in countless applications, such as optical quality inspection in manufacturing processes or hazard detection in autonomous vehicles.

[0006] In recent years, machine learning models have been widely and successfully used for classification tasks. These machine learning models—also called classifiers or classifier models—can be configured to classify images, sounds (e.g., speech), and many other forms of information. Typically, training a classifier model involves providing multiple training samples, which include a provided ground truth classification. For example, training samples might be pictures of cats, and the ground truth classification would be provided as "cat". The model uses a large number of such training samples to train multiple model parameters. However, the number of model parameters to be trained often far exceeds the number of training samples, leading to overfitting to negative log-likelihood (NLL) loss. This overfitting results in overconfident predictions, whose output predictions have high confidence levels that do not represent their true classification ability. In other words, the model appears to be better than it actually was trained. Overconfident predictions cannot be relied upon because they do not reject risky predictions, meaning that classification models cannot be reliably used in applications such as sensor fusion, where the importance of different sensors is weighted when making fused predictions from them. Therefore, the topic of prediction confidence calibration is the focus of a considerable amount of research.

[0007] Perfect calibration of a training classification model means that the distribution of true labels for some recurring events matches the distribution of predictions made by the model. Calibration focuses on the model's performance over a range of predictions, rather than any single prediction. For example, a well-calibrated classifier that predicts "dog" with an 80% probability should be correct 80% of the time. This is typically evaluated by applying multiple validation and / or test samples—in which the underlying ground truth classifications are known—determining the probability of a particular class (e.g., "dog," "tree," etc.), and determining whether the model correctly predicts the class with the corresponding frequency. For this to work, there should be a sufficient number of validation samples such that the model outputs the same distribution of predictions, allowing the distribution of the underlying ground truth labels to be approximated well by frequency estimates over a finite set of samples. For models that output a continuous distribution of predictions, sample-based calibration evaluation requires binning to quantify the output.

[0008] Several model optimizations are employed during model training to improve calibration. For example, modifying the training loss to incorporate calibration error, estimating unsampled model uncertainty, data augmentation techniques, and ensemble distribution distillation can all be implemented during model training to improve calibration. However, these methods are resource-intensive and highly complex, incurring significant overhead and introducing substantial latency. In many cases, it is desirable to alternatively calibrate the model after it has been trained (e.g., post-hoc) so that retraining or modification of the model is unnecessary.

[0009] Several methods exist for post-hoc calibration (e.g., after model training), including fine-tuning (also known as scaling) and grading. Calibration performance is highly dependent on the grading scheme used to quantify the predicted probabilities or logit odds. Typically, grading schemes involving determining bin edges (also known as bin boundaries) and bin representation are performed after the model has been trained. The most common criteria for defining bin edges are equidistant grading and equal-quality grading. In equidistant grading, each bin has an equal width, and the intervals are evenly divided into multiple bins. In equal-quality grading, bin edges are defined such that each bin contains the same number of samples (this is also known as equal-frequency grading).

[0010] However, grading introduces information loss. As an example, using a small number of bins leads to coarse quantization, which loses a significant amount of information—all samples within a bin are represented by a single value, and approximating samples with a single value when they have significantly different values ​​results in a loss of information about heterogeneous samples. Using more refined bins (e.g., a large number of bins) is generally not feasible because the validation set has a finite number of samples. Two commonly used grading schemes—equidistant grading and equal-quality grading—ignore the fact that (1) the label information conveyed by log odds is of primary interest, and (2) log odds itself also encodes class-independent data information.

[0011] Fine-tuning is a post-processing solution based on a simple fine-tuning model, which adjusts the original log-odds and loss function to optimize the fine-tuned model parameters on the validation set. When fine-tuning the log-odds, the loss function is used to penalize misclassified samples and encourage correct classification, aiming to reduce overconfidence. Negative log-likelihood (NLL) and Brier score are commonly used loss functions. In such loss functions, misclassified samples—also known as members of the negative class—are the driving force for penalizing overconfident predictions, while correctly classified samples belonging to the positive class encourage more confident predictions. For any properly trained classification model, the positive and negative classes are naturally imbalanced. However, both NLL loss and Brier score loss simply treat the minority of negative classes and the majority of positive classes equally. Therefore, these methods fail to best utilize the relatively small number of misclassified samples.

[0012] The paper "Verified Uncertainty Calibration" by Kumar et al. (arXiv preprint arXiv: 1909.10155, 2019) describes how calibration errors in continuous methods are typically underestimated and proposes a method for post-calibration involving both scaling and grading. This method involves first fitting a parameter scaling and then grading the output of the scaling function values ​​to ensure calibration. The paper describes how to select a uniform quality grading scheme above equal-width gradations to enable estimation of calibration errors.

[0013] The paper "Calibrating Deep Neural Networks using Focal Loss" by Mukhoti et al. (arXiv preprint arXiv: 2002.09437, 2020) describes training a model that incorporates a focal loss to achieve a more naturally calibrated training model. The paper describes how to use the focal loss during training to direct the model's attention toward samples that are currently predicting a low probability for the correct class. Summary of the Invention

[0014] Ideally, a training classification model should be calibrated in a sample-efficient manner without requiring retraining or tuning. Validation sets are often finite in size. Validation samples are typically created by capturing data (such as photographs or images) and having people manually classify the objects in each image or photograph. In many cases, generating large-scale validation sets is time-consuming and often impractical. Therefore, calibrating a training classification model in a sample-efficient manner reduces overconfident predictions and allows for classification-based decisions based on accurate indications of prediction uncertainty.

[0015] According to a first aspect of the invention, as defined in claims 1 and 10, a computer-implemented method and a corresponding calibration system are described for calibrating a trained classification model by training a calibration module that can be attached to the trained classification model. According to an aspect of the invention, as defined in claims 11 and 14, a computer-implemented method and a corresponding classification system are described for classifying input samples using a calibrated classification model, the calibrated classification model including a calibration module attached to the trained classification model. According to another aspect of the invention, as defined in claim 15, a computer-readable medium is described.

[0016] The various aspects involve calibrating a trained classification model and using the calibrated classification model, which includes a training calibration module attached to the trained classification model. The trained classification model can be trained to classify input samples according to multiple classes and provide associated predicted probabilities, and may include multiple hidden layers and at least one activation layer. The trained classification model can be calibrated by training the calibration module without retraining the training classification model or changing its parameters. The trained classification model can be applied to multiple validation samples, each with a ground truth label indicating the ground truth class of that sample. For example, if the validation sample is an image of a cat, the ground truth class would be "cat". For each validation sample, an output log-odds vector can be obtained from the layer of the trained classification model before the last activation layer. The output log-odds vectors of the multiple validation samples can then be used to train the calibration module. The calibration module may include a fine-tuning submodule, a grading submodule, or both, for adjusting the predicted probabilities. The calibration module can be trained to calibrate or adjust the predicted probabilities by determining the adjusted predicted probabilities based on the output log-odds vector. Once trained, the calibration module can be attached to the trained classification module to obtain a calibrated classification model.

[0017] The calibration module can be trained using the output log-odds vector obtained from the training classification model, potentially calibrating predicted probabilities without retraining or adjusting the model parameters of the training classification model. When the calibration module accepts one or more output log-odds vectors as input, it can be trained in the same way (e.g., using the same method) for many classification models without modification and without knowing how the classification model was trained.

[0018] Using a calibrated classification model enables the classification of samples with accurate predicted probabilities. For example, in autonomous vehicles and automated optical inspections, classifying samples with accurate predicted probabilities is crucial when the system makes autonomous decisions based on classification. Sensor fusion applications also rely on predicted probabilities when determining how to bring together inputs from multiple remote sensors to form a single model.

[0019] Optionally, the calibration module includes a grading submodule for adjusting predicted probabilities by grading the output log-odds vector. The grading submodule includes a grading scheme defining bin boundaries and bin representatives. Bin representatives can be used as adjusted predicted probabilities. Training the calibration module may include determining the grading scheme. For each validation sample, the grading scheme can be determined by casting the output log-odds vector into a binary classification set, each binary classification including a binary label and an associated scalar log-odds. Ideally, bin boundaries should be defined in a way that maximizes the mutual information between the binary labels and the resulting quantized scalar log-odds. An equivalent problem can be defined—that is, minimizing the grading loss function parameterized by the bin boundaries and the set of auxiliary variables is equivalent to maximizing the mutual information. Therefore, bin boundaries can be determined by minimizing the grading loss function. After determining the bin boundaries, the scalar log-odds of the validation samples can be quantified by grading the scalar log-odds according to the determined bin boundaries. Then, for each bin, a bin representative value can be determined to represent the scalar log-odds within that bin. The bin boundaries and bin representatives can then be stored, allowing the grading scheme to be defined in the calibration module.

[0020] Segmentation inherently leads to information loss about those samples and often results in reduced accuracy after calibration. However, by defining a segmentation scheme based on maximizing mutual information, the preservation of crucial information relating label information to log odds is improved. This improves sample efficiency and accuracy after segmentation and thus enables reliable calibration of the log odds even with a low number of available validation samples, while maintaining classification accuracy.

[0021] Quantizing log-odds based on the hierarchical scheme determined above has additional benefits in the field of remote sensor fusion. For example, if sensors are remotely connected to a centralized node, the log-odds (e.g., prediction confidence or prediction probability) are required at the centralized node to determine how the sensor outputs should be fused. Quantization is a necessary step when transmitting the log-odds from the local sensors to the centralized node using wireless communication. It is important to ensure that the bit budget is met to fulfill the transmission power and latency constraints imposed by the application. This is particularly beneficial in segmentation tasks, which require a large number of bits to encode the multi-class distribution of each pixel. Hierarchical or quantized log-odds based on mutual information as described improves bit coding efficiency.

[0022] Optionally, minimizing the hierarchical loss function involves updating at least one of the bin boundaries and auxiliary variables using at least one of gradient descent and closed-form computation.

[0023] Gradient descent and its closed-form computation allow bin boundaries and auxiliary variables to be updated iteratively and in a simple manner. While certain mathematical criteria must be met (e.g., it may be impossible to directly compute the minimum of the tiered loss function in every case), using closed-form computation to update bin boundaries and / or auxiliary variables is more resource-efficient and faster than using gradient descent. For example, if the minimum of the tiered loss function cannot be directly computed, then the use of gradient descent is therefore useful.

[0024] Optionally, determining the warehouse representative for each warehouse includes: determining the proportion of validation samples whose corresponding scalar log probability falls into the warehouse, the proportion of validation samples having the same underlying truth label, and setting the warehouse representative of the warehouse to the predicted probability corresponding to the determined proportion.

[0025] Using this sample-based approach to determine the bin representative based on bins defined by maximizing mutual information provides a simple and accurate calibration for predictive probabilities.

[0026] Optionally, converting the output log odds vector into a binary classification set includes one of the following: a) calibrating the first k predictions to binary classification using the first k calibrations, where k is an integer; b) calibrating the predictions for each of the multiple classes using class-by-class calibration; and c) calibrating the predictions for each of the multiple classes using a shared class-by-class calibration.

[0027] Depending on the size of the validation sample set, different methods can be used to transform the output log-odds vector. For example, shared class-by-class calibration is particularly beneficial when the number of available validation samples is small. If there exists... K If there are multiple classes, then the validation set used for class-by-class calibration becomes... K Double the size.

[0028] Optionally, the calibration module includes a fine-tuning module, and training the calibration module includes training the fine-tuned model. For each validation sample among multiple samples, the parameters of the fine-tuned model are determined by the following: - Determine the underlying ground truth probability from the output log-odds vector. The underlying ground truth probability is a predicted probability associated with the underlying ground truth class of the validation sample. - The anchor probability is determined from the output log-odds vector; the anchor probability is the highest probability of the incorrect class, and - The difficulty of prediction is determined by subtracting the underlying true probability from the anchored probability.

[0029] The model parameters for the fine-tuned model are then determined by minimizing a fine-tuning loss function averaged over multiple validation samples. The fine-tuning loss function includes a modulation term based on the prediction difficulty of each validation sample. The determined model parameters define the training of the fine-tuned model. The trained fine-tuned model is then stored in the calibration module.

[0030] By modulating the fine-tuning loss function based on prediction difficulty, a sample-efficient fine-tuning model can be determined. The use of anchor probabilities improves the weights of the fine-tuning loss function for negative (misclassified) and positive (correctly classified) samples.

[0031] Optionally, the fine-tuning loss function includes at least one of the negative log-likelihood function, the Brier scoring function, and the hinge loss function.

[0032] These loss functions are well-suited for fine-tuning log odds and can even be used, for example, in post-calibration after training a classification model.

[0033] Optionally, the loss function is fine-tuned based on a parametric model, such as a matrix scaling model and / or a Dirichlet calibration model.

[0034] These parametric models can be trained by minimizing the fine-tuning loss function to determine the model's parameters. For example, both matrix scaling models and Dirichlet calibration models are less complex than Gaussian processes, thus saving resources and reducing power consumption.

[0035] Optionally, the output log-odds vector for each validation sample is fine-tuned based on the trained fine-tuning model before being quantized to determine the grading scheme. Then, determining the bin representative for each bin may include: for each scalar log-odds of falling into the bin, calculating the sigmoid response of the scalar log-odds, determining the average of the calculated sigmoid responses of the scalar log-odds of falling into the bin, and setting the bin representative of the bin to the determined average.

[0036] The use of the sigmoid average improves bin representation determination, even when bins are relatively unfilled. For example, if there are only a few scalar log odds in a particular bin, calculating bin representation based on the sigmoid average reduces estimation errors that might occur when using frequency-based methods to determine bin representation. However, in the average sigmoid method, the fine-tuned log odds are considered to be post-calibrated before grading, thus improving sample efficiency.

[0037] Those skilled in the art will appreciate that two or more of the above-mentioned embodiments, implementations and / or alternative aspects of the invention can be combined in any manner deemed useful.

[0038] Any computer-implemented method or any computer-readable medium may be modified and varied by those skilled in the art based on this description, corresponding to modifications and variations of the described system, and vice versa. Attached Figure Description

[0039] These and other aspects of the invention will become clear from the embodiments described below, and will be further illustrated with reference to the embodiments described below. In the accompanying drawings, Figure 1 A system for training a calibration module for training a classification model is shown according to an embodiment of the present invention; Figure 2 A system for classifying input instances according to an embodiment of the present invention is shown; Figure 3 A detailed example of training a calibration module using a fine-tuning method according to an embodiment of the present invention is shown; Figure 4 A detailed example of training a calibration module using a hierarchical optimization scheme according to an embodiment of the present invention is shown; Figure 5a A computer-implemented method for training a calibration module according to an embodiment of the present invention is shown; Figure 5b A computer-implemented method for training a calibration module using fine-tuning, according to an embodiment of the present invention, is shown. Figure 5c A computer implementation method for training a calibration module using a hierarchical optimization scheme according to an embodiment of the present invention is shown; Figure 6 A computer-implemented method for classifying input instances according to an embodiment of the present invention is shown; and Figure 7 A computer-readable medium containing data is shown.

[0040] It should be noted that items with the same reference numerals in different figures have the same structural features and the same function, or the same signal. Since the function and / or structure of such items have already been explained, there is no need to repeat this explanation in the specific implementation. Detailed Implementation

[0041] The following relates to a calibration system and a corresponding computer-implemented method for calibrating a trained classification model, and a classification system and a corresponding computer-implemented invention for classifying input instances. The calibration system and classification system are respectively referenced... Figure 1 and Figure 2 Describe and refer to Figure 3 Figure 5 details the computer-implemented calibration method, and refers to... Figure 6 Describe the classification method implemented by a computer.

[0042] Figure 1A calibration system 100 for calibrating a trained classification model is illustrated according to an embodiment of the present invention. Calibrating the trained classification model may include a training calibration module. The calibration module can be trained such that it can be attached to the trained classification model. The trained classification model may include a deep neural network (DNN) having multiple layers between an input layer and an output layer. Samples (such as validation samples) are input to the input layer, and the output layer outputs the classification of the samples and their associated predicted probabilities. The DNN may include activation layers that take log odds as input. The activation layers may be, for example, softmax activation layers or sigmoid activation layers. For example, the softmax function or sigmoid function can be used to transform the log odds—which can be any real number—into a probability distribution.

[0043] The calibration system 100 may include a data interface 120 and a processor subsystem 140, the processor subsystem 140 being configured to communicate internally via data communication 121. The data interface 120 enables access to data 040 defining the calibration module and model data 050 defining the training classification model. Data 040 may include at least a parameter set for fine-tuning the model and / or a parameter set for a grading scheme. The data interface 120 may further enable access to a set of validation samples 030, each sample including input instances such as images and ground truth classifications. Model data 050 may be used in conjunction with model data 030 for purposes such as... Figure 2 The classification system 200 classifies the input samples according to the method described in this paper.

[0044] The processor subsystem 140 can be configured to access data 030, 040, and 050 using the data interface 120 during operation of the calibration system 100. For example, as Figure 1 As shown, data interface 120 can provide access 122 to an external data storage device 021, which may include the data 030, 040, and 050. Alternatively, data 030, 040, and 050 can be accessed from an internal data storage device that is part of the calibration system 100. Alternatively, data 030, 040, and 050 can be received from another entity via a network. Generally, data interface 120 can take various forms, such as a network interface to a local area network or a wide area network, such as the Internet, a storage interface to an internal or external data storage device, etc. Data storage device 021 can take any known and suitable form.

[0045] The processor subsystem 140 can obtain a set of validation samples 030. Validation samples can be, for example, images, documents including text, audio files, sensor outputs, etc. Each validation sample further includes a base ground truth classification. In other words, the classification of each validation sample is known. In this figure, the validation samples are accessed via data interface 120, but this is not mandatory; for example, the validation samples can be obtained from the sensors during the training of the calibration module. In such cases, the base ground truth classification for each sample can also be input.

[0046] The processor subsystem 140 can train the calibration module based on the validation sample set 030. The calibration module can be configured to receive an output log-odds vector as input, which includes the layers of the trained classification model from each validation sample. K Each corresponding to each of the classes K The log-odds ratio is a vector of values. The output log-odds vector can be the output of a layer in the trained classification model before the last activation layer.

[0047] The calibration module may include a fine-tuning submodule, a grading submodule, or both. The fine-tuning submodule can be configured to optimize the model parameters of the fine-tuned model by minimizing a fine-tuning loss function averaged over multiple validation samples. For each validation sample, the prediction difficulty can be determined by subtracting the probability associated with the sample's underlying ground truth class from the probability associated with the incorrect class with the highest probability. The probability associated with the incorrect class with the highest probability can be referred to as the anchor probability. For example, a classifier can be configured to classify a sample into one of the classes "cat," "tree," and "fish." If a sample has the underlying ground truth class "cat," then "tree" and "fish" would be incorrect classes. In this example, the classifier determines that the probability of the sample being in the "tree" class is 0.35, and the probability of the sample being in the "fish" class is 0.15. Among these incorrect classes, the most likely class is therefore "tree," because the probability of "tree" is higher than that of "fish." Thus, the probability corresponding to "tree"—0.35—would be the anchor probability.

[0048] Fine-tuning the loss function can include a modulation term based on the difficulty of prediction. This will refer to... Figure 3 To describe in more detail.

[0049] The grading module can be configured to determine a grading scheme that quantizes the output log-odds vector. The grading scheme defines bin boundaries, also known as bin edges, and bin representatives. The output log-odds vector is quantized into the bin boundaries, and the bin representatives are values ​​assigned to each bin. The grading scheme can be optimized by finding a quantization function that maximizes the mutual information between the label information and the log-odds. This will refer to... Figure 4 Describe in detail.

[0050] As an optional component, the calibration system 100 may include a sensor interface (not shown) for obtaining sensor data from one or more sensors to be used as input to the classification model. Based on this sensor data, sensor outputs, such as images from a camera, can be determined to be used as validation samples in the training calibration module. For example, measurements may include or be based on readings from one or more sensors. See also Figure 2 Describe the sensor interface.

[0051] Reference Figure 3 and Figure 4 The various details and aspects of the operation of the calibration system 100 are further clarified, including its optional aspects.

[0052] Generally, the calibration system 100 may be embodied in or within a single device or apparatus, such as a workstation, e.g., a laptop- or desktop-based workstation, or a server. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem may be embodied by a single central processing unit (CPU), and also by a system or combination of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory like RAM, or non-volatile memory like flash memory. Alternatively, the system's functional units, such as data interfaces and processor subsystems, may be implemented in a device or apparatus in the form of programmable logic, e.g., as a field-programmable gate array (FPGA) and / or a graphics processing unit (GPU). Generally, each functional unit of the calibration system 100 may be implemented in the form of a circuit. Note that the calibration system 100 may also be implemented in a distributed manner, e.g., involving different devices or apparatuses, such as distributed servers in the form of cloud computing.

[0053] Figure 2 A classification system 200 for classifying input samples is shown. The classification system 200 may use a calibrated classification model, which includes a training classification model having a training calibration module attached thereto.

[0054] The classification system 200 may include a data interface 220 and a processor subsystem 240, the processor subsystem 240 being configured to communicate internally via data communication 221. The data interface 220 may be used to access training classification model data 040 defining the training classification model and calibration module data 050 defining the calibration module. The classification model data 040 may include at least a set of DNN parameters (e.g., weights). The calibration module data 050 may include at least a set of parameters defining at least one of a fine-tuning model and an optimized classification scheme. The calibration module may, for example, be accessed via... Figure 1The calibration system 100 is trained according to the training method described herein. The training classification model can be any training classification model trained by any training system. The use of the calibration module enables post-calibration without any retraining of the classification model or any adjustment of the classification model's parameters. The classification system 200 can also be configured to train models in addition to applying them; for example, the classification system 200 can be used with... Figure 1 The calibration system consists of 100 phases.

[0055] The processor subsystem 240 can be configured to access data 040, 050 using the data interface 220 during operation of the system 200. For example, as Figure 2 As shown, data interface 220 can provide access to external data storage device 022, which may include the data 040, 050. Alternatively, data 040, 050 can be accessed from an internal data storage device that is part of classification system 200. Alternatively, data 040, 050 can be received from another entity via a network. Generally, data interface 220 can take various forms, such as a network interface to a local area network or a wide area network such as the Internet, a storage interface to an internal or external data storage device, etc. Data storage device 022 can take any known and suitable form.

[0056] The processor subsystem 240 can be configured to acquire, during operation of the classification system 200, data such as training classification models, training calibration modules, and sensor data for classifying input samples using the data interface 220. The classification of input samples can be used, for example, in automated optical inspection, hazard detection in autonomous driving, or sensor fusion.

[0057] Processor subsystem 240 can be configured to obtain the output log-odds vector from the layers of a trained classification model prior to the last activated layer. Processor subsystem 240 can then provide the output log-odds vector as input to a training calibration module configured to adjust, in other words, calibrate, the predicted probabilities. The predicted probabilities are typically derived from the output log-odds vector, and calibration of the output log-odds vector adjusts the predicted probabilities derived from it. In other words, by calibrating or adjusting the output log-odds vector, the predicted probabilities derived from it are adjusted or calibrated. Calibrating the output log-odds vector can also be referred to as adjusting the predicted probabilities. The training calibration module can include at least one of training a fine-tuning model and a grading scheme to adjust the output log-odds vector, and thus adjust the predicted probabilities. In some embodiments, the output log-odds vector can be fine-tuned by applying it to training the fine-tuning model to obtain a fine-tuned output log-odds vector. In some embodiments, the output log-odds vector can be graded according to a grading scheme defined based on maximizing mutual information. In some embodiments, the output log-odds vector can be fine-tuned by a fine-tuning model before being graded or quantized by a defined grading scheme. The grading scheme can quantize the output log-odds vector (or fine-tune the log-odds vector) and can output calibrated predicted probabilities.

[0058] The output of the training calibration module can then be used for the classification of input samples. In some embodiments, the classification, including adjusted predicted probabilities, can be output by the calibration module. In some embodiments, the adjusted predicted probabilities can be output by the calibration module, while the classification decision (e.g., the most likely class or the top k classes, etc.) can be output by the classification model.

[0059] In embodiments where the training and calibration module includes a hierarchical submodule, the output of the training and calibration module may include a quantized output log odds. This is particularly advantageous, for example, in sensor fusion, where multiple sensors can be remotely connected to a centralized node, such as the cloud. For fusion decisions, transmitting hard decisions (e.g., the top-ranked prediction class) to the cloud is often insufficient. In these cases, transmitting quantized log odds is preferable. Quantization enables wireless communication to transmit information to the cloud, transmitting only binary bits and not continuous values ​​directly. While this may not be a performance limiting factor in image classification, the bit budget associated with segmentation tasks can be finite. By transmitting quantized log odds, the transmitted bits efficiently encode label information.

[0060] As an optional component, the classification system 200 may include a sensor interface 260 for acquiring sensor data 224 from one or more sensors to be used as input samples for classification. As a non-limiting example, a single sensor 072 is shown in the figure. The sensor interface 260 may communicate internally with the processor subsystem 240 via data communication 223. The sensor data 224 may include images, audio output (e.g., audio files), text, video, sensor outputs such as LiDAR, radar, ultrasonic sensors, etc.

[0061] Sensor interface 260 can be configured for various types of sensor signals, such as the measurement of various physical quantities like temperature, pressure, light intensity, motion, and acceleration, and various signals that can determine such physical quantities—e.g., video signals, radar / LiDAR signals, ultrasonic signals, etc. One or more sensors 072 can be located within a computer-controlled entity, but remote sensing is also possible. Sensor interface 260 can also access sensor data from elsewhere, such as from a data storage device or network location. Sensor interface 260 can have any suitable form, including but not limited to: low-level communication interfaces such as I2C or SPI-based data communication, but also data storage interfaces such as memory interfaces or persistent storage interfaces, or personal area network, local area network, or wide area network interfaces such as Bluetooth, Wi-Fi, or Ethernet interfaces, or fiber optic interfaces. Sensor 072 can be included in classification system 200.

[0062] As an optional component, the classification system 200 may include a display output interface or any other type of output interface (not shown) for outputting one or more defined classifications to a presentation device such as a display. For example, the display output interface may generate display data for the display, thereby causing the display to present one or more predictions in a perceptible manner, such as as a visualization on the screen.

[0063] The defined classification can be used to control a computer-controlled system. For example, the computer-controlled system may include the classification system 200 itself, or the classification system 200 may communicate with an external computer-controlled system to control it, for example, by providing classifications.

[0064] For example, classification system 200 can be coupled to a computer-controlled system for interacting with the environment, such as a robotic system, an automated optical inspection system, or an autonomous driving system. Classification system 200 can be coupled to an actuator interface (not shown). The actuator interface can be used to provide actuator data to actuators, thereby causing the actuators to perform actions in the environment. For example, actuators can be electric, hydraulic, pneumatic, thermal, magnetic, and / or mechanical actuators. Specific but non-limiting examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, stepper motors, etc. Processor subsystem 240 can be configured to determine actuator data based on classification; for example, a classification of a hazard detected in an autonomous vehicle system can lead to specific adaptations in vehicle control (e.g., braking, sudden steering, etc.).

[0065] Generally, the classification system 200 can be embodied in or within a single device or apparatus, such as a workstation, e.g., a laptop- or desktop-based workstation, or a server. For example, the classification system 200 can be an embedded computer, part of a control system such as a computer-controlled entity, for example, built into an autonomous vehicle or robotic agent. The device or apparatus may include one or more microprocessors executing appropriate software. For example, a processor subsystem can be embodied by a single central processing unit (CPU), and also by a system or combination of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory like RAM, or non-volatile memory like flash memory. Alternatively, the system's functional units, such as data interfaces and processor subsystems, can be implemented in a device or apparatus in the form of programmable logic, e.g., as a field-programmable gate array (FPGA) and / or a graphics processing unit (GPU). Generally, each functional unit of the system can be implemented in circuit form. Note that the classification system 200 can also be implemented in a distributed manner, e.g., involving different devices or apparatuses, such as distributed servers in the form of cloud computing.

[0066] Figure 3 A detailed, but non-limiting, example of training a calibration module using a fine-tuning method according to an embodiment of the present invention is shown.

[0067] Multiple validation samples VS 305 can be provided to the training classification model TCM 310. Each validation sample includes an input instance—such as an image, video, document, etc.—and a ground truth classification indicating the true classification of the input instance. The training classification model TCM 310 can be a classification model that has been previously trained using the same in-system training method as calibration system 100 or an external training system. The training classification model TCM 310 can include a deep neural network. The training classification model TCM 310 can be configured to classify the input into... K A multi-class classifier that provides one or more classes and associated predicted probabilities, such as a multi-class multi-label classifier that can classify a single input into multiple class labels.

[0068] After inputting the validation sample VS 305 into the training classification model TCM 310, the output log-odds vector can be obtained from the training classification model TCM 310. More specifically, the output log-odds vector can be the output of the DNN layer before the last activation layer of the training classification model TCM 310. The last activation layer can be, for example, a softmax activation layer or a sigmoid activation layer. For example, if the training classification model TCM 310 is trained to classify samples as... K If one or more of the classes are selected, the log-odds vector will be output. It can be Typically, the predicted probabilities are derived from the output log-odds vector. The calibration module determines the adjusted predicted probabilities by adjusting or calibrating the output log-odds vector.

[0069] The output log-odds vector can be input into the calibration module CalMod 315. The calibration module CalMod 315 may include at least one of the fine-tuning submodule FT 320 and the grading submodule Bin325. Reference Figure 4 The methods associated with the grading submodule Bin 325 are discussed. In some embodiments, the calibration module CalMod 315 includes a fine-tuning submodule FT 320, wherein the grading submodule Bin 325 is optional. This is in Figure 3 The dashed bounding box diagram of the hierarchical submodule Bin 325 is shown.

[0070] Referring now to the fine-tuning submodule FT 320, FT 320 receives the output log-odds vector as input from the trained classification model TCM 310. The fine-tuning submodule FT 320 can include fine-tuning models, such as matrix-scaled models, Dirichlet calibration models, temperature-scaled models, vector-scaled models, or beta models, although this list is not exclusive. The fine-tuning model can be a parametric model that accepts the output log-odds vector. As input. In some examples, a Gaussian process can be used, although a parametric model is preferred to reduce complexity.

[0071] The matrix scaling model and the Dirichlet calibration model are shown below: (Equation 1) (Equation 2) in It is a square array, and It is the bias vector. and Both are trainable parameters for fine-tuning the model.

[0072] In other words, training the fine-tuning submodule FT 320 can include parameters for training such a model. Throughout the specification, example calculations will be shown based on the matrix scaling model shown in Equation 1. However, other known models can be similarly employed.

[0073] To train and fine-tune the model's parameters, the FTF350 fine-tuning loss function can be used. The FTF350 fine-tuning loss function can be based on negative log-likelihood (NLL) loss, Brier score loss, hinge loss, etc. In the following description, for ease of explanation, NLL loss will be used: (Equation 3) in, S Indicate the sample size. Labeled Sample s The basic truth class, for example, ;and This indicates the predicted probability after fine-tuning the log odds. The model parameters can be determined by minimizing the NLL loss, which is averaged over all validation samples. and .

[0074] However, determining model parameters solely by minimizing the average NLL loss over all validation samples is sample-inefficient, as it equally weights the NLL losses of negative (misclassified) and positive (correctly classified) samples. Even though the per-sample NLL loss for the positive class is typically smaller than that for the negative class, the average NLL loss over all samples will still be dominated by the positive samples due to their much larger number.

[0075] The fine-tuning loss function can be modified by adding a modulation term that considers the prediction difficulty PD 340 for each sample. For each validation sample, the anchoring probability P is determined. AN 335. Anchoring probability P AN335 is the highest predicted probability for the incorrect class, and therefore the predicted probability for the most competitive correspondent to the fundamental truth. In other words, among the non-fundamental truth classes, the predicted probability of the most likely incorrect class is chosen as the anchor probability P. AN 335: (Equation 4).

[0076] To determine the prediction difficulty PD 340, we need to start from the anchoring probability P AN 335 minus the predicted probability of the true value P GT 330. The fine-tuning loss function can therefore take the form of anchored loss, as defined in the following equation 5: (Equation 5) in It's a hyperparameter.

[0077] As shown in Equation 5, the anchoring loss is simply the NLL loss with a modulation term that includes the prediction difficulty PD 340. Although the NLL loss is used in this example, the result is also obtained along with other losses such as the Brier score loss or the hinge loss. Therefore, the fine-tuning loss function FTF350 can include losses modulated by a modulation term based on the prediction difficulty PD 340, such as NLL, Brier score, or hinge loss.

[0078] When the underlying true value is not a top-ranked prediction, the modulation term is upscaled to the NLL loss because the anchoring probability P AN 335 is greater than the predicted probability P of the fundamental truth value. GT 330. Otherwise, if the underlying true value is a top-ranked prediction, the modulation term is downscaled by NLL because the underlying true value prediction probability P... GT 330 is greater than the anchoring probability P AN 335. The upscaling and downscaling are proportional to the prediction discrepancy; for example, the prediction difficulty PD 340 is between the base true prediction and its competing counterpart prediction.

[0079] Therefore, the fine-tuned model is trained by training the fine-tuned model parameters. Training the fine-tuned model parameters includes, for each sample, as described above, training the output log-odds vector of the validation samples. Determine the basic true prediction probability P GT 330 and anchoring probability P AN 335. Basic true value prediction probability P GT 330 and anchoring probability P AN335 is then used to determine the prediction difficulty PD340. The prediction probability PD340 can then be used in the modulation term of the loss function FTF350 for fine-tuning. The fine-tuning of the loss function FTF350 further depends on fine-tuning the model parameters FMP'345—in the example above, and The fine-tuned model parameters FMP'345 are adjusted to minimize the fine-tuned loss function FLF 350, depicted by the dashed box OPT 355, through optimization. After optimization, the determined fine-tuned model parameters FMP 360 are output and stored. The fine-tuned model parameters FMP 360 can then be used in the training calibration module to fine-tune the log odds of the trained classification model before classifying sample inputs.

[0080] In this embodiment, the modulation loss is based on the focus loss (FL) instead of the anchor loss. The focus loss is based solely on the probability predicted by the base truth and describes the uncertainty associated with the base truth class. The modulation term based on the focus loss is given in Equation 6 below: (Equation 6).

[0081] Therefore, when the predicted probability of the base ground truth class is sufficiently high, focus loss is used to downscale the loss. Depending on the situation at hand, focus loss or anchored loss may be advantageous. For example, focus loss might reduce the loss value for easily classifiable samples, while using anchored probability dynamically rescales the loss based on how difficult it is to classify the sample. For samples that are moderately or moderately difficult to classify, using anchored probability in the loss function may provide better fine-tuning results. For this reason, while focus loss can provide faster computation for easily classifiable samples, the use of anchored probability is dynamic based on the difficulty of prediction and therefore can be applied more generally.

[0082] Figure 4 A detailed, but non-limiting, example of training a calibration module using a hierarchical optimization scheme according to an embodiment of the present invention is shown.

[0083] Multiple validation samples VS 305 can be provided to the training classification model TCM 310. Each validation sample includes an input instance—such as an image or document—and a ground truth classification indicating the true classification of the input instance. The training classification model TCM 310 can be a classification model that has been previously trained using the same in-system training method as calibration system 100 or an external training system. The training classification model TCM 310 can include a deep neural network. The training classification model TCM 310 can be configured to classify the input into... K A multi-class classifier that can classify one or more classes, for example, a multi-class multi-label classifier that can classify a single input into multiple class labels.

[0084] After inputting the validation sample VS 305 into the training classification model TCM 310, the output log-odds vector can be obtained from the training classification model TCM 310. More specifically, the output log-odds vector can be the output of the DNN layer before the last activation layer of the training classification model TCM 310. The last activation layer can be, for example, a softmax activation layer or a sigmoid activation layer. For example, if the training classification model TCM 310 is trained to classify samples as... K If one or more of the classes are selected, the log-odds vector will be output. It can be .

[0085] As previously described, the calibration module may include one or both of the fine-tuning submodule FT 320 and the grading submodule Bin 325. In embodiments where both the fine-tuning submodule FT 320 and the grading submodule Bin 325 are used, it may be more preferable for the operation of the grading submodule Bin 325 to be performed after the operation of the fine-tuning submodule FT 320, for example, to grade the fine-tuned output log-odds vector. More specifically, it may be advantageous to fine-tune the output log-odds vector before the scalar log-odds is quantized. That is, the output log-odds vector may be fine-tuned before being converted to a binary classification, or equivalently, the output log-odds vector may first be converted to a binary classification and the resulting binary classification (e.g., binary scalar labels). Scalar logarithmic probability ) can be based on Figure 3 Fine-tuning was performed using this method. It has been found that this provides a greater improvement than fine-tuning or grading alone. The output log-odds vector obtained from training the classification model TCM 310 can be referred to as the original output log-odds vector, and has been, for example, based on... Figure 3 The output log-odds vector that has been fine-tuned by the method can be called the fine-tuned output log-odds vector. The original output log-odds vector and the fine-tuned output log-odds vector are examples of the output log-odds vector.

[0086] The hierarchical submodule Bin 325 can convert the output log-odds vector (e.g., the original output log-odds vector or the fine-tuned output log-odds vector) into a binary classification BC 430 set. Converting a multi-class log-odds vector into a binary classification set can be achieved, for example, by using a pre-... k Predictive calibration can be performed individually or in several ways using class-by-class predictive calibration. Each binary classification includes a binary scalar label. Scalar logarithmic probability .

[0087] forwardk Each predictive calibration focuses on the front k Whether a prediction is correct. In the simplest case, such as in a multi-class classifier that is not a multi-label classifier, this becomes top-ranked label (e.g., top 1) prediction calibration, which focuses on whether the top-ranked prediction is correct. In this case, binary scalar labels y coding ,in It is an indicator function, and It is the first k The labels of each class, and the scalar-log probability becomes In a more general case, before the classifier output... k When making a prediction, the scalar log odds become ,in ,and I Marking k A set of top-ranked predictions.

[0088] Class-by-class prediction calibration transforms the output log-odds vector (or fine-tunes the output log-odds vector) into... K Each binary category provides a label indicating whether it belongs to a class. k The inference is based on the binary scalar label y encoding. The predicted probability of class 1, i.e. It is the sigmoid response of scalar log odds. .

[0089] Once the output log odds (or fine-tuned output log odds) have been converted to binary classification BC 430, they can be quantified, for example, by the use of bins. Scalar log odds are classified into bins defined by bin boundaries, and all entries (e.g., scalar log odds) within the same bin are represented by a single common value called the bin representative. Existing classification schemes achieve this either by defining an equal-width classification scheme or by defining bin boundaries such that each bin contains the same number of entries (this is called equal-quality classification), in which the interval is uniformly divided into multiple bins. However, in both of these classification schemes, the label information conveyed by the scalar log odds (e.g., This was not taken into account. Therefore, the hierarchical submodule Bin 325 defines the bin boundaries, making the label information... Log-probability of scalar Hierarchical (e.g., quantization) functions for performing operations The mutual information between them is shown in the expression below: (Equation 7) in It represents the quantified logarithmic probability.

[0090] According to Shannon's theorem, the solution to Equation 7 is the optimal encoder that maximizes the retention of the tag information given in the quantization budget—such as the number of warehouses.

[0091] The problem equivalent to that in Equation 7 can therefore be solved independently. The settings are defined as follows.

[0092] The quantizer can be defined in Equation 8 as shown below. : (Equation 8) in Define warehouse boundaries, and It represents the associated warehouse.

[0093] Based on probability density function Probability mass function Warehouse boundary BB 435 and warehouse representative Maximizing mutual information is equivalent to: (Equation 9) The graded loss function is defined in Equation 10 as shown below. BLF 440: (Equation 10) And among them The auxiliary variable set, or hierarchical optimization variable BOP 445, is introduced to facilitate optimization.

[0094] In some embodiments, the bin boundary that minimizes the hierarchical loss function BLF 440 can be found. BB435 and hierarchical optimization variables To make the graded loss function BLF 440 is minimized. This is achieved by calculating the value relative to the warehouse boundary. The derivative of the boundary value can be used to update the boundary value of the UBB 455 warehouse using gradient descent. : (Equation 11).

[0095] In some embodiments, closed-form computation can also be used to update the bin boundaries. Because It is positive, therefore The stationary point equations are: (Equation 12).

[0096] The left-hand side of equation 12 is The function. If the function is invertible, then The value can be calculated directly. Otherwise, a lookup table can be constructed to approximate it. The value of that The value of produces the right-hand side term of equation 13 below, which is obtained from Bayes' theorem. The obtained value is... The outpost.

[0097] (Equation 13).

[0098] There is also a special case for equation 13, in which ,in It is the sigmoid function. In this special case, the left-hand side of equation 13 simply becomes .

[0099] Given warehouse boundaries BB 435, variable The stationary point of the hierarchical loss function of BOP 445 is expressed as: (Equation 14) This updates the UBOP 450 variable. BOP 445 is completed in a closed form.

[0100] However, in some embodiments, by employing a graded loss function BLF 440 relative to The partial derivatives are also used to update the UBOP 450 variables using gradient descent. BOP 445, as shown in Equation 15: (Equation 15).

[0101] and Both can be updated iteratively and alternately in the corresponding update operations UBB 455 and UBOP 450.

[0102] Once the bin boundaries are determined by minimizing the hierarchical loss function BLF 440 This established the final warehouse boundary FBB 460. The final warehouse boundary FBB 460 was then used to determine the warehouse representative BR 465.

[0103] Each warehouse is defined as the interval between the boundaries of two consecutive warehouses, for example The intervals are defined. A bin representative is created for each bin. The bin representative is a value that will be used to represent the log odds of falling within a bin. There are several ways to determine the bin representative. For example, when the trained grading scheme is used to adjust the predicted probabilities, the bin representative value can be used as the adjusted predicted probability of a sample.

[0104] In some embodiments, a bin representative is determined based on samples within the bin, such as log odds. Each log odds is associated with a validation sample that has a corresponding underlying truth label. The proportion of log odds in the bin associated with validation samples having the same underlying truth label can then be determined. This proportion can then be used to obtain the corresponding predicted probability, which is set as the bin representative.

[0105] For example, a bin could contain log odds corresponding to 1000 samples. Of these 1000 samples, 80% have the base truth label "1". This means that the calibration probability of a test sample falling into this bin having the label "1" will also be 80%, or 0.8. The bin representative would then be set to 0.8.

[0106] In some embodiments, the output log-odds vector can be as follows: Figure 3 The parameters described above are fine-tuned before being input into the grading submodule Bin325. In this case, the log-probability quantized by the grading submodule Bin325 is fine-tuned, for example, using a matrix scaling model or a Dirichlet calibration model, the parameters of which are already as described in the reference. Figure 3 The determination is made. In this case, the warehouse representative of a particular warehouse can be determined by calculating the average of the sigmoid values ​​of the logarithmic odds falling within the warehouse.

[0107] In some embodiments, the original output log-odds vector can be converted into a binary classification BC 430 set. Based on the binary classification BC 430, the bin boundaries can be determined as described above. Figure 3 The method determines the warehouse representative BR 465 by fine-tuning the scalar log odds of binary classification BC 430.

[0108] The final bin boundary FBB 460 and bin representative BR 465 can then be used to define the grading scheme and can be stored in the calibration module CalMod 315.

[0109] In some embodiments, validation sample sets can be constructed to train the calibration module CalMod315 for each specific class. In other words, multiple validation sample sets can be used to train a fine-tuning model and / or grading scheme for calibrating each class. In such a case, Figure 3 and Figure 4 The method is repeated for each class, for example, for each validation set. For instance, to calibrate a multi-class classifier for classifying K classes, K validation sets can be used—one for each class. Each class can then have its own fine-tuning and / or grading scheme. For example, this can be achieved by using a set with parameters... and The fine-tuning model calibrates a class by fine-tuning the log odds predicted for that class, which can be done by using a parameterized model. and The fine-tuning model is adjusted to calibrate the second class by corresponding to the log odds of the second class, and so on. Similarly, the first class can be calibrated based on, for example, having bin boundaries. Representative of Hecang The first classification scheme is classified, and the second category can be classified according to, for example, having warehouse boundaries. Representative of Hecang The second classification scheme is classified.

[0110] In some embodiments, a single validation sample set can be used to train all validation samples. K A single fine-tuning model and / or a single hierarchical scheme for each class. For example, validation sets for multiple classes or even all classes can be merged into a single validation set. Then, based on... Figure 3 and / or Figure 4 One or more methods can determine a single fine-tuning and / or grading scheme for all classes. This can be referred to as shared class-by-class calibration.

[0111] Figure 5a A block diagram of a computer-implemented method 500 for training a calibration module according to an embodiment of the present invention is shown. Method 500 may correspond to... Figure 1 The operation of calibration system 100. However, this is not a limitation, because method 500 can also be performed using another system, apparatus or device.

[0112] This method can include accessing a 510-sample training classification model and multiple validation samples in an operation titled "Accessing the Training Classification Model and Validation Samples". The training classification model can be trained to classify input samples into... K One or more of the classes, providing associated predicted probabilities. The classification model can be trained using any training method. Each validation sample includes the sample input and a corresponding ground truth label indicating the true classification of the sample input. The training classification model and validation samples can be accessed separately from, for example, different storage locations. For instance, the training classification model and one of multiple validation samples can be accessed from local storage, and another can be accessed remotely.

[0113] This method can be implemented in an operation titled "Applying the trained classification model to validation samples," which involves applying the trained classification model to 520 or more validation samples. Validation samples can be input into the trained classification model.

[0114] This method can include, in the operation titled "Obtaining the Output Log-Odds Vector," obtaining 530 output log-odds vectors for each validation sample. It can includeK Each of the classes k The logarithmic odds, for example .

[0115] This method can include training a calibration module (540) within an operation titled "Training the Calibration Module" to determine the calibrated or adjusted predicted probabilities for the classification of sample inputs. The output log-odds vector and associated underlying ground truth labels can be input into the calibration module to train it. The calibration module may include at least one of a fine-tuning submodule and a grading submodule.

[0116] In the suboperation titled "Training the Fine-Tuning Submodule," training the 540 calibration module can include training the 550 fine-tuning submodule. The method for training the fine-tuning submodule will refer to... Figure 5b To describe in further detail.

[0117] In the suboperation titled "Training the Grading Submodule," training the 540 calibration module can include training the 560 grading submodule to obtain a grading scheme. The method for training the grading submodule will refer to... Figure 5c To describe in further detail.

[0118] In some embodiments, training the 540 calibration module may include training both the 550 fine-tuning submodule and the 560 grading submodule. In such a case, it is more preferable that the fine-tuning submodule and the fine-tuned output log-odds vector are trained by the trained fine-tuning submodule before training the grading submodule and subsequently applying the resulting grading scheme. That is, if both the fine-tuning submodule and the grading submodule are trained, it is more preferable that the output of the fine-tuning submodule—e.g., the fine-tuned output log-odds vector—is used as the input to the grading submodule.

[0119] This method can include attaching the training calibration module to the training classification model in an operation entitled "Attaching the training calibration module to the training classification model" to obtain a calibrated classification model.

[0120] Despite Figure 5a Not described in the text, but method 500 may further include using test data of the evaluation calibration to evaluate the calibrated classification model, such as a trained classification model with a training calibration module attached thereto.

[0121] Figure 5b A block diagram of a computer implementation method 550 for a fine-tuning submodule for training a calibration module according to an embodiment of the present invention is shown. Method 550 may correspond to Figure 1 The operation of calibration system 100. However, this is not a limitation, because method 550 can also be performed using another system, apparatus, or device.

[0122] This method can include, in the operation titled "Determining Baseline Truth Probabilities," determining 552 baseline truth probabilities for each validation sample. The baseline truth probabilities are predicted probabilities associated with the baseline truth class of the validation sample. That is, the output log-odds vector of the validation samples can be used to obtain... K The predicted probability for each of the classes, one of which is the ground truth class of the validation sample. The predicted probability can be determined from the output log odds vector by fine-tuning the model, such as a parametric model that takes the output log odds vector as input.

[0123] This method can include, in an operation titled "Determining Anchor Probabilities," determining a 553 anchor probability for each validation sample. The anchor probability is defined as the probability corresponding to the incorrect class with the highest probability. For example, if a classification model is trained to classify sample inputs into five classes, A, B, C, D, and E, and if a validation sample has the underlying truth class C, then the incorrect classes will be A, B, D, and E. The anchor probability will then be the highest probability among A, B, D, and E. In other words, the anchor probability represents the probability of the most competitive incorrect class.

[0124] This method can include, in an operation titled “Determining Prediction Difficulty”, determining the prediction difficulty for each validation sample by subtracting the base true probability from the anchored probability.

[0125] This method can include determining the parameters of a 555 fine-tuned model in an operation titled "Determining Model Parameters". For example, the fine-tuned model could be a matrix scaling model or a Dirichlet calibration model. The parameters of the fine-tuned model can be determined by minimizing a fine-tuning loss function averaged over multiple validation samples. The fine-tuning loss function can include a modulation term based on the prediction difficulty of each validation sample.

[0126] This method can include storing a fine-tuned model trained in operation 556 within an operation titled "Storing a Trained Fine-tuned Model". The trained fine-tuned model can include the model parameters determined in operation 555. The trained fine-tuned model can be stored in the calibration module.

[0127] The method may include, in an optional operation entitled “Fine-tuning the output log-odds vector,” fine-tuning the output log-odds vector for each of the 557 validation samples. This operation may appear in embodiments that further include training a tiered submodule, since the tiered submodule can be trained, for example, using the fine-tuned output log-odds vector after the fine-tuning submodule has been trained and applied.

[0128] It should be noted that, Figure 5bThe order described is not required, and the operations described therein may be performed simultaneously or in different orders. For example, operations 552 and 553 may be performed simultaneously, or operation 553 may be performed before operation 552.

[0129] Figure 5c A block diagram of a computer-implemented method 560 for training a grading submodule by determining a grading scheme according to an embodiment of the present invention is shown. The grading scheme is defined by bin boundaries and bin representatives, whereby bin boundaries define the interval of each bin, and bin representatives represent any sample falling into the corresponding bin. Method 560 may correspond to Figure 1 The operation of calibration system 100. However, this is not a limitation, as another system, apparatus, or device may also be used to perform method 560.

[0130] This method can include, in an operation titled "Transforming Output Log-Odds Vectors," transforming each output log-odds vector into a binary category. Each binary category can include a binary label and an associated scalar log-odds.

[0131] This method can include determining the warehouse boundaries of the 563 grading scheme by minimizing the grading loss function in the operation titled "Determining Warehouse Boundaries". To maximize the mutual information between label information and quantization log odds (which are quantized based on the grading scheme), the grading loss function is used as an equivalent problem. The grading loss function can be parameterized by the warehouse boundaries and by a set of auxiliary variables. Auxiliary variables can be provided simply to facilitate optimization. The grading loss function can then be minimized, which is equivalent to maximizing mutual information, thereby determining the warehouse boundaries and auxiliary variables that minimize the grading loss function.

[0132] This method can include, in an operation entitled “Quantizing the output log odds vector”, quantizing the scalar log odds 564 into a bin defined by the bin boundary established in operation 563.

[0133] This method can include determining a warehouse representative for each of the 565 warehouses in an operation entitled “Determining Warehouse Representatives”. A warehouse representing a particular warehouse can be a value representing the quantized scalar log odds of falling within that warehouse—for example, falling between warehouse boundaries defining the warehouse. Warehouse representatives can be determined based on the quantized scalar log odds falling within each warehouse. For example, in a particular warehouse, the proportion of scalar log odds corresponding to a validation sample having a specific underlying truth class can be used as the predicted probability of a test sample falling within that warehouse, classifying the product of said specific underlying truth class. The predicted probability can then be used as the warehouse representative. In another example, if the output log odds vector is fine-tuned before being converted to a binary classification and subsequently quantized, the warehouse representative can be determined by calculating the average of the sigmoid responses of the quantized scalar log odds within the warehouse.

[0134] This method can include a storage grading scheme (566) in an operation titled "Storage Grading Scheme". The grading scheme is defined by the warehouse boundaries established in operation 563 and the warehouse representatives established in operation 565. The grading scheme, such as the warehouse boundaries and warehouse representatives, can be stored by the grading submodule of the calibration module.

[0135] Generally speaking, one will understand Figure 5a , Figure 5b and Figure 5c The operations of method 500 can be combined and performed in any suitable order, such as sequentially, simultaneously, or in combination thereof, which, where applicable, conform to a specific order necessary, for example, by the input / output relationship. For example, when using both grading and fine-tuning, operation 562 can... Figure 5b The method is executed before or after 550.

[0136] Figure 6 A block diagram of a computer-implemented method 600 for classifying input samples according to an embodiment of the present invention is shown. Method 600 may correspond to... Figure 2 The operation of classification system 200. However, this is not a limitation, as another system, apparatus, or device may also be used to perform method 600.

[0137] This method may include obtaining 610 input samples in an operation entitled "Obtaining Input Samples". Input samples can be obtained from memory (such as local or external memory), a server, a remote entity (such as the cloud), or a networked device. Input samples can be obtained from sensors coupled to the system implementing this method. For example, input samples can be images obtained from a camera. However, this is not a limitation. Input samples can be audio samples, video, LiDAR, radar, text, etc., or combinations thereof.

[0138] This method can include, in the operation titled "Input Samples to Training the Classification Model," feeding the input samples 620 into the training classification model. The training classification model can then be trained to classify the input samples into... K One or more of the classes, providing associated predicted probabilities. The training classification model may include a deep neural network comprising multiple hidden layers between an input layer and an output layer, and may include at least one activation layer.

[0139] This method can include obtaining a 630 output log-odds vector from the trained classification model in an operation titled "Obtaining the Output Log-Odds Vector". More specifically, the output log-odds vector can be the output of a layer in the trained classification model before the final activation layer.

[0140] This method may include calibrating the 640 output log-odds vector in an operation entitled "Calibrating the Output Log-Odds Vector". Calibrating the output log-odds vector can also be referred to as adjusting the predicted probabilities. The predicted probabilities are typically derived from the output log-odds vector, and calibrating the output log-odds vector adjusts the predicted probabilities derived from said output log-odds vector. In other words, by calibrating or adjusting the output log-odds vector, the predicted probabilities derived from the output log-odds vector are adjusted or calibrated. The output log-odds vector may be passed as input to the calibration module. The calibration module may include at least one of a fine-tuning submodule and a grading submodule. The fine-tuning submodule may include, for example, according to... Figure 3 and Figure 5b The method involves training a fine-tuned model. Calibrating the output log-odds vector can include fine-tuning the output log-odds vector by applying the trained fine-tuned model to the output log-odds vector to obtain calibrated predicted probabilities. The grading submodule can include a grading scheme, defined for example according to... Figure 4 and Figure 5c The method determines the bin boundaries and bin representatives. Calibrating the output log-odds vector, or calibrating the fine-tuned log-odds vector where calibration also includes fine-tuning, can include: converting the output log-odds vector (or the fine-tuned log-odds vector) into a binary classification set, each binary classification including a binary label and a scalar log-odds, and for each class, classifying the scalar log-odds according to bin boundaries. The scalar log-odds can then be represented by the bin representative to which the scalar log-odds are classified. The bin representative value can then be provided as a predicted probability associated with the corresponding class.

[0141] This method may include, in an operation titled "Output Classification," a classification of 650 input samples. This classification may include predicted probabilities for one or more classes. In some embodiments, the class with the highest predicted probability may be output as the classification, for example, as the final decision. In some embodiments, the classification including adjusted predicted probabilities may be output by a calibration module. In some embodiments, the adjusted predicted probabilities may be output by the calibration module, while the classification decision (e.g., the most likely class or the top k classes, etc.) may be output by the classification model.

[0142] Generally speaking, to understand Figure 5a , Figure 5b and Figure 5c Method 500 and Figure 6 The operations of method 600 can be performed in any suitable order, such as sequentially, simultaneously, or a combination thereof, which, where applicable, conform to a specific order necessary, for example, by the input / output relationship. Some or all of the methods can also be combined; for example, method 600, which applies the training calibration module, can then be applied to the training calibration module trained according to method 500.

[0143] One or more methods can be implemented on a computer as computer-implemented methods, dedicated hardware, or a combination of both. For example... Figure 7 As illustrated, instructions for a computer (e.g., executable code) may be stored on a computer-readable medium 700, for example, in the form of a series of machine-readable physical symbols 710 and / or as a series of elements having different electrical (e.g., magnetic) or optical properties or values. The executable code may be stored in a temporary or non-temporary manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 7 The image shows CD 700.

[0144] Examples, embodiments, or optional features—whether or not indicated as non-limiting—should not be construed as limiting the invention as claimed.

[0145] It should be noted that the embodiments mentioned above are illustrative and not limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments.

[0146] In the claims, any reference marks placed between parentheses should not be construed as limiting the claims. The use of the verb "comprising" and its variations does not exclude the presence of elements or steps other than those recited in the claims. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one of..." when preceding a list or group of elements indicate the selection of all elements or any subset of elements from that list or group. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all A, B, and C. The invention can be implemented by means of hardware comprising several different elements, and by means of a suitably programmed computer. In a device claim enumerating several components, several of these components can be embodied by the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

Claims

1. A computer-implemented method (500) for calibrating a trained classification model, said training classification model being trained to classify input samples according to multiple classes and provide classification decisions and associated predicted probabilities, said input samples being images, said training classification model being an image classification model and including multiple hidden layers and at least one activation layer, said method comprising: - Access (510) to train the classification model; - Access (510) multiple validation samples, each validation sample having a basic truth label indicating a basic truth class; - Apply the trained classification model (520) to the multiple validation samples; - For each validation sample, obtain the (530) output log odds vector from the layer of the training classification model before the last activation layer; - The training (540) includes a calibration module for adjusting the prediction probabilities, which are derived from the output log odds vector. The calibration module comprises a fine-tuning submodule for adjusting the prediction probabilities by fine-tuning the output log odds vector and a grading submodule for adjusting the prediction probabilities by grading the output log odds vector; and The training of the calibration module includes training a (550) fine-tuning model, wherein the model parameters of the fine-tuning model are determined by the following: -For each of the plurality of verification samples: - Determine the (552) base truth probability from the output log odds vector. The base truth probability is the predicted probability associated with the base truth class of the validation sample. - Determine the (553) anchoring probability from the output log-odds vector, which is the highest probability of the incorrect class, and - The prediction difficulty (554) is determined by subtracting the base true probability from the anchored probability; - The model parameters of the fine-tuned model (555) are determined by minimizing a fine-tuning loss function averaged over the plurality of validation samples. The fine-tuning loss function includes a modulation term based on the prediction difficulty of each validation sample. The determined model parameters define the training of the fine-tuned model; and - Store the trained fine-tuned model (556) in the calibration module; and - Attach the training calibration module (570) to the training classification model to obtain a calibrated classification model.

2. The method of claim 1, wherein the calibration module includes a grading submodule, the grading submodule including a grading scheme defining bin boundaries and bin representations, and wherein training the calibration module includes determining (560) the grading scheme by: - For each validation sample, the output log-odds vector is transformed (562) into a binary classification set, each binary classification consisting of a binary label. and associated scalar log odds ; - The bin boundary is determined by minimizing the hierarchical loss function, which is equivalent to maximizing the mutual information between the quantized scalar log odds and the binary labels associated with the quantized scalar log odds, by updating the bin boundary and the set of auxiliary variables, wherein the quantized scalar log odds are quantized according to the hierarchical scheme. - Quantize the scalar log odds (564) into the bins defined by the determined bin boundaries; - For each warehouse, determine (565) a warehouse representative value to represent the quantified scalar log odds in the warehouse; and - Store (566) the bin boundaries and bin representatives of the defined grading scheme in the calibration module.

3. The method of claim 2, wherein minimizing the hierarchical loss function comprises updating at least one of the bin boundary and auxiliary variables by using at least one of gradient descent and closed-form computation.

4. The method of claim 2 or claim 3, wherein determining the warehouse representative of each warehouse comprises determining the proportion of validation samples whose corresponding scalar logarithmic probability falls into the warehouse, the proportion of validation samples having the same underlying truth label, and setting the warehouse representative of the warehouse to a predicted probability corresponding to the determined proportion.

5. The method according to any one of claims 2 to 3, wherein converting the output log-odds vector into a binary classification set comprises one of the following: -Before use k The calibration will be before k Each prediction is calibrated for binary classification. k It is an integer; - Use class-by-class calibration to calibrate the prediction for each of the plurality of classes; and - Use a shared class-by-class calibration to calibrate the prediction for each of the multiple classes.

6. The method according to any one of claims 2-3, wherein the fine-tuning loss function comprises at least one of the negative log-likelihood function, the Brier scoring function, and the hinge loss function.

7. The method according to any one of claims 2-3, wherein the fine-tuning loss function is based on a parametric model.

8. The method of claim 7, wherein the parameter model comprises a matrix scaling model and / or a Dirichlet calibration model.

9. The method according to any one of claims 2-3, further comprising, for each validation sample, fine-tuning the output log odds vector (557) according to the trained fine-tuning model, wherein the output log odds vector for each validation sample is fine-tuned before being quantized, and wherein determining the bin representative for each bin comprises: - For each scalar log-probability of falling into the bin, calculate the sigmoid response of the scalar log-probability. - Determine the average value of the calculated sigmoid response on the verification samples falling into the chamber, and - Set the warehouse representative of the warehouse to a determined average value.

10. A system (100) for calibrating a trained classification model, the trained classification model being trained to classify input samples according to multiple classes and provide classification decisions and associated predicted probabilities, the input samples being images, the trained classification model being an image classification model and including multiple hidden layers and at least one activation layer, the system comprising: - Data interface (120) is configured as follows: - Access the training classification model; - Access multiple verification samples; - The processor subsystem (140) is configured as follows: - Apply the trained classification model to the multiple validation samples; - For each validation sample, obtain the output log odds vector from the layers of the trained classification model before the last activation layer; - A calibration module is trained to adjust the prediction probabilities, which are derived from the output log odds vector. The calibration module includes a fine-tuning submodule for adjusting the prediction probabilities by fine-tuning the output log odds vector and a grading submodule for adjusting the prediction probabilities by grading the output log odds vector; and The training of the calibration module includes training a (550) fine-tuning model, wherein the model parameters of the fine-tuning model are determined by the following: -For each of the plurality of verification samples: - Determine the (552) base truth probability from the output log odds vector. The base truth probability is the predicted probability associated with the base truth class of the validation sample. - Determine the (553) anchoring probability from the output log-odds vector, which is the highest probability of the incorrect class, and - The prediction difficulty (554) is determined by subtracting the base true probability from the anchored probability; - The model parameters of the fine-tuned model (555) are determined by minimizing a fine-tuning loss function averaged over the plurality of validation samples. The fine-tuning loss function includes a modulation term based on the prediction difficulty of each validation sample. The determined model parameters define the training of the fine-tuned model; and - Store the trained fine-tuned model (556) in the calibration module; and - Attach the training calibration module to the training classification model to obtain a calibrated classification model.

11. A method (600) for classifying input samples according to multiple classes using a trained classification model and providing associated predicted probabilities, the method comprising: - Obtain a calibrated classification model, the calibrated classification model comprising a training classification model and a calibration module, wherein the training classification model is trained to classify input samples according to multiple classes and provide classification decisions and associated prediction probabilities, the input samples being images, the training classification model being an image classification model and comprising multiple hidden layers and at least one activation layer, and the calibration module comprising a fine-tuning submodule for adjusting prediction probabilities by fine-tuning the output log odds and a grading submodule for adjusting prediction probabilities by grading the output log odds; The model parameters for fine-tuning the model are determined by the following items: -For each of the multiple validation samples: - Determine the (552) base truth probability from the output log odds vector. The base truth probability is the predicted probability associated with the base truth class of the validation sample. - Determine the (553) anchoring probability from the output log-odds vector, which is the highest probability of the incorrect class, and - The prediction difficulty (554) is determined by subtracting the base true probability from the anchored probability; - The model parameters of the fine-tuned model (555) are determined by minimizing a fine-tuning loss function averaged over the plurality of validation samples. The fine-tuning loss function includes a modulation term based on the prediction difficulty of each validation sample. The determined model parameters define the training of the fine-tuned model; and - Store the trained fine-tuned model (556) in the calibration module; and - Obtain (610) input samples; - Input the input samples (620) into the training classification model; - Obtain the (630) output log odds vector from the layers of the training classification model before the last activation layer of the training classification model; - The adjusted prediction probability (640) is obtained by adjusting the prediction probability through a calibration module. The prediction probability is derived from the output log odds vector. Adjusting the prediction probability includes at least one of fine-tuning the output log odds vector according to the trained fine-tuning model and grading the output log odds vector according to the grading scheme; and - The output (650) corresponds to the classification of the input sample, which includes the obtained adjusted predicted probabilities.

12. The method of claim 11, wherein obtaining the input sample comprises obtaining the input sample from a sensor.

13. The method of claim 11 or claim 12, wherein the calibration model includes a grading scheme defining bin boundaries and bin representatives, and the method further includes quantifying the output log odds vector by: grading the output log odds vector into at least one bin according to the grading scheme, and representing the output log odds vector by bin representatives corresponding to said bins; and The output classification includes a quantized log odds indicating the calibration prediction probability determined by the calibration module, and the method further includes transmitting the quantized log odds to a central node connected to multiple sensors.

14. A system for classifying input samples according to multiple classes using a trained classification model and providing associated predicted probabilities, the system comprising: - Data interface (220) is configured as follows: - Obtain the input sample; - Access a calibrated classification model, which includes a training classification model and a calibration module. The training classification model is trained to classify input samples according to multiple classes and provide classification decisions and associated prediction probabilities. The input samples are images. The training classification model is an image classification model and includes multiple hidden layers and at least one activation layer. The calibration module includes a fine-tuning submodule for adjusting the prediction probabilities by fine-tuning the output log odds and a grading submodule for adjusting the prediction probabilities by grading the output log odds. The model parameters for fine-tuning the model are determined by the following items: -For each of the multiple validation samples: - Determine the (552) base truth probability from the output log odds vector. The base truth probability is the predicted probability associated with the base truth class of the validation sample. - Determine the (553) anchoring probability from the output log-odds vector, which is the highest probability of the incorrect class, and - The prediction difficulty (554) is determined by subtracting the base true probability from the anchored probability; - The model parameters of the fine-tuned model (555) are determined by minimizing a fine-tuning loss function averaged over the plurality of validation samples. The fine-tuning loss function includes a modulation term based on the prediction difficulty of each validation sample. The determined model parameters define the training of the fine-tuned model; and - Store the trained fine-tuned model (556) in the calibration module; and; - The processor subsystem (240) is configured as follows: - Input the input samples into the training classification model; - Obtain the output log odds vector from the layers of the training classification model before the last activation layer of the training classification model; - Adjusted prediction probabilities are obtained by adjusting the prediction probabilities through a calibration module. These prediction probabilities are derived from the output log-odds vector. Adjusting the prediction probabilities includes at least one of fine-tuning the output log-odds vector based on the trained fine-tuning model and grading the output log-odds vector according to a grading scheme; and - The output corresponds to the classification of the input sample, which includes adjusted predicted probabilities.

15. A computer-readable medium (1100) comprising temporary or non-temporary data (1110) representing instructions that, when executed by a processor system, cause the processor system to perform a computer-implemented method according to any one of claims 1-9 or 11-13.

Images