On-the-fly calibration of image classifier

Abstract

To provide a method for improving likelihood estimates formed by class probabilities that are output by an image classifier.SOLUTION: A computer-implemented method (700) for training an image classifier includes: accessing trainable parameters of the image classifier and a training dataset including a plurality of training images and corresponding class labels (710); performing one or more initial training steps of training the image classifier to acquire current parameter values of the trainable parameters (720); selecting training images and corresponding class labels from the training dataset (730); determining a current label vector for training the image classifier on the training images (740); and performing a current training step of training the image classifier on the training images using the current label vector (750).SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method for training an image classifier, and a corresponding system. The present invention further relates to a computer-implemented method for using a trained image classifier, and a corresponding system. The present invention further relates to a computer-readable medium. [Background technology]

[0002] Background of the Invention Machine learning-enabled image classification techniques, such as deep neural networks, can now surpass traditional methods in computer vision tasks and are increasingly being deployed in essential life applications such as driver assistance systems and medical imaging. In these types of applications, it is not enough for the image classifier to simply output an accurate classification (for example, predicting the correct class of an image in multi-class classification, detecting the correct object in an image in object detection, or predicting the correct class of each part of the input image in image segmentation). It is also important that the image classifier can accurately estimate likelihoods, for example, by outputting class probabilities that accurately represent the likelihood that the class is correct. For example, driver assistance systems can use these probabilities to weight the importance of a classification in relation to other available measurements of the environment, or to switch to a safe mode if the image classifier is uncertain.

[0003] Many image classifiers are known to be unable to accurately estimate likelihoods for unknown data. This phenomenon is called miscalibration and often leads to overconfidence in image classifiers. This poses a problem for setting high assurance. For example, if an autonomous vehicle recognizes a stop sign as a no-speed sign with very high confidence, an accident could occur, endangering human lives. There is a need for classifiers that output an appropriate confidence level commensurate with the accuracy of the prediction. The deviation of confidence from ground truth can be measured according to various metrics, such as prediction calibration error and negative log-likelihood. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Rethinking the Inception Architecture for Computer Vision,” by Szegedy et al., Proceedings CVPR 2016 (available at https: / / arxiv.org / abs / 1512.00567 and incorporated herein by reference). [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] Generally, two types of techniques for improving the calibration of image classifiers are known: post-hoc calibration and on-the-fly calibration. In post-hoc calibration, the image classifier itself remains unchanged, but a trained calibration model is applied to its output. On-the-fly calibration changes how the image classifier is trained. A known on-the-fly calibration method is called label smoothing and is disclosed in “Rethinking the Inception Architecture for Computer Vision,” by Szegedy et al., Proceedings CVPR 2016 (available at https: / / arxiv.org / abs / 1512.00567 and incorporated herein by reference). Instead of training the image classifier with a label vector representing the one-hot coding of class labels, in label smoothing, all non-target classes in the label vector are assigned a small, non-zero probability mass. As a result, the ground truth distribution has fewer spikes, leading to an image classifier that outputs a lower confidence level for the evaluation sample. This reduces the problem of overconfidence, but the likelihood estimate of the image classifier needs further improvement. [Means for solving the problem]

[0006] Summary of the Invention According to a first aspect of the present invention, a computer-implemented method and a corresponding system for training an image classifier are provided, as specified by claims 1 and 13. According to another aspect of the present invention, a computer-implemented method and a corresponding system for using a trained image classifier are provided, as specified by claims 10 and 14. According to one aspect of the present invention, a computer-readable medium is described as specified by claim 15.

[0007] Various embodiments relate to on-the-fly calibration techniques for image classification and image classifiers trained according to such on-the-fly calibration techniques. The image classifier can be configured to determine a prediction vector for an input image. The prediction vector may include class probabilities for each of several classes, for example, for the entire image if the image classifier is a multi-class classifier, or for each part of the input image if the image classifier is a multi-class semantic segmentation model. The objective of the techniques provided is to improve the likelihood estimate formed by the class probabilities output by the image classifier.

[0008] An image classifier can be trained by performing each training step. In such a training step, the image classifier is applied to a training image according to the current values ​​of its trainable parameters, and the current prediction vector can be obtained. The training signal can be derived by comparing the current prediction vector with the label vector. In the prior art, this label vector is usually a one-hot code of a known class label. In label smoothing, this one-hot code is smoothed. Based on this training signal, the trainable parameters of the image classifier can be updated, for example, by gradient descent. Typically, training an image classifier involves, for example, multiple such training steps and is performed, for example, over multiple epochs.

[0009] The inventors studied how known label smoothing techniques function and identified several factors that weaken their effectiveness. In particular, label smoothing uses a constant smoothing coefficient throughout the entire training process, which reduces its effectiveness in mitigating miscalibration. The inventors found that while label smoothing can reduce overconfidence in the early phases of training, in later training phases, the previous smoothing coefficient becomes too large to fit the current state of the model. In other words, the inventors recognized that label smoothing effectively overkills and underconfidentizes some predictions in the later phases of training. Due to the introduction of this new miscalibration, label smoothing may reduce the superiority over baseline at the end of training.

[0010] Another known aspect of label smoothing is the use of a uniform distribution for non-target classes. This is undesirable because a more accurate relative error for non-target classes can lead to a more effective training signal. For example, the probability of any image of a "passenger car" being misclassified as a "truck" might be predicted to be higher than the probability of it being misclassified as a "carrot," because "passenger cars" share more features with "trucks" than "carrots." As we have recognized, label smoothing effectively removes this relative error information, resulting in suboptimal training performance.

[0011] Another known aspect of label smoothing is to use the same label for all samples within a class. For example, in the class "baby" in the CIFAR-100 dataset, some samples are "baby girls" and others are "baby boys." Considering the common characteristics of these classes, it is reasonable to expect that, in addition to the target class "baby," a larger probability mass of the non-target class "girl" might be expected for the former sample group, and a larger probability mass of the non-target class "boy" might be expected for the latter. The inventors have recognized that label smoothing does not take this type of relevance into account and is therefore less effective in reducing miscalibration.

[0012] Interestingly, the inventors have discovered an effective on-the-fly calibration technique that improves the likelihood estimates provided by prior art image classifiers. According to this technique, the image classifier is trained using label vectors determined as soft labels based on prediction vectors output by the image classifier itself, in order to train the image classifier to correctly classify images it currently does. Soft labels, in contrast to hard labels (also known as one-hot coding), are label vectors containing multiple non-zero probabilities.

[0013] Generally, at some point during the training of an image classifier, one or more initial training steps have been performed, resulting in the current parameter values ​​of the image classifier's trainable parameters. The image classifier can be used according to these current parameter values ​​to determine a calibrated label vector. That is, the image classifier can be applied to training images according to the current parameter values ​​to obtain a predicted vector of class probabilities for the training images. The predicted class can be inferred from the predicted vector and compared to the ground truth class labels of the training dataset. If the predicted class is equal to the class label, a soft label can be determined based on the predicted vector and used for subsequent training steps.

[0014] Accordingly, according to the provided technology, the training step of an image classifier can be performed using a label vector determined based on the current state of the image classifier, or at least a relatively recent label vector, for example, a label vector determined in the current or previous training phase. In particular, at some point during training, the image classifier can perform both determining the label vector and performing a training step in which the parameters are updated, for example, by gradient descent, using a similar set of parameters. The label vector can be updated multiple times during training, for example, after a fixed number of training epochs have been performed, or once per training phase. This allows the current label vector used to train the image classifier to be determined based on the image classifier's time-calibrated and accurate predictions.

[0015] The provided on-the-fly calibration technique, called Label Smoothing from Predictions (LSP), offers several advantages. By using predictions as soft labels, training actively prevents the image classifier from generating high confidence for semantically ambiguous and inappropriately labeled samples. On the other hand, because predictions are used, unnecessary underconfidence is not introduced. As a result, the provided technique has been found to outperform state-of-the-art techniques across various datasets and image classifier architectures. LSP also exhibits good compatibility with different post-calibration methods as well as a wide range of training techniques and schedules. For example, given an image classifier with a known suitable training schedule, the image classifier can be retrained using LSP with a similar training schedule, with little to no hyperparameter customization. Compared to post-calibration, the provided technique requires less training data because it does not require an additional holdout set to train the post-calibration, thus improving effectiveness while avoiding the overhead of adding a calibration component. Compared to perfectly probabilistic models such as Bayesian models, the image classifiers trained as described herein are more efficient to train and use, making them applicable to larger and more complex data, and typically yielding higher accuracy.

[0016] In particular, the inventors have identified multiple types of training images that can cause mis-calibration. The provided technique is particularly effective for a training dataset that includes one or more training images of these types. Generally, these types of training images are semantically ambiguous and / or are inappropriately labeled. In particular, the following six types, namely, (a) multiple objects (e.g., where a train and a forest coexist in the same image), (b) a single object belonging to two classes (e.g., where one object simultaneously includes a maple tree and a forest), (c) a single object that is unrecognizable (e.g., due to low resolution, poor lighting, or similar characteristics, it is difficult to distinguish whether it is a kangaroo or a mouse), (d) a single object that is atypical in its class (e.g., while most pears in the training set look like real objects, the pear looks like a cartoon), (e) a hidden single object (e.g., where the head of a crocodile is hidden), (f) a single object that is inappropriately labeled (e.g., a more appropriate label would be a road rather than a plain) have been identified. According to the prior art techniques, hard labels encourage the image classifier to output predictions with 100% confidence for these samples, ignoring their inherent uncertainties. As a result, the trained image classifier may overfit to these samples and be mis-calibrated. In contrast, LSP uses calibrated soft labels to train the image classifier, maintaining the natural uncertainties in the corrected ground truth and suppressing the overconfidence of the image classifier.

[0017] Importantly, the label vector used for training can be based on the predictions of the image classifier itself and can be updated as the training of the image classifier progresses. This is in contrast to knowledge distillation techniques that use predictions from other pre-trained classifiers. Such techniques may be useful, for example, for training a simpler model using a more advanced model, but are not useful for reducing overconfidence as done in the provided technique.

[0018] As described herein, the image classifier trained in various ways can be used. When an input image is provided, the image classifier can be applied to determine a prediction vector for the input image, and the prediction can be output based on the prediction vector. The prediction can be, for example, a classification into a class, a class probability for a specific class, or a prediction vector of the class probability itself. By using the provided technique, for example, more accurate prediction class probabilities can be output, so that it can be used more safely, for example, in a control system. It has also been found that the classification accuracy of the image classifier may be improved, especially that the image classifier generalizes better for unknown examples. Therefore, even when simply outputting a classification into a specific class, the classifier trained as described is advantageous. Also, as described elsewhere, an improvement in the ability to recognize out-of-distribution samples has been found. Finally, the performance of the image classifier as a feature extractor is also improved. For this purpose, instead of applying the complete image classifier to the input image, only the initial part can be applied. For example, by applying a convolutional part and a subset of the fully connected layers of the model following it, a feature space representation of the input image can be obtained. It has also been found that this feature space representation generalizes better for unknown examples by using the provided technique.

[0019] Optionally, the determined current label vector of the training image can be used in a plurality of training steps for training the image classifier for the training image. For example, the label vector may be determined for the training images of the training dataset, and then a plurality of epochs of training, for example, at least 10 or at least 20 epochs, can be performed using this label vector. This increases the efficiency and stability of training compared to updating the label vector of the image for each training step performed on the image. This essentially avoids the label vector becoming a moving target and making the training impossible to converge.

[0020] Optionally, one or more initial training steps for training an image classifier may be performed independently of the image classifier, using label vectors determined, for example, as one-hot encoded class labels. Thus, training may consist of a first part in which label vectors derived from class labels are used, followed by a second part in which soft labels based on predictive vectors are used for accurate classification and are optionally updated during this second part. In the first part, sufficient accuracy and debias can be achieved so that the second part of training may be particularly effective. Failure to perform such initial training steps may result in degraded performance of the trained image classifier. It is a good choice for the first part to continue for a predetermined number of epochs (e.g., one or two) after, for example, a first decay of the learning rate.

[0021] Optionally, training may be performed over multiple training phases, with each training phase beginning with learning rate decay. It has been found that updating the label vector during such training phases is advantageous only after multiple training steps have been performed, for example, after one, two, or another fixed number of epochs have entered the training phase. This has the advantage that the updated label vector can utilize the confidence and accuracy boost associated with learning rate decay, since this boost typically occurs over multiple training steps. The remaining training phases can then use, for example, the updated label vector and the decayed learning rate.

[0022] In particular, one or more previous training steps of the image classifier may be performed according to the previous learning rate and the previous label vector, after which the previous learning rate may be decayed. After learning rate decay, one or more further training steps can be performed to train the image classifier with the decayed learning rate and the previous label vector to obtain updated parameter values ​​for the trainable parameters. The label vector of the training images can then be updated by applying the image classifier according to the updated parameter values, and then further training steps of the phase can be performed using the updated label vector. For example, the label vector may be updated after each learning rate decay, after every other learning rate decay, and so on.

[0023] The training steps performed between learning rate decay and label vector updates, at the discretion of the trainer, form a single training epoch. This has proven to be a good choice for most architectures and datasets because the aforementioned boost is sufficiently effective, and the updated labels can be used in the residual phase.

[0024] Optionally, if the class predicted by the image classifier for a training image is not equal to the class label for that training image, the current label vector may be determined based on the class label, independently of the prediction vector. For example, the label vector may be set to the one-hot encoding of the class label. While it is possible to use soft labels, it has been found that using hard one-hot encoding is preferable, for example, according to the label smoothing of prior art. This already has the effect of reducing the confidence of the model in erroneous predictions, encouraging the model to classify images correctly and eliminating the need to use soft labels.

[0025] Optionally, the techniques provided can be combined with post-calibration models. Post-calibration models can be applied during training to determine the current label vector from the predicted vector. The predicted class can be inferred from either the original predicted vector or the result of post-classification. Both are possible, as post-classification typically does not affect the predicted class. However, post-classification is not applied when the image classifier is applied as part of the training step. In use, the post-calibration model can be applied to the predicted vector determined by the image classifier. Interestingly, it has been found that combining the on-the-fly calibration described herein with post-calibration methods, particularly by already using post-calibration during training based on on-the-fly calibration, can yield a better overall calibrated likelihood than applying either technique individually. Various post-calibration methods are known and can be used herein, such as scaling methods like temperature scaling and latent Gaussian processes, or binning methods like histogram binning, especially I-MAX.

[0026] Optionally, a post-calibration model can be trained for an image classifier, for example, each time the post-calibration model is applied to a new set of parameter values ​​for the image classifier. In this way, throughout the training process, the post-calibration model can evolve along with the image classifier.

[0027] Optionally, at the time of use, a pre-trained image classifier can be used to determine the out-of-distribution score of an input image from the predicted vector output by the image classifier. For example, the out-of-distribution score may be determined based on the entropy of the predicted vector. Interestingly, as an additional benefit of less overconfidence, the provided technique has been found to improve the ability of image classifiers to recognize out-of-distribution samples. Many known image classifiers tend to output high confidence for out-of-distribution samples. In terms of determining the out-of-distribution score, the provided technique has been found to outperform prior art on-the-fly calibration methods. Furthermore, a small variance has been observed with respect to the provided technique, demonstrating robustness across different data and architectures.

[0028] Generally, the technologies provided apply to various types of image data, including video data, radar data, LiDAR data, ultrasound data, motion data, thermal images, or sonar data. Image data can be represented, for example, by pixel values ​​(e.g., for 2D image data), voxel values ​​(e.g., for 3D image data), or point clouds (e.g., for 2D or 3D image data). Image data can also represent various types of sensor data, such as ECG data represented as ECG images, or audio data represented as spectrograms.

[0029] In particular, the images input to the image classifier can represent a computer-controlled system and / or its environment, and the image classifier can be used to infer control data for controlling the computer-controlled system. For example, the system may be a vehicle such as an autonomous or semi-autonomous vehicle, a robot, manufacturing machinery, or a building. In principle, the technology provided can also be applied to other types of data, such as sensor data, such as time series of sensor measurements.

[0030] Depending on the type of data and content available, various model architectures and parameters can be used for the image classifier. For example, the image classifier may be a deep neural network, such as a convolutional neural network or a fully convolutional neural network. The image classifier may also be a semantic segmentation model configured to calculate class probabilities for each part of the input image.

[0031] Optionally, feature space representations obtained by applying a portion of a trained image classifier can be used as input to a further trained model. This further trained model may, for example, be used to infer control data for controlling a computer control system. Thus, the image classifier can be used as the basis for other downstream tasks. In this way, the image classifier can also be used as a feature extractor to impart its knowledge to these other tasks through transfer learning. Since uncertainty calibration is improved for the classification task of the image classifier, a correspondingly better predictive uncertainty can be obtained in the downstream tasks.

[0032] Those skilled in the art will understand that two or more of the above-described embodiments, realizations, and / or any aspects of the present invention can be combined in any way that is deemed useful.

[0033] Any modifications and variations of any system and / or any computer-readable medium corresponding to the described modifications and variations of the corresponding computer-implemented methods can be performed by those skilled in the art based on this description.

[0034] These and other aspects of the present invention will be further clarified and made clear by reference to the embodiments described as examples in the following description and by reference to the accompanying drawings. [Brief explanation of the drawing]

[0035] [Figure 1] This diagram shows a system for training a classifier. [Figure 2] This diagram shows a system for using a classifier. [Figure 3] This figure shows a detailed example of how to use a classifier. [Figure 4] This figure shows a detailed example of how to train a classifier. [Figure 5a] This figure shows a detailed example of how to train a classifier. [Figure 5b] This figure shows a detailed example of label calibration. [Figure 6] This figure shows a detailed example of an image classifier. [Figure 7] This figure shows a computer-implemented method for training a classifier. [Figure 8] This diagram shows a computer-implemented method using a classifier. [Figure 9] This is a diagram showing a computer-readable medium containing data.

[0036] Please note that the drawings are only schematic and are not drawn to scale. In the drawings, elements corresponding to elements already described may have the same reference numeral. [Modes for carrying out the invention]

[0037] Detailed description of the embodiment Figure 1 shows a system 100 for training a classifier. The classifier may be configured to determine a prediction vector for input instances. For example, the classifier may be an image classifier that acts on input images. The prediction vector may include class probabilities for each of several classes.

[0038] System 100 may include a data interface 120. The data interface 120 may be for accessing a training dataset 030 containing multiple training instances and corresponding class labels. For example, the number of training instances may be at least 1,000, at least 10,000, or at least 100,000. Alternatively or in addition to this, the data interface 120 may be for accessing trainable parameters 040 of the classifier. The parameters 040 may be for using the classifier in the manner described herein, for example, by system 200 in Figure 2 or system 300 in Figure 3.

[0039] For example, as shown in Figure 1, the data interface 120 may be comprised of a data storage interface 120 that can access data 030,040 from data storage device 021. For example, the data storage interface 120 may be a memory interface or a permanent storage interface, such as a hard disk or SSD interface, or it may be a personal, local or wide area network interface such as a Bluetooth, Zigbee or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 021 may be an internal data storage device of the system 100, such as a hard drive or SSD, or it may be an external data storage device, such as a network-accessible data storage device. In some embodiments, data 030,040 may each be accessed from different data storage devices, for example, via different subsystems of the data storage interface 120. Each subsystem may be of the type described above for the data storage interface 120.

[0040] System 100 may further include a processor subsystem 140 which can be configured to train a classifier during the operation of System 100. Training by the processor subsystem 140 may include performing one or more initial training steps to train a classifier and obtaining the current parameter values ​​of the trainable parameters 040. Training by the processor subsystem 140 may include selecting a training instance and its corresponding class label from the training dataset 030. Training by the processor subsystem 140 may include determining the current label vector for training a classifier on the training instance. To determine the current label, the processor subsystem 140 may apply the classifier to the training instance according to the current parameter values ​​to obtain a prediction vector, infer a predicted class from a plurality of classes from the prediction vector, compare the predicted class with the class label, and if the predicted class is equal to the class label, determine the current label vector as a soft label based on the prediction vector. Training by the processor subsystem 140 may include performing the current training step to train a classifier on the training instance using the current label vector.

[0041] The system 100 may further include an output interface for outputting trained data 040 representing the parameters of a trained (or "trained") model. For example, as shown in Figure 1, the output interface may be comprised of a data interface 120, which in these embodiments is an input / output ("IO") interface through which the trained parameters 040 are stored in a data storage device 021. For example, the trainable parameters of an "untrained" model may be at least partially replaced during or after training by trained parameters, such as the weights of a neural network and other types of parameters, to reflect the training in the training data 030. This is also shown in Figure 1 by records 040 representing both the untrained and trained parameters. In other embodiments, the trained parameters 040 may be stored separately from the "untrained" parameters. In some embodiments, the output interface may be separate from the data storage device interface 120, but generally, it may be of the type described above with respect to the data storage device interface 120.

[0042] Figure 2 shows, for example, a system 200 for using a pre-trained classifier as described herein, using the device 100 of Figure 1. The system 200 may include a data interface 220 for accessing the pre-trained parameters 040 of the classifier. In addition to using the model, the system can train the model and can be combined with, for example, the system in Figure 1.

[0043] For example, as shown in Figure 2, the data interface may consist of a data storage interface 220 that can access data 040 from data storage device 022. Generally, the data interface 220 and data storage device 022 can be of the same type as those described with reference to Figure 1 for data interface 120 and data storage device 021. The storage device may be part of the system 200. Although not shown in this figure, storage device 022 may also contain input instances to which a classifier can be applied.

[0044] The system 200 may further include a processor subsystem 240 which can be configured to acquire an input instance, such as an input image, during the operation of the system 200. The processor subsystem 240 may be further configured to apply a classifier to the input instance to determine a prediction vector for the input instance and to output a prediction based on the prediction vector. Alternatively or in addition to this, the processor subsystem 240 may be configured to apply a portion of the classifier to the input instance to acquire a feature space representation of the input instance and to output the feature space representation.

[0045] It will be understood that the same considerations and implementation options as those for the processor subsystem 140 in Figure 1 apply to the processor subsystem 240. Unless otherwise noted, it will be further understood that the same considerations and implementation options as those for system 100 in Figure 1 generally apply to system 200.

[0046] Figure 2 further illustrates various optional components of system 200. For example, in some embodiments, system 200 may include a sensor interface 260 for direct access to sensor data 224 acquired by sensor 072 in the environment 082. The sensor may be located in the environment 082, but may be located remotely from the environment 082 if, for example, the quantity can be measured remotely. Sensor 072 may be part of system 200, but does not necessarily have to be. Sensor 072 may have any suitable form, such as an image sensor, lidar sensor, radar sensor, pressure sensor, or contents temperature sensor. For example, as shown in the figure, sensor 072 may be a camera that captures an input image 226 representing system 200 and / or its environment. Sensor data interface 260 may have any suitable form corresponding to the type of sensor, including, but not limited to, a low-level communication interface based on I2C or SPI data communication, or a data storage interface of the type described above for data interface 220.

[0047] In some embodiments, the system 200 may include an actuator interface 280 for providing control data 226 to an actuator (not shown) in the environment 082. Such control data 226 is generated by a processor subsystem 240 and can be used to control the actuator based on the use of a classifier, for example, based on predictions by the classifier, or based on a feature space representation obtained by applying a part of the classifier, and the control data 226 can be inferred by applying a further trained model to the feature space representation. The actuator may be part of the system 200. For example, the actuator may be an electric, hydraulic, pneumatic, thermal, magnetic, and / or mechanical actuator. Specific but non-limiting examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, stepper motors, etc. Such types of control are illustrated with reference to Figure 3 for a (semi)autonomous vehicle.

[0048] In other embodiments (not shown in Figure 2), the system 200 may include output interfaces to rendering devices such as displays, light sources, loudspeakers, and vibration motors, which can be used to generate sensory-perceptible output signals that may be generated based on the use of the classifier. The sensory-perceptible output signals may directly represent the classifications, class probabilities, and / or feature space representations output by the classifier, but may also represent derived sensory-perceptible output signals for use in guidance, navigation, or other types of control of the system 200, for example.

[0049] In general, each system described herein, including but not limited to System 100 in Figure 1 and System 200 in Figure 2, may be embodied as a single device or apparatus, such as a workstation or server, or within one such device. The device may be an embedded device. The device or apparatus may include one or more microprocessors running appropriate software. For example, the processor subsystem of each system may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such CPU and / or other types of processing units. The software may be downloaded and / or stored in corresponding memory, such as volatile memory such as RAM or non-volatile memory such as flash. Alternatively, the processor subsystem of each system may be implemented in the device or apparatus in the form of programmable logic, such as a field-programmable gate array (FPGA). In general, each functional unit of each system may be implemented in the form of a circuit. Each system may also be implemented in a distributed manner with different devices or apparatus, such as distributed local or cloud-based servers. In some embodiments, the system 200 may be part of a vehicle, robot, or similar physical entity, and / or may represent a control system configured to control a physical entity.

[0050] Figure 3 illustrates the above example, in that the system 200 is shown as a control system for a (semi)autonomous vehicle 62 operating within the environment 50. The autonomous vehicle 62 may be autonomous in that it may include an autonomous driving system or a driver assistance system, the latter also referred to as a semi-autonomous system. The autonomous vehicle 62 may incorporate the system 200 to control the steering and braking of the autonomous vehicle based on sensor data acquired from a video camera 22 incorporated in the vehicle 62. For example, the system 200 may control the electric motor 42 to perform (regenerative) braking in the event of an unexpected traffic situation, for example, when it is predicted that the autonomous vehicle 62 will collide with a traffic participant. The system 200 can control the steering and / or braking in the event of a dangerous situation to avoid, for example, a collision with a traffic participant. The system 200 can apply an image classifier to the input images provided by the video camera. The system can determine, for example, whether the classification by the image classifier has a class probability above a threshold, or whether the input image has an out-of-distribution score above a threshold. Based on this, the system 200 may determine, for example, whether to use a classification vector or a prediction vector to control the vehicle 62 (e.g., if the class probability and / or out-of-distribution score shows sufficient confidence), or to use a fallback control mechanism, such as a braking mechanism, instead (e.g., if the class probability and / or out-of-distribution score shows insufficient confidence).

[0051] Next, we evaluate and describe several bottlenecks in the label smoothing (LS) of conventional on-the-fly calibration methods. In label smoothing, the label vector is determined using only the ground truth class label, not the predictive vector from the classifier itself. On the WRN-28-10 dataset CIFAR-100, the LS is compared to a cross-entropy one-hot coding baseline. A smoothing coefficient of 0.1 is used. Several bottlenecks were identified.

[0052] A constant smoothing coefficient over time, known as the Predictive Calibration Error (ECE), can be used to empirically investigate how much a model deviates from a well-calibrated state. However, this information is excluded in the absolute difference calculation of ECE, so in this case, the sign of the deviation becomes unknown. Therefore, we propose an alternative index called FlatECE, which replaces the absolute difference of ECE with a sum. FlatECE is,

number

[0053] In a constant smoothing coefficient (LS) across various classes and instances, a uniformly distributed probability is assumed for all non-target classes. This assumption has its drawbacks. It is intuitively understandable that the probability of some non-target classes may be higher than that of others. For example, in CIFAR-10, for an image labeled as a passenger car, a larger non-target probability mass should be assigned to a truck rather than a cat or dog, without even looking at the specific image, because a truck has a more similar appearance to a passenger car than a cat or dog.

[0054] Another assumption in LS is that all samples within a class have the same smoothed label. This means that it is assumed that the same features are shared across different samples within a class. Again, we can provide a counterexample. For example, consider two images annotated as babies. In one photo, the baby may look slightly more "feminine" and possess more "feminine" features than the baby on the left. In this case, the non-target class "girl" should be given a higher stochastic weight than the other photo.

[0055] Figure 4 shows a detailed but non-restrictive example of how to train a classifier. An image classifier is used as an illustrative example.

[0056] The image classifier IC420 can be applied to an input image. For example, the figure shows the image classifier IC applied to the training image TI(431) of the training dataset. Generally, the input image can be represented in various different ways, for example, by pixel values ​​(e.g., at least 128 or at least 1024 pixels), voxel values ​​(e.g., at least 128 or at least 1024 voxels), or a point cloud (e.g., a point cloud of at least 1024 or at least 4096 points). The image may be, for example, a black and white image, a grayscale image, or a color image.

[0057] The image classifier IC can be configured to determine a prediction vector for an input image. The prediction vector may include class probabilities for each of several classes. For example, the figure shows an image classifier IC applied to a training image TI(431) to obtain class probabilities CP1(421),...,CPn(422). For example, the number of classes may be at least 3, at least 10, or at least 20. The training image may have ground truth class labels CL(432) according to a training dataset, e.g., a single class to which the training image TI is assigned. Although not shown in this figure, the image classifier IC may also be a semantic segmentation model configured to output a set of class probabilities CP1,...,CPn for each part of the input image, e.g., a pixel, in which case the training dataset may include each class label CL that assigns a class to each part of the training image TI.

[0058] The image classifier IC may have various known model architectures suitable for the input image to be applied. The image classifier may be parameterized by a set of trainable parameters PARS(440). The number of trainable parameters may be, for example, at least 1000, at least 10000, or at least 100000. The image classifier IC may be a neural network, such as a deep neural network or a convolutional neural network. In this case, the set of parameters may include, for example, the weights of the nodes of the neural network. An example of such an image classifier is illustrated with reference to Figure 6.

[0059] As shown in the figure, the image classifier IC can be trained in the training operation "Training" 460. "Training" can be a training known from the prior art, except for using the label vectors determined as described in this specification. For example, for training, hyperparameters and / or training schedules known to function well for the in-hand data can be used. Generally, training can include respective training steps in which the image classifier IC is applied to respective training images TI. In such training steps, the image classifier IC can be applied to the training image TI according to the current value of the trainable parameter PARS, and current prediction vectors CP1,..., CPn can be obtained. "Training" can derive a training signal by comparing the current prediction vector CPi with the label vectors LV1(451),..., LVn(452) of the training image TI, and update the parameter PARS based on the training signal.

[0060] In one example, the image classifier IC may be, for example, a multi-class classifier with K>2 classes. Here,

Number

Number

Number

number

number

[0061] For example, “training” may be performed using a stochastic method such as stochastic gradient descent, for example, using the Adam optimizer disclosed in “Adam: A Method for Stochastic Optimization” by Kingma and Ba (incorporated herein by reference and available at https: / / arxiv.org / abs / 1412.6980). As is well known, such optimization methods may be heuristic and / or may reach a local optimum. Training may be performed instance by instance or in batches, for example, with a maximum or at least 64 instances, or a maximum or at least 256 instances.

[0062] Furthermore, a label calibration operation LC(450) is shown that determines the label vector LVi used in “training”. Interestingly, in the embodiments described herein, the label vector LVi of the training image TI may be determined based on the prediction vector CPi obtained by applying the image classifier IC to the training image TI according to the current set of parameters PARS.

[0063] Typically, during one or more initial training steps for training an image classifier IC, the label vector LVi is determined by the operation LC independently of the classification or class probability determined by the image classifier IC. For example, the label vector LVi may be set for one-hot coding of the class label LC.

[0064] At some point during training, the label calibration operation LC can update the label vector LVi of the training image TI. This is typically done less frequently, for example, once per learning rate decay, rather than at each training step performed for each training image. A detailed example of when the label vector LVi may be updated is provided with respect to Figure 5. Typically, the label calibration operation LC updates the label vectors of all images in the training dataset, but this is not mandatory.

[0065] To update the label vector LVi of the training image TI, inference can be performed using the image classifier IC according to its current parameter values ​​to obtain the prediction vector CPi of the training image TI. From this prediction vector, the predicted class for the training image TI can be inferred. Predictions can be divided according to their accuracy. Misclassified predictions have no clue to their soft label. The label vector LVi may be set to a one-hot coded label representing the hard label for the class label CL in this case, but otherwise, it may be determined independently of the prediction vector, for example, as a smoothed one-hot label corresponding to the class label.

[0066] On the other hand, for correctly classified samples, the label vector LVi can be set based on the prediction vector CPi. In particular, one possibility is to set the label vector LVi to be equal to the prediction vector CPi.

[0067] Furthermore, as shown in the figure, measures to further improve calibration performance include applying the post-calibration model PHC(470) to the prediction vector CPi, and using the post-calibration predictions CC1(471), ..., CCn(472) as the soft label LVi in the calibration framework, rather than the directly predicted CPi.

[0068] In principle, any post-hoc calibrator (PHC) can be used. A post-hoc calibrator could be, for example, a scaling post-calibrator. This method allows for scaling the raw output of the image classifier, such as the logit, using a continuous calibration function. For example, a scaling post-calibrator could be a temperature scaling (TS) or a latent Gaussian process (GP). In temperature scaling, the logit can be divided by a temperature coefficient T, which can be obtained by optimizing the NLL for the validation set. A latent Gaussian process (GP) can learn a continuous curve rather than a single temperature factor. For negative log-likelihood (NLL) metrics, GP has been experimentally found to perform better than TS in most situations. For predictive calibration error (ECE) metrics, TS has been found to perform better than GP when a limited number of calibration samples are available. In other cases, GP, with its larger number of configurable parameters, has been found to perform better.

[0069] A post-calibrator PHC can also be a binning post-calibrator. A binning post-calibrator can adjust the logit via a discrete binning function. For example, histogram binning, in particular, a method that optimizes the bin edges such as the I-MAX histogram binning method may be used. I-MAX can further improve sampling efficiency by merging class-specific predictions from the calibration set. Generally, scaling methods can maintain prediction accuracy but cannot have consistent performance for ECE when the number of bins in the ECE evaluation changes. Binning methods can have more consistent performance for ECE despite the loss of accuracy due to quantification.

[0070] The post-calibration model PHC is preferably trained for the image classifier IC according to its current parameters PARS, for example, according to updated parameters PARS each time it is applied. Training the post-calibration model PHC typically involves using a separate holdout set, such as a validation set, from the training dataset used to train the image classifier IC.

[0071] Therefore, the label vector LVi determined by the label calibration LC can be a) a calibrated accurate prediction and b) a label based on the class of the wrong sample, for example, a one-hot coded label. The determined label LVi can be used to train the image classifier for subsequent training steps, typically multiple training steps for each training image TI. After this, the process of determining an updated label vector LVi may be performed again using the trained parameters PARS obtained as a result of the subsequent training steps. Thus, the training steps, "training" and label calibration LC, are performed alternately until the end of the training schedule, resulting in a natively well-calibrated image classifier IC that can be used in some embodiments without requiring an additional calibration layer.

[0072] Figure 5a shows a detailed but non-restrictive example of how to train a classifier, such as an image classifier. This example is based on the example in Figure 4 and shows at what point in training the label calibration LC(550) may be applied. As explained in Figure 4, the label calibration LC can typically determine the label vector for all training images in the training dataset. Initially, the label vector used during training can be determined independently of the image classifier during training, for example, as one-hot encoding of the class labels. At some point in training, the label calibration LC can use the image classifier according to the current values ​​of its trained parameters by setting the label vector for images correctly classified as soft labels based on the prediction vector determined by the image classifier. For misclassified images, the label vector can be determined independently of the prediction vector, for example, as a hard label.

[0073] In this example, the image classifier is trained by stochastic gradient descent (SGD) using a multistep scheduler. Such training is well known, for example, in deep learning. Multistep scheduling refers to using a learning rate that decays multiple times during training. The period between learning rate decays can be called a training phase. At the start of training, the learning rate may be relatively large. Typically, accuracy and confidence increase initially, and then plateau at a certain level. After multiple training steps, for example, a fixed number of epochs, learning rate decay can be performed, which reduces the learning rate. This typically results in an increase in accuracy and confidence after several iterations, and then plateaus at a new, higher level, and so on thereafter.

[0074] As shown in the figure, training can be initiated using labels calibrated by label calibration LC at a point between the first and second learning rate decays. Waiting for the first learning rate decay typically has the advantage of sufficiently high training accuracy and less bias in predictions. After learning rate decay, one or more training steps can be performed, in which the previous label vectors are used, while the decayed learning rate is used. For example, a fixed number of training epochs ΔT(565) can be performed, followed by a label calibration LC. After label calibration, further training steps of the training phase can be performed according to the updated label vectors and decayed learning rate, for example, multiple epochs.

[0075] The time at the start of the training phase when the previous label vector is used is sometimes referred to as the waiting phase, referring to the typical training effect where accuracy and confidence increase only occur after waiting several iterations following learning rate decay. Predictions from epoch T after learning rate decay generally coincide with future predictions before the next learning rate decay, thus avoiding extraneous miscalibrations like those in known label smoothing. For most architectures and datasets, T=1 has been found to be a good choice.

[0076] Furthermore, label calibration LC may be applied after subsequent learning rate decay, and in this case, typically, a waiting phase including the same number of training steps, e.g., ΔT epochs, may be applied. For example, label calibration may be performed once per training phase after each learning rate decay. However, variations are also possible, for example, label calibration may be performed in every other training phase, or it may be performed first after the second learning rate decay instead of the first learning rate decay, etc. Label calibration LC can be performed throughout the entire training until the end of training 569 is reached. For example, the number of label calibrations performed based on the use of an image classifier may be at most three or at least three, or at most five or at least five. The waiting time, e.g., the number of epochs ΔT, and / or the label calibration frequency can be adjusted as hyperparameters.

[0077] Figure 5b shows a detailed but non-restrictive example of a label-calibrated LC(550). This label-calibrated LC can be used in combination with Figures 4 and / or 5a to determine label vectors for training images using an image classifier.

[0078] Label calibration LC can first apply the image classifier to the training images according to the current parameter values ​​to obtain the respective prediction vectors 552. The figure shows three such predictions [0.7,0.2,0.1], [0.1,0.8,0.1], and [0.5,0.3,0.2]. By inferring the prediction class from the prediction vectors and comparing the prediction class with the class label, the predictions can be split into two groups in comparison operation CP(557). In this example, the predictions are split into two groups: accurate predictions 553[0.7,0.2,0.1] and [0.1,0.8,0.1] and inaccurate predictions 555[0.5,0.3,0.2].

[0079] For images where the predicted class is not equal to the class label, the label vector may be determined independently of the prediction vector, based on the class label, for example, using one-hot encoding. In this example, the original hard label for this sample, [0.0,0.0,1.0], is used because the inaccurate prediction 555[0.5,0.3,0.2] is likely to mislead the image classifier.

[0080] For images where the predicted class is equal to the class label, the label vector can be determined as a soft label based on the predicted vector 553. In this example, the accurate predictions are calibrated by a post-calibration module PHC(570), such as temperature scaling (TS) or latent Gaussian process (GP), to apply additional regularization to the classifier. For illustrative purposes, the figure returns the calibrated soft labels 554[0.5,0.3,0.2] and [0.2,0.6,0.2]. The introduction of post-calibration PHCs allows for negative feedback to the image classifier. For example, if TS is integrated, when the image classifier is extremely overconfident, a temperature coefficient T>1 may be returned, resulting in stronger regularization as the soft labels are smoother. Overconfidence can be further mitigated in future training, and vice versa. An alternative is to use predictions that are not calibrated by post-calibration, in other words, to apply identity calibration.

[0081] The calibrated predictions 554 for correctly classified samples and the model-independent labels 555 for misclassified samples can be combined for future training as new soft labels 556[0.5,0.3,0.2], [0.2,0.6,0.2] and [0.0,0.0,1.0] (CB(552)).

[0082] Figure 6 shows a detailed but non-limiting example of a classifier IC(040) for use with the techniques described herein. In this example, the classifier is a neural network classifier, in particular a deep neural network. The illustrated process describes, for illustrative purposes, the classification of an input image in a convolutional neural network (CNN) at a high level and in a greatly simplified manner. As those skilled in the art will understand, other types of neural networks, such as deep neural networks, may also be used, and the following example is non-limiting and merely illustrative. Often, a CNN can be conceptually divided into two components: a convolutional portion containing at least one convolutional layer and a dense layer portion (e.g., fully-connected) containing at least one dense layer. However, it is not necessary to have a dense layer. For example, a trained classifier TC may be, for example, a fully convolutional neural network that does not contain any fully-connected layers.

[0083] The input image II(610) is input to the image classifier IC, more specifically to the first layer of the filter CL1(612-1) (or in some cases, a single filter). Each filter can perform a convolution on the input image II(610) and together generate an activated volume or "latent representation". Each "slice" of the activated volume can be one output of the filter. The latent representation may have a different spatial resolution than the input image II. The latent representation can then be used as input to the subsequent layer CL2(612-2). The filter in CL2 can then perform a convolution on the latent representation output by the first layer CL1 to generate a second latent representation. This process can be continued to further layers 612.

[0084] It should be noted that throughout the entire convolutional portion of a CNN, the spatial dimension of each latent representation may differ, or may differ from the spatial dimension of the input image, but the spatial relationship of the latent representation to the input image is preserved.

[0085] After the last layer CLk(612-k) of the convolutional portion of the CNN (often referred to as the last convolutional layer, but in some embodiments, the last layer CLk of the convolutional portion does not necessarily have to be a convolutional layer, provided that spatial relationships are preserved as described above), the resulting latent representation LRk(622) is input to the first layer of the high-density layer DL1(614-1). The high-density layer 614 may be a fully connected layer in which every neuron in each fully connected layer is connected to all neurons in the other layers. After passing through the high-density portion of the CNN, class probabilities CP1(621), ..., CPn(622) for each class may be output.

[0086] In various embodiments, instead of using a pre-trained image classifier by determining prediction vectors, it is also possible to use a pre-trained image classifier by applying only a portion of the pre-trained image classifier to the input image to obtain a feature space representation of the input image. In the exemplary image classifier IC shown in this figure, the feature space representation may be the output of the high-density layer DLi of the image classifier. This feature space representation can have various upstream applications, for example, by applying a further trained model to the feature space representation to infer control data for controlling a computer control system.

[0087] In various embodiments, the out-of-distribution score for an input image to which the model is applied can be determined using the prediction vector output by the image classifier, for example, the class probability CPi of the image classifier IC. As an additional advantage of less overconfidence, an image classifier trained as described herein may have an improved ability to recognize out-of-distribution samples. Known image classifiers tend to output high confidence for out-of-distribution samples. Therefore, compared to determining the out-of-distribution score based on the prediction vector of a known image classifier, using a trained classifier as described herein can improve the accuracy of the score. For example, outlier samples can be detected using the entropy of the prediction. This can be done not only for the CNN shown in the figure, but also for any image classifier trained as described herein.

[0088] The use of an image classifier to determine out-of-distribution scores can be evaluated using a test set in which in-distribution and out-of-distribution samples have different labels. The inventors performed such an evaluation by using the entropy of predictions as a threshold in AUC to detect outlier samples. CIFAR-10 was used as in-distribution data, and Street View House Number (SVHN) and CIFAR-10-C Gaussian Noise Severity 5 were used as out-of-distribution data. The provided technique was found to outperform other on-the-fly calibration methods across multiple tested dataset-network pairs. It also exhibits lower variability and robustness to different data and architectures compared to other calibration methods.

[0089] Next, we will describe the details and evaluation results of several implementations. A widely used metric to measure how much the confidence of a classifier deviates from its accuracy is the predictive calibration error (ECE). ECE mimics the behavior of a confidence diagram by considering the top 1 predictive probability as the confidence of the sample. The sample is sorted and divided into M bins according to its confidence. ECE is given by the following equation

number

[0090] The number of bins M is an important hyperparameter. Too few bins can lead to an underestimation of calibration errors (bias) because a lot of information may be smoothed out. Too many bins can lead to the index being affected by high variance. Traditionally, 15 bins are used, but recent studies have also reported results using 100 bins. The inventors used static binning with 100 bins for evaluation because they found that with 100 bins, different binning schemes did not change the ranking of different methods.

[0091] ECE is simple and common. However, another problem with ECE is that, in addition to its bias-variance trade-off, there are always trivial cases where the ECE is zero. This means that while a classifier can be perfectly ECE calibrated, it may output a fixed probability for any given input, which is undesirable. The literature suggests that these problems can be avoided by using the negative log-likelihood (NLL) as an indicator. NLL takes generalization and calibration into account. If two models have similar accuracy, the model with the lower NLL will be less likely to be miscalibrated.

[0092] For evaluation, the proposed technique is implemented using the software PyTorchImageClassification. Standard data augmentation (random cropping and random flipping) and He / Xavier initialization are performed for all network architectures. The experiment is repeated five times with different seeds. The training hyperparameters are the same as those used to train the original model. An SGD optimizer with a Nesterov momentum of 0.9 is used without dropout. The model is evaluated after full training (no early stopping).

[0093] WRN-28-10 is trained for 200 epochs with learning rates of 0.1 for the first 60 epochs, 0.02 for the next 60 epochs, 0.004 for the next 40 epochs, and 0.0008 for the last 40 epochs. The training batch size is set to 128, and the weight decay is 5 × 10⁻⁴.

[0094] DenseNet-BC-100 is trained for 300 epochs using a batch size of 64. The initial learning rate is 0.1, and it decreases by 90% at epochs 150 and 225, respectively. A slightly different weight decay is used, in this case 10⁻⁴. In the experiment, the compression ratio of DenseNet is 0.5 and the growth rate is k=12.

[0095] ResNeXt-29 uses a learning rate schedule similar to DenseNet-BC-100, with the base and initial channels set to 64. Weight decay is 5 × 10⁻⁴, and the batch size is set to 128.

[0096] For all datasets, 2% of the training samples are split for validation (calibration), resulting in 1000 samples for CIFAR-10 / CIFAR-100 and 2000 samples for TinyImageNet. Images in TinyImageNet are downsampled to 32x32 so that the model designed for CIFAR can be used efficiently.

[0097] In the experiment, a waiting phase of ΔT=1 epoch was used, and the number of label calibrations depended on the training schedule of the specific architecture. WRN has three learning rate decays and performs three label calibrations. In contrast, DenseNet and ResNeXt have only two learning rate decays. Correspondingly, label calibration is performed twice in these networks.

[0098] To evaluate the effectiveness of the proposed techniques, preliminary experiments were conducted on WRN using CIFAR-100. These techniques were applied without post-calibration (LSP identity) and with temperature scaling (LSP-TS) and compared to a cross-entropy (CE) one-hot coding baseline. Performance was evaluated with respect to top-1 accuracy (Acc@1), ECE, and NLL.

[0099] LSP identity is found to perform significantly better than baseline for ECE and NLL. As an additional benefit of soft labeling, LSP identity is also more accurate than baseline. Due to the favorable negative feedback of applying post-calibration, LSP-TS was found to perform better than LSP identity for all metrics. For this reason, LSP-TS was used as the standard LSP setting in the following experiments and is also referred to as LSP. In further experiments, LSP was found to perform better than not only the cross-entropy baseline but also the entropy penalty, label smoothing, and focus loss calibration. LSP was found to gain an extra accuracy boost from soft labeling for WRN and ResNeXt. LSP accuracy is competitive throughout the experiments, and even where other methods provide slightly higher accuracy, LSP provides a much better NLL, indicating that LSP is far less affected by miscalibration than other methods.

[0100] The training method provided can be configured according to at least two hyperparameters, namely the number of epochs in the waiting phase ΔT, and the frequency of label calibration. Experiments were conducted to comprehensively search for the best hyperparameters on CIFAR-100 for WRN-28-10.

[0101] Waiting Phase: The training schedule for WRN-28-10 has three learning rate decays. In this experiment, the calibration frequency is fixed at 3, and label calibration is performed for ΔT epochs after each learning rate decay. The search is performed for the best ΔT ∈ {-20, -10, -1, 0, +1, +10, +20}. This search shows that ΔT = 1 works best for both generalization and calibration.

[0102] Frequency: A separate experiment was conducted to determine the results for different frequencies. In the experiments with frequencies ∈ {1, 2, 3}, ΔT=1 was fixed, and the effect of label calibration at different learning stages was investigated. The results showed that performing label calibration at least once per learning rate decay is very beneficial. For WRN-28-10, frequency=3 generally performed best, which is expected to be a generally good setting. For the experiments with frequencies ∈ {4, 6, 199}, it was investigated whether additional label calibration before the first learning rate decay (frequency=4), in the middle of each learning stage (frequency=6), or after each epoch (frequency=199) is beneficial. It was found that performing additional label calibration before the first learning rate decay improved ECE at the expense of approximately 1% lower accuracy and even worse NLL. No improvement was found for frequencies=6 and 200 in this dataset. Therefore, it is preferable to perform label calibration at a maximum of one learning rate decay.

[0103] The experiment as a whole shows that performing label calibration after one epoch of each learning rate decay is most beneficial.

[0104] In further experiments, training using temperature smoothing was compared with training using a latent Gaussian process as a post-calibrator. In these experiments, LSP-GP improved the NLL but slightly degraded the ECE. This may be due to the high complexity of GP, which requires more calibration samples. Overall, the GP post-calibrator has been shown to work well in combination with the proposed training technique and to have good compatibility with other calibration methods.

[0105] Multiple options for post-calibration were compared: none, temperature smoothing, latent Gaussian process, and I-MAX. Experiments were performed on WRN-28-10 on TinyImageNet, as well as on WRN-28-10, DenseNet-BC-100, and ResNeXt-29 on CIFAR-100. It was found that using only post-calibration without the proposed on-the-fly calibration technique did not improve the accuracy of the image classifier, or even worsened it. On the other hand, it was found that combining the proposed on-the-fly calibration with post-calibration resulted in better generalization. It was also found that most post-calibration methods reduced miscalibration with respect to NLL and ECE. Miscalibration was further reduced when the image classifier was a priori normalized using the provided techniques. In the case of I-MAX, it was found that the provided technique did not improve calibration with respect to ECE. This is because I-MAX as a binning method has particular compatibility with the binning scheme of the ECE calculation and provides the best ECE performance among all post-calibration methods. In such cases, adding on-the-fly calibration as proposed herein may slightly degrade performance with respect to ECE. Nevertheless, the combination of the provided technique and I-MAX is beneficial because it avoids the image classifier accuracy degradation caused by I-MAX.

[0106] In another experiment, the proposed technique was compared with self-distillation. Self-distillation improves the performance of one model using soft labels from one model. This is in contrast to the provided technique, which uses soft labels from the model being trained on itself. Interestingly, the provided technique was found to perform better than self-distillation. Specifically, an experiment was conducted comparing the provided technique with a technique called unsupervised knowledge distillation (TfKD-self: Teacher-free Knowledge Distillation). The comparison was performed on CIFAR-10 / 100 for Acc@1, NLL, and ECE. The parameters used were τ=20 and α=0.95. Other hyperparameters were the same as in other experiments.

[0107] The provided technique was found to offer significantly better NLL and ECE across the entire setup, suggesting a substantial improvement in preventing miscalibration. This comes at the cost of lower accuracy for most dataset-model pairs. This corresponds to the fact that TfKD-self is designed for generalization, not to prevent miscalibration.

[0108] We identified several reasons why TfKD-self performs worse than the provided technique in terms of calibration. First, in TfKD-self, the cross-entropy loss with hard labels is applied to all samples. In the provided technique, hard labels are only used for misclassified samples when updating the label vector. Although the forward coefficient 1-α of the cross-entropy in TfKD-self is usually small, using one-hot coded labels without selection still increases the risk of overconfidence. If α is set to 1 to avoid overconfidence, a heavily regularized teacher model with training accuracy not reaching 100% may provide false soft labels, degrading the performance of the student model. The sample selection mechanism described herein does not require confronting such trade-offs.

[0109] Furthermore, the teacher model in TfKD-self is a pre-trained model. Predictions from a pre-trained model trained with hard labels are relatively close to the hard labels and are generally overconfident. Overconfidence in the student model can be expected because TfKD-self enforces consistency between the teacher's output and the student's output. In contrast, the technique provided allows calibrated predictions to be used as soft labels in different learning phases of training the model, which is equivalent to mimicking a well-calibrated model.

[0110] Furthermore, in TfKD-self, temperature scaling is applied simultaneously to logits from both the teacher and student models for complete imitation, whereas the technique provided only allows for the optional application of post-calibration (e.g., temperature scaling) to the logit from the teacher model (previous predictions). This allows the output of the student model to deviate from the output of the teacher model, increasing the degree of freedom for additional calibration for training.

[0111] Figure 7 shows a block diagram of a computer-implemented method 700 for training a classifier, such as an image classifier. The classifier may be configured to determine a prediction vector for an input instance, such as an input image. The prediction vector may include class probabilities for each of several classes. Method 700 may correspond to the operation of system 100 in Figure 1. However, this is not limiting, in that method 700 may be performed using other systems, apparatus, or devices.

[0112] Method 700 may include accessing the trainable parameters of a classifier and a training dataset containing multiple training instances and corresponding class labels in an operation titled “ACCESS CLASSIFIER, TRAINING DATA” (710).

[0113] Method 700 may include training a classifier. Training may include performing one or more initial training steps to train the classifier in an operation titled “INITIAL TRAINING STEPS” to obtain the current parameter values ​​of the trainable parameters (720). Training may include selecting corresponding class labels from training instances, e.g., training images and training datasets, in an operation titled “SELECT TRAINING DATA” (730).

[0114] Training may include an operation titled “DETERMINE LABEL VECTOR” in which the current label vector for training a classifier on the training instance is determined (740). Determining (740) may include an operation titled “APPLY CLASSIFIER” in which the classifier is applied to the training instance according to the current parameter values ​​to obtain a predictive vector (742). Determining (740) may include an operation titled “INFER CLASS” in which the predicted class among several classes is inferred from the predictive vector (744). Determining (740) may include an operation titled “COMPARE INFERRED TO CLASS LABEL” in which the predicted class is compared to the class label (746). Determining (740) may include determining the current label vector as a soft label based on the prediction vector if the predicted class is equal to the class label (748) in an operation titled “DETERMINE LABEL VECTOR BASED ON PREDICTION”.

[0115] Training may include performing the current training step, which is an action titled “TRAINING STEP,” in which the classifier is trained on the training instance using the current label vector (750).

[0116] Figure 8 shows a block diagram of a computer-implemented method 800 that uses a trained classifier, such as an image classifier. Method 800 may correspond to the operation of system 200 in Figure 2. However, this is not limiting, in that method 800 may be performed using other systems, apparatus, or devices.

[0117] Method 800 may include accessing the trained parameters of a classifier in an operation titled “ACCESS CLASSIFIER” (810). The classifier may be previously trained according to the methods described herein, for example, Method 700 in Figure 7.

[0118] Method 800 may include, in an operation titled “OBTAIN INSTANCE”, acquiring an input instance, such as an input image (820).

[0119] Method 800 may include applying a classifier to an input instance (830) in an operation titled “APPLY CLASSIFIER” to determine a prediction vector for the input instance. Method 800 may further include outputting a prediction based on the prediction vector (835) in an operation titled “OUTPUT PREDICTION”.

[0120] In lieu of or in addition to applying (830) and outputting (835), Method 800 may include, in an operation titled “APPLY CLASSIFIER PART”, applying a part of the classifier to an input instance (840) to obtain a feature space representation of the input instance. Method 800 may further include, in an operation titled “OUTPUT FEATURE SPACE REPRESENTATION”, outputting the feature space representation (845).

[0121] In general, it will be understood that the operation of Method 700 in Figure 7 and Method 800 in Figure 8 can be performed in any suitable order, for example, sequentially, simultaneously, or in combination thereof, according to the specific order required by the input / output relationship, where applicable. Some or all of the methods can be combined; for example, Method 800 using a trained classifier may be applied after the classifier has been trained according to Method 700.

[0122] The method may be implemented on a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. Furthermore, as shown in Figure 9, instructions for the computer, such as executable code, may be stored on a computer-readable medium 900, for example, in the form of a series of machine-readable physical marks 910 and / or, for example, as a series of elements having various electrical, magnetic, or optical properties or values. The medium 900 may be temporary or non-temporary. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, and online software. Figure 9 shows an optical disc 900. Alternatively, the computer-readable medium 900 may include data 910 representing trained parameters of a trained classifier, as described herein.

[0123] Examples, embodiments, or optional features, whether shown as non-limiting or not, should not be understood as limiting the claimed invention.

[0124] The embodiments described above are illustrative, not limiting, of the invention, and it should be noted that those skilled in the art can design many alternative embodiments without departing from the scope of the appended claims. Any reference numerals enclosed in parentheses in the claims should not be construed as limiting the claims. The use of the verb "comprise" and its conjugations does not exclude the existence of elements or stages other than those described in the claims. The article "a" or "an" preceding an element does not exclude the existence of multiple such elements. Expressions such as "at least one of" preceding a list or group of elements represent a selection of all or any subset of elements from the list or group. For example, the expression "at least one of A, B, and C" should be understood as including only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The invention can be implemented by hardware comprising multiple distinct elements and by a appropriately programmed computer. In a device claim listing multiple means, some of these means may be embodied by identical hardware components. The mere fact that certain means are described in different dependent claims does not imply that combinations of these means cannot be used advantageously.

Claims

1. 1. A computer-implemented method (700) for training an image classifier, the image classifier being configured to determine a prediction vector for an input image, the prediction vector including class probabilities for each of a plurality of classes, the method comprising: - accessing (710) trainable parameters of the image classifier and a training dataset comprising a plurality of training images and corresponding class labels; - performing one or more initial training steps to train the image classifier to obtain current parameter values ​​for the trainable parameters (720); - selecting (730) training images and corresponding class labels from the training dataset; - determining (740) a current label vector for training the image classifier on the training images by applying the image classifier to the training images according to the current parameter values ​​to obtain a prediction vector (742), inferring a predicted class from the plurality of classes from the prediction vector (744), comparing the predicted class with the class label (746), and determining (748) a current label vector as a soft label based on the prediction vector if the predicted class is equal to the class label; - performing a current training step (750) of training the image classifier on the training images using the current label vector; A method (700) comprising:

2. The method comprises: - performing one or more previous training steps of said image classifier according to a previous learning rate and according to a previous label vector; - decaying said previous learning rate; - performing one or more further training steps of training the image classifier according to a decayed learning rate and the previous label vector to obtain updated parameter values ​​for the trainable parameters; - determining the current label vectors of the training images by applying the image classifier according to the updated parameter values; - performing the current training step according to the current label vector and the decayed learning rate; Including, 10. The method (700) of claim 1.

3. the one or more further training steps forming one training epoch.

3. The method (700) of claim 2.

4. The method includes performing the one or more initial training steps using label vectors determined independently of the image classifier.

10. The method (700) of claim 1.

5. the image classifier is a semantic segmentation model configured to calculate class probabilities of the plurality of classes for each portion of the input image, and the method includes determining and using a current label vector for the each portion of the input image.

10. The method (700) of claim 1.

6. the input image is represented by pixel values, voxel values ​​or a point cloud; 10. The method (700) of claim 1.

7. If the predicted class does not equal the class label, the method further includes determining the current label vector based on the class label independent of the predicted vector.

10. The method (700) of claim 1.

8. determining the current label vector based on the predicted vector includes applying a post-calibration model to the predicted vector; 10. The method (700) of claim 1.

9. The method further includes training the post-calibration model for the image classifier.

9. The method (700) of claim 8.

10. 1. A computer-implemented method (800) of using a trained image classifier, the method comprising: - accessing (810) the trained parameters of an image classifier trained according to the method of claim 1; - obtaining an input image (820); applying 830 the image classifier to the input image to determine a prediction vector for the input image and outputting 835 a prediction based on the prediction vector, and / or applying 840 a part of the image classifier to the input image to obtain a feature space representation of the input image and outputting 845 the feature space representation; The method (800) includes:

11. The method includes determining an out-of-distribution score for the input image from the prediction vector. The method (800) of claim 10.

12. the input image represents a computerized control system and / or an environment of the computerized control system, the method comprising applying the portion of the image classifier to obtain the feature space representation; and applying a further trained model to the feature space representation to infer control data for controlling the computerized control system. The method (800) of claim 10.

13. 1. A system (100) for training an image classifier, the image classifier configured to determine a prediction vector for an input image, the prediction vector including class probabilities for each of a plurality of classes, the system comprising: a data interface (120) for accessing a training dataset (030) comprising a plurality of training images and corresponding class labels, and for accessing trainable parameters (040) of said image classifier; a processor subsystem (140), - performing one or more initial training steps to train the image classifier to obtain current parameter values ​​for the trainable parameters; - selecting training images and corresponding class labels from said training dataset; determining a current label vector for training the image classifier on the training images by applying the image classifier to the training images according to the current parameter values ​​to obtain a prediction vector, inferring a predicted class from the plurality of classes from the prediction vector, comparing the predicted class with the class label, and determining a current label vector as a soft label based on the prediction vector if the predicted class is equal to the class label; - performing a current training step of training the image classifier on the training images using the current label vector; a processor subsystem (140) configured to train the image classifier by A system (100) comprising:

14. A system (200) for using an image classifier, the system comprising: - a storage device (022) containing trained parameters (042) of an image classifier trained according to the method of claim 1; a processor subsystem (240), - obtaining an input image; applying the image classifier to the input image to determine a prediction vector for the input image and outputting a prediction based on the prediction vector, and / or applying a part of the image classifier to the input image to obtain a feature space representation of the input image and outputting the feature space representation; a processor subsystem (240) configured to: A system (200) comprising:

15. A transitory or non-transitory computer-readable medium (900), comprising: - instructions that, when executed by a processor system, cause said processor system to perform the computer-implemented method of claim 1; or - instructions that, when executed by a processor system, cause the processor system to perform the computer-implemented method of claim 10; or - trained parameters of an image classifier trained according to the computer-implemented method of claim 1 A transitory or non-transitory computer-readable medium (900) containing data (910) representing:

Images