Afterwards calibration method and system for uncertainty estimation

Abstract

The invention discloses a post calibration method and system for uncertainty estimation. The method comprises the following steps: acquiring a first output vector of a target basic classification model; performing multi-dimensional feature extraction operation on the first output vector to obtain a multi-dimensional feature vector; constructing a temperature function of a multi-feature temperature scaling algorithm according to the multi-dimensional feature vector; according to the temperature function, training an initial characterization calibrator to obtain a target characterization calibrator; and performing post calibration on the second output vector of the target basic classification model through the target characterization calibrator to obtain an uncertainty estimation result. The method can improve the calibration accuracy, reduces the calibration cost, and can be widely applied to the technical field of deep learning.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a post-calibration method and system for uncertainty estimation. Background Art

[0002] Traditional neural network models suffer from a widespread problem of overconfidence, particularly when faced with unknown or out-of-distribution (OOD) samples, often making high-confidence erroneous predictions. Numerous studies have shown that while modern deep neural network models offer high accuracy, their probability outputs are often poorly calibrated. This research has found that as model complexity increases, network calibration errors also increase, a phenomenon particularly pronounced in modern architectures such as ResNet. This widespread problem of poor calibration severely impacts the reliability and trustworthiness of models in practical applications.

[0003] Currently, there are two types of solutions: model training-based methods and post-processing-based calibration methods. Model training-based methods are theoretically more comprehensive, but they typically require multiple forward propagations or the training of multiple models, increasing computational complexity and storage requirements, which has certain limitations in large-scale practical applications. Post-processing-based calibration methods are relatively simple, but existing methods often lack flexibility and cannot fully capture the uncertainty characteristics of different types of predictions. Summary of the Invention

[0004] In view of this, the main purpose of the embodiments of the present invention is to provide a post-calibration method and system for uncertainty estimation, in order to solve at least one of the problems of the prior art. The present invention can improve the accuracy of calibration and reduce the cost of calibration.

[0005] To achieve the above object, an embodiment of the present invention provides a post-calibration method for uncertainty estimation, the method comprising:

[0006] Obtain the first output vector of the target basic classification model;

[0007] Performing a multidimensional feature extraction operation on the first output vector to obtain a multidimensional feature vector;

[0008] Constructing a temperature function of a multi-feature temperature scaling algorithm based on the multi-dimensional feature vector;

[0009] Training an initial characterization calibrator according to the temperature function to obtain a target characterization calibrator;

[0010] The target characterization calibrator is used to perform post-calibration on the second output vector of the target basic classification model to obtain an uncertainty estimation result.

[0011] In some embodiments, before obtaining the first output vector of the target basic classification model, the following steps are further included:

[0012] Training the initial basic classification model to obtain the target basic classification model;

[0013] A freezing operation is performed on the model parameters of the target basic classification model.

[0014] In some embodiments, performing a multidimensional feature extraction operation on the first output vector to obtain a multidimensional feature vector includes the following steps:

[0015] Obtaining the maximum value, minimum value, and average value of all elements in the first output vector;

[0016] Obtaining a difference between the maximum value and the minimum value;

[0017] Obtaining the degree of dispersion of each element in the first output vector to obtain a standard deviation;

[0018] Obtaining a distribution uncertainty indicator of the first output vector;

[0019] Obtaining the multidimensional feature vector according to the maximum value, the minimum value, the average value, the difference, the standard deviation, and the distribution uncertainty index;

[0020] The multidimensional feature vector includes an entropy value, a Gini coefficient, and a maximum probability value.

[0021] In some embodiments, training the initial characterization calibrator according to the temperature function to obtain the target characterization calibrator includes the following steps:

[0022] Inputting the multidimensional feature vector into the temperature function to obtain a temperature factor;

[0023] Performing calibration prediction on the first output vector according to the temperature factor to obtain a prediction result;

[0024] Obtaining a negative log-likelihood loss based on the prediction result;

[0025] The initial featurization calibrator is trained according to the negative log-likelihood loss to obtain the target featurization calibrator.

[0026] In some embodiments, the formula used to input the multidimensional feature vector into the temperature function to obtain the temperature factor includes:

[0027]

[0028] Among them, T(z i ) represents the temperature factor; represents the temperature function; represents a multidimensional feature vector; z i Represents the first output vector.

[0029] In some embodiments, the first output vector is calibrated and predicted based on the temperature factor to obtain a prediction result, and the formula used includes:

[0030]

[0031] in, Indicates the prediction result; z i represents the first output vector; T(z i ) represents the temperature factor.

[0032] In some embodiments, the negative log-likelihood loss is obtained based on the prediction result, and the formula used includes:

[0033]

[0034] Among them, L cal represents the negative log-likelihood loss; B cal represents a sampled data set; NLL(·) represents the negative log-likelihood function; softmax(·) represents the softmax function; Represents the prediction result; y i Represents the true value label; i represents the sample data set B cal The i-th data in ; λ represents the regularization coefficient; represents the initial characterized calibrator parameters.

[0035] To achieve the above objectives, another aspect of an embodiment of the present invention provides a post-calibration system for uncertainty estimation, the system comprising:

[0036] A first module is used to obtain a first output vector of a target basic classification model;

[0037] A second module is configured to perform a multidimensional feature extraction operation on the first output vector to obtain a multidimensional feature vector;

[0038] A third module is used to construct a temperature function of a multi-feature temperature scaling algorithm based on the multi-dimensional feature vector;

[0039] A fourth module is configured to train the initial characterization calibrator according to the temperature function to obtain a target characterization calibrator;

[0040] The fifth module is used to perform post-calibration on the second output vector of the target basic classification model through the target characterization calibrator to obtain an uncertainty estimation result.

[0041] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the post-calibration method of uncertainty estimation described above.

[0042] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the post-calibration method of uncertainty estimation described above.

[0043] To achieve the above objectives, another aspect of an embodiment of the present invention provides a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned post-calibration method for uncertainty estimation.

[0044] The embodiments of the present invention include at least the following beneficial effects: The present application provides a post-calibration method and system for uncertainty estimation, which obtains a first output vector of a target basic classification model; performs a multidimensional feature extraction operation on the first output vector to obtain a multidimensional feature vector; constructs a temperature function of a multi-feature temperature scaling algorithm based on the multidimensional feature vector; trains an initial characterization calibrator based on the temperature function to obtain a target characterization calibrator; and post-calibrates the second output vector of the target basic classification model through the target characterization calibrator to obtain an uncertainty estimation result. The temperature function is constructed by integrating multidimensional statistical features, and the trained model output is post-processed, thereby improving the calibration accuracy and reducing the calibration cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a flowchart of the steps of a post-calibration method for uncertainty estimation provided by an embodiment of the present application;

[0047] Figure 2 This is an overall framework diagram of a post-calibration method for uncertainty estimation provided by an embodiment of the present application;

[0048] Figure 3 This is a framework diagram of a multi-feature adaptive temperature scaling algorithm provided in an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this application. They are merely examples of devices and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0051] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0052] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0054] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0055] Post-calibration refers to calibrating the model using an additional calibration set after model training is completed. By optimizing certain hyperparameters, the model's predicted probability is made more consistent with the actual situation, thereby improving the calibration of the model's prediction.

[0056] Uncertainty estimation refers to the assessment of the uncertainty of the model's prediction results and the measurement of the model's confidence in the prediction results. The results can be used to assist decision-making and evaluate the reliability of model predictions.

[0057] Expected calibration error (ECE) is an indicator used to measure the calibration error of the model confidence. It is measured by calculating the deviation between the accuracy of samples in different probability intervals and the average prediction confidence. The smaller the value, the closer the model prediction confidence is to the actual accuracy.

[0058] KS error (Kolmogorov-Smirnov Error) is an indicator used to test the maximum deviation between the predicted distribution and the true distribution. The smaller the value, the better the model calibration effect.

[0059] Negative log-likelihood (NLL) is an indicator used to evaluate the difference between the model's predicted probability and the true label. The smaller the value, the higher the model's prediction quality.

[0060] Temperature scaling (TS) is a method that improves the calibration of the model's probability distribution by adjusting the logits output by a neural network by introducing a scalar temperature parameter, T. Logits refer to the raw output values of the last layer of the neural network, which are scores before being processed by the activation function. Temperature scaling optimizes the calibration of the probability distribution by adjusting the scale of these scores to make them more consistent with the true confidence level.

[0061] Adaptive Temperature Scaling (ATS) refers to a calibration method that expands the temperature parameter into a function related to logits. By analyzing the statistical properties of logits, the temperature factor is estimated and the overconfidence level of different predictions is fine-grainedly adjusted.

[0062] With the widespread application of deep learning in safety-critical fields such as autonomous driving, medical diagnosis, financial risk management, and climate forecasting, uncertainty estimation in neural networks has become crucial. Uncertainty in predictions can improve model reliability, help decision makers assess the credibility of model predictions, and provide comprehensive information support for critical decisions. For example, in autonomous driving systems, accurate uncertainty estimation enables vehicles to make safer decisions in complex or unknown scenarios. In multimodal fusion systems, accurate uncertainty can guide the modal fusion process, resulting in a more reliable and interpretable fusion process.

[0063] However, traditional neural network models suffer from a widespread problem of overconfidence, particularly when faced with unknown or out-of-distribution (OOD) samples, often making high-confidence erroneous predictions. Numerous studies have shown that while modern deep neural network models offer high accuracy, their probability outputs are often poorly calibrated. This research has found that as model complexity increases, network calibration errors also increase, a phenomenon particularly pronounced in modern architectures such as ResNet. This widespread problem of poor calibration severely impacts the reliability and trustworthiness of models in practical applications.

[0064] Currently, there are two main approaches to this problem: model-training-based methods (such as Bayesian neural networks (BNNs), ensemble methods, and MC-Dropout techniques) and post-processing-based calibration methods (such as Platt scaling and temperature scaling). Model-training-based methods are theoretically more robust, but they typically require multiple forward propagations or the training of multiple models, resulting in high computational complexity and storage requirements, making them difficult to deploy in large-scale, practical applications. For example, ensemble methods require the training and storage of multiple models, while MC-Dropout requires multiple sampling during inference, both of which significantly increase deployment costs and inference latency. Post-processing-based calibration methods are relatively simple, but existing methods often lack flexibility and fail to fully capture the uncertainty characteristics of different types of predictions. For example, the traditional temperature scaling (TS) method introduces a scalar temperature parameter, T, to adjust the logits of the neural network output. T is optimized by minimizing the negative log-likelihood loss on the calibration set. However, this method implicitly assumes that all samples have the same degree of calibration error. In practice, this assumption often fails when the model exhibits systematic bias in predictions for different categories or confidence intervals. Consequently, traditional temperature scaling methods fail to fully capture the uncertainty characteristics of different predictions. The adaptive temperature scaling (ATS) method improves this, however, ATS only relies on simple statistics of the original logits and cannot fully capture the multidimensional characteristics of prediction uncertainty.

[0065] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a post-calibration method for uncertainty estimation, which may include but is not limited to steps S100 to S500:

[0066] Step S100, obtaining a first output vector of a target basic classification model;

[0067] Step S200, performing a multidimensional feature extraction operation on the first output vector to obtain a multidimensional feature vector;

[0068] Step S300, constructing a temperature function of a multi-feature temperature scaling algorithm based on the multi-dimensional feature vector;

[0069] Step S400: training an initial characterization calibrator according to the temperature function to obtain a target characterization calibrator;

[0070] Step S500: performing post-calibration on the second output vector of the target basic classification model through the target characterization calibrator to obtain an uncertainty estimation result.

[0071] Before step S100 in some embodiments, the method further includes training the initial basic classification model to obtain the target basic classification model; and freezing the model parameters of the target basic classification model. For example, when training the initial basic classification model, a small batch of data is sampled from the training set, and a prediction result is calculated by forward propagation. The negative log-likelihood loss is calculated based on the prediction result, and the parameters θ of the initial basic classification model are updated by back propagation until convergence, thereby obtaining the target basic classification model f θ , and freeze the parameters θ of the fixed target base classification model. By freezing the fixed parameters θ, there is no need to modify the target base classification model architecture or retrain the base classification model in the subsequent post-processing process, which reduces the computational cost.

[0072] In step S100 of some embodiments, a rough uncertainty estimation is performed by using a target basic classification model with fixed parameters to obtain a first output vector, which lays a good foundation for subsequent multi-feature extraction.

[0073] In step S200 of some embodiments, multidimensional statistical features of logits are integrated. These multidimensional statistical features may include basic statistics and distribution uncertainty indicators. Optionally, basic statistics include the maximum value of all elements in the logits output by the target basic classification model, the minimum value of all elements, the average value of all elements, the difference between the maximum and minimum values, and the standard deviation; distribution uncertainty indicators include entropy, Gini coefficient, and maximum probability value. These multidimensional statistical feature vectors together constitute the input vector of the temperature function.

[0074] In some embodiments, step S200 may include but is not limited to steps S210 to S250:

[0075] Step S210, obtaining the maximum value, minimum value and average value of all elements in the first output vector;

[0076] Step S220, obtaining the difference between the maximum value and the minimum value;

[0077] Step S230, obtaining the degree of dispersion of each element in the first output vector to obtain a standard deviation;

[0078] Step S240, obtaining a distribution uncertainty index of the first output vector;

[0079] Step S250, obtaining the multidimensional feature vector according to the maximum value, the minimum value, the average value, the difference, the standard deviation, and the distribution uncertainty index;

[0080] The multidimensional feature vector includes an entropy value, a Gini coefficient, and a maximum probability value.

[0081] In steps S210 to S230 of some embodiments, the maximum value z of all elements in the first output vector is obtained. max , minimum value z min ,average value The difference between the maximum and minimum values Δz i , standard deviation σ z Alternatively, there is the following formula:

[0082] z max =max(z i )

[0083] z min =min(z i )

[0084]

[0085] Δz i =z max -z min

[0086]

[0087] Among them, z i represents the first output vector; K represents the number of dimensions of the output vector; k represents the kth dimension; z k Represents the logits value of the kth dimension.

[0088] In step S240 of some embodiments, a distribution uncertainty index of the first output vector is obtained, and the distribution uncertainty index includes an entropy value H(p i ), Gini coefficient G(p i ) and the maximum probability value p max Alternatively, there is the following formula:

[0089]

[0090] p k =σ SM (z i ) k

[0091]

[0092] p max,i =max(p i )

[0093] Among them, p i represents the i-th probability vector; p k represents the kth value of the probability vector; σ SM represents softmax; (z i ) k Represents the kth value of the i-th logits vector.

[0094] In step S300 of some embodiments, the multidimensional statistical features extracted from the first output vector of the target basic classification model together constitute the input vector of the temperature function The temperature function of the multi-feature temperature scaling algorithm is in, is a learnable mapping function (such as a multi-layer perceptron). By jointly optimizing the parameters of the temperature function and the regularization term, the multi-feature temperature scaling algorithm can adaptively balance the contribution of different features to the temperature factor.

[0095] In some embodiments, step S400 may include but is not limited to steps S410 to S440:

[0096] Step S410, inputting the multidimensional feature vector into the temperature function to obtain a temperature factor;

[0097] Step S420: performing calibration prediction on the first output vector according to the temperature factor to obtain a prediction result;

[0098] Step S430, obtaining a negative log-likelihood loss according to the prediction result;

[0099] Step S440: Training the initial characterization calibrator according to the negative log-likelihood loss to obtain the target characterization calibrator.

[0100] In step S410 of some embodiments, the multidimensional feature vector is mapped to a sample-specific temperature factor through a temperature function to achieve personalized temperature adjustment for different samples. The calculation formula is as follows:

[0101]

[0102] Among them, T(z i ) represents the temperature factor; represents the temperature function; represents a multidimensional feature vector; z i Represents the first output vector.

[0103] In step S420 of some embodiments, the temperature factor is used to calibrate and predict the first output vector to obtain a prediction result. The calculation formula is as follows:

[0104]

[0105] in, Indicates the prediction result; z i represents the first output vector; T(z i ) represents the temperature factor.

[0106] In step S430 of some embodiments, based on the prediction results, the negative log-likelihood loss of the mini-batch dataset sampled on the calibration set is minimized, and the calculation formula is as follows:

[0107]

[0108] Among them, L cal represents the negative log-likelihood loss; B cal represents a sampled data set; NLL(·) represents the negative log-likelihood function; softmax(·) represents the softmax function; Represents the prediction result; y i Represents the true value label; i represents the sample data set B cal The i-th data in ; λ represents the regularization coefficient; represents the initial characterized calibrator parameters.

[0109] In step S440 of some embodiments, the initial characterization calibrator is trained according to the negative log-likelihood loss to obtain the target characterization calibrator. For example, the parameters of the initial calibrator are updated according to the negative log-likelihood loss. Right now Until the convergence condition is reached, the target characterization calibrator is finally output Here, η represents the learning rate.

[0110] In step S500 of some embodiments, a second output vector of the target basic classification model is obtained, and the second output vector is post-calibrated by a target characterization calibrator to obtain an uncertainty estimation result without modifying the architecture of the target basic classification model or retraining the target basic classification model.

[0111] refer to Figure 2 , Figure 2 It shows the complete process from input samples to basic model training, constructing a temperature function with multi-feature temperature scaling by extracting multi-dimensional statistical features of logits, calculating temperature parameters, training the calibrator, and post-calibrating the output of the basic classification model through the trained calibrator to obtain the uncertainty estimation results, presenting the data flow and processing process.

[0112] The Multi-Feature Temperature Scaling (MTS) proposed in the embodiments of the present invention integrates the multidimensional statistical features of logits (including basic statistics and distribution uncertainty indicators) to construct a temperature function, thereby achieving fine-grained calibration adjustments for different predictions and improving the flexibility and interpretability of calibration.

[0113] refer to Figure 3 , Figure 3 This paper presents the application framework of a multi-feature adaptive temperature scaling algorithm, which can be divided into two main phases: model training and calibrator training. The overall process is as follows: first, the base classification model is trained, followed by the training of a featurized calibrator while keeping the model parameters fixed. This unifies the feature extraction, temperature prediction, and probabilistic calibration processes, and performs a joint optimization by minimizing the negative log-likelihood loss on the calibration set. During this optimization process, both the accuracy of the model prediction and the stability of the temperature function are considered.

[0114] For example, Figure 3 As shown, in the first stage of model training, when training the initial basic classification model, the training set D train Sampling small batches of data Calculate the predicted result z through forward propagation i =f θ (x i ), then calculate the negative log-likelihood loss And use back propagation to update the parameters θ of the initial basic classification model, then we have Until the convergence condition is reached, the target basic classification model f is obtained θ Among them, x i In the second stage of calibrator training, the target base classification model f is fixed θ After the parameters θ are obtained from the calibration set D cal Sampling small batches of data Get the logits output by the target basic classification model (i.e. the first output vector z i ), then we have logits: z i =f θ (x i ), Perform multidimensional feature extraction on the first output vector to extract an 8-dimensional feature vector have Then calculate the temperature factor Performing calibration prediction, we can obtain Then calculate the calibration loss L based on the calibration prediction results cal , then Update the initial characterization calibrator parameters based on the calculated calibration loss have Until the convergence condition is reached, the target characterization calibrator is finally output

[0115] An embodiment of the present invention further provides a post-calibration system for uncertainty estimation, which can implement the above-mentioned post-calibration method for uncertainty estimation. The system includes:

[0116] A first module is used to obtain a first output vector of a target basic classification model;

[0117] A second module is configured to perform a multidimensional feature extraction operation on the first output vector to obtain a multidimensional feature vector;

[0118] A third module is used to construct a temperature function of a multi-feature temperature scaling algorithm based on the multi-dimensional feature vector;

[0119] A fourth module is configured to train the initial characterization calibrator according to the temperature function to obtain a target characterization calibrator;

[0120] The fifth module is used to perform post-calibration on the second output vector of the target basic classification model through the target characterization calibrator to obtain an uncertainty estimation result.

[0121] It can be understood that the contents of the above method embodiments are applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0122] An embodiment of the present invention further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the aforementioned post-calibration method for uncertainty estimation. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0123] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0124] refer to Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0125] The processor 601 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0126] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called by the processor 601 to execute a post-calibration method for uncertainty estimation in the embodiments of this application.

[0127] Input / output interface 603, used to implement information input and output;

[0128] Communication interface 604, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0129] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0130] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0131] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned post-calibration method for uncertainty estimation.

[0132] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] An embodiment of the present invention further provides a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned post-calibration method for uncertainty estimation.

[0134] In summary, the post-calibration method and system for uncertainty estimation according to the embodiments of the present invention have the following advantages:

[0135] 1. The uncertainty estimation method based on post-calibration in the embodiment of the present invention improves the accuracy and credibility of uncertainty estimation by post-processing the output of the trained model. There is no need to modify the original model architecture or retrain the model, which greatly reduces the computational overhead. It is easier to deploy and apply in actual scenarios, adapt to different types of prediction scenarios, and achieves high-precision calibration with low computational cost.

[0136] 2. The multi-feature temperature scaling (MTS) algorithm proposed in the embodiment of the present invention integrates the multidimensional statistical features of logits to construct an adaptive temperature function, breaking through the limitations of traditional temperature scaling methods. It can perform fine-grained calibration adjustments for different samples and prediction scenarios, fully capture the multidimensional features of prediction uncertainty, and achieve accurate calibration of different predictions.

[0137] Through experimental verification on multiple standard datasets such as CIFAR10, CIFAR100 and ImageNet, as well as various network architectures, the embodiments of the present invention perform excellently in evaluation indicators such as expected calibration error (ECE), KS error and negative log-likelihood (NLL), with the average calibration error reduced by about 30%. It is significantly better than existing methods in model confidence calibration, especially on large datasets and complex models.

[0138] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0139] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0140] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0141] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0142] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0143] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0144] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0145] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0146] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A post-calibration method for uncertainty estimation, characterized in that The following steps are involved: Obtain the first output vector of the target basic classification model; Performing a multidimensional feature extraction operation on the first output vector to obtain a multidimensional feature vector; Constructing a temperature function of a multi-feature temperature scaling algorithm based on the multi-dimensional feature vector; Training an initial characterization calibrator according to the temperature function to obtain a target characterization calibrator; The target characterization calibrator is used to perform post-calibration on the second output vector of the target basic classification model to obtain an uncertainty estimation result.

2. A post-calibration method for uncertainty estimation according to claim 1, characterized in that: Before obtaining the first output vector of the target basic classification model, the following steps are also included: Training the initial basic classification model to obtain the target basic classification model; A freezing operation is performed on the model parameters of the target basic classification model.

3. The post-calibration method for uncertainty estimation according to claim 1, characterized in that: The step of performing a multidimensional feature extraction operation on the first output vector to obtain a multidimensional feature vector includes the following steps: Obtaining the maximum value, minimum value, and average value of all elements in the first output vector; Obtaining a difference between the maximum value and the minimum value; Obtaining the degree of dispersion of each element in the first output vector to obtain a standard deviation; Obtaining a distribution uncertainty indicator of the first output vector; Obtaining the multidimensional feature vector according to the maximum value, the minimum value, the average value, the difference, the standard deviation, and the distribution uncertainty index; The multidimensional feature vector includes an entropy value, a Gini coefficient, and a maximum probability value.

4. The post-calibration method for uncertainty estimation according to claim 1, characterized in that: The training of the initial characterization calibrator according to the temperature function to obtain the target characterization calibrator comprises the following steps: Inputting the multidimensional feature vector into the temperature function to obtain a temperature factor; Performing calibration prediction on the first output vector according to the temperature factor to obtain a prediction result; Obtaining a negative log-likelihood loss based on the prediction result; The initial featurization calibrator is trained according to the negative log-likelihood loss to obtain the target featurization calibrator.

5. A post-calibration method for uncertainty estimation according to claim 4, characterized in that: The formula used to input the multidimensional feature vector into the temperature function to obtain the temperature factor includes: Among them, T(z i ) represents the temperature factor; represents the temperature function; represents a multidimensional feature vector; z i Represents the first output vector.

6. A post-calibration method for uncertainty estimation according to claim 4, characterized in that: The first output vector is calibrated and predicted according to the temperature factor to obtain a prediction result, and the formula used includes: in, Indicates the prediction result; z i represents the first output vector; T(z i ) represents the temperature factor.

7. The post-calibration method for uncertainty estimation according to claim 4, characterized in that: According to the prediction result, the negative log-likelihood loss is obtained, and the formula used includes: Among them, L cal represents the negative log-likelihood loss; B cal represents a sampled data set; NLL(·) represents the negative log-likelihood function; softmax(·) represents the softmax function; Represents the prediction result; y i Represents the true value label; i represents the sample data set B cal The i-th data in ; λ represents the regularization coefficient; represents the initial characterized calibrator parameters.

8. A post-calibration system for uncertainty estimation, characterized in that include: A first module is used to obtain a first output vector of a target basic classification model; A second module is configured to perform a multidimensional feature extraction operation on the first output vector to obtain a multidimensional feature vector; A third module is used to construct a temperature function of a multi-feature temperature scaling algorithm based on the multi-dimensional feature vector; A fourth module is configured to train the initial characterization calibrator according to the temperature function to obtain a target characterization calibrator; The fifth module is used to perform post-calibration on the second output vector of the target basic classification model through the target characterization calibrator to obtain an uncertainty estimation result.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images