Posterior Calibration Method, Device, Computer Equipment and Medium of Classification Model

Abstract

This application is applicable to the field of artificial intelligence technology, and particularly relates to a posterior calibration method, device, computer device and medium for a classification model. This method inputs the obtained validation training set into N pre-trained classification models respectively, obtains the prediction vector and label vector corresponding to each data in the validation training set under each classification model, constructs a conversion function corresponding to one classification model according to each prediction vector and the corresponding label vector, determines N conversion functions, performs weighted summation on all conversion functions, determines that the weighted summation function is the updated loss function of the classification model to be calibrated, and uses the pre-training set to retrain the classification model to be calibrated after the updated loss function until the updated loss function converges, and then obtains the posterior calibrated classification model. By integrating different pre-trained classification models to construct the updated loss function, the effectiveness and generalization ability of the updated loss function are improved, and the accuracy of the calibrated classification model is relatively high.

Description

Technical Field

[0001] This application is applicable to the field of artificial intelligence technology, and particularly relates to a posterior calibration method, device, computer device, and medium for a classification model. Background Art

[0002] At present, the rapid development of deep learning has greatly improved the accuracy of neural networks. Since the inception of the deep residual network, the performance of neural networks has been significantly enhanced. Currently, neural networks have been widely applied to various decision-making systems, such as image classification, object detection, etc. However, in practical applications, decision-making systems should not only give correct decisions but also indicate when they may be incorrect to provide guidance for users. For classification, a large number of experimental results show that classification models of modern neural networks often give overly confident predictions. Therefore, the confidence scores of classification models cannot represent the true classification probabilities and need to be calibrated before use. Existing calibration methods mainly calibrate the output of classification models. Classic methods include histogram binning method, Platt scaling method, Bayesian binning method, etc. However, these methods have problems such as decreased accuracy, large amounts of data required during the training process, and insufficient expressiveness in practical applications, which affect the calibration results. Therefore, how to improve the posterior calibration process to enhance the training accuracy and robustness of classification models has become an urgent problem to be solved. Summary of the Invention

[0003] In view of this, embodiments of this application provide a posterior calibration method, device, computer device, and medium for a classification model to solve the problem of how to improve the posterior calibration process to enhance the training accuracy and robustness of classification models.

[0004] In a first aspect, embodiments of this application provide a posterior calibration method for a classification model. The posterior calibration method includes:

[0005] Input the obtained validation training set into N pre-trained classification models respectively to obtain a prediction vector and a label vector corresponding to each data in the validation training set under each classification model. The N pre-trained classification models are obtained by training the classification model to be calibrated using a pre-training set and N test sets, and N is an integer greater than zero;

[0006] For any one of the classification models, construct a conversion function corresponding to one classification model according to each prediction vector and the corresponding label vector, and determine N conversion functions;

[0007] Perform weighted summation on all the conversion functions to determine that the function after weighted summation is the updated loss function of the classification model to be calibrated;

[0008] Retrain the classification model to be calibrated after updating the loss function using the pre-training set until the updated loss function converges, and then obtain a posteriori calibrated classification model.

[0009] In one embodiment, for any classification model, constructing a corresponding conversion function according to each prediction vector and the corresponding label vector includes:

[0010] For any classification model, use a piecewise function constructed based on isotonic regression to transform each prediction vector to obtain a corresponding transformation result;

[0011] Determine the conversion function corresponding to a classification model according to the mean square loss between each conversion result and the corresponding label vector.

[0012] In one embodiment, determining the conversion function corresponding to a classification model according to the mean square loss between each conversion result and the corresponding label vector includes:

[0013] Calculate the mean square loss between each conversion result and the corresponding label vector, perform a minimization transformation on the mean square loss, and determine the transformed function as the conversion function corresponding to a classification model.

[0014] In one embodiment, if the piecewise function is a piecewise constant, and the piecewise constant includes M piecewise interval numbers, and θ j is the constant value of the j-th interval, then calculating the mean square loss between each conversion result and the corresponding label vector and performing a minimization transformation on the mean square loss includes:

[0015] Calculate the mean square loss between each conversion result and the corresponding label vector, and the mean square loss is: where, p i represents the prediction vector of the i-th data, f(p i ) represents the conversion result of the prediction vector of the i-th data, y i represents the label vector of the i-th data, and n is the number of all data;

[0016] Perform a minimization transformation on the mean square loss, and the obtained transformed function is:

[0017]

[0018] where, a 1 ...a M+1 represents the interval boundary, a j represents the lower boundary of the j-th interval, and a j+1 represents the upper boundary of the j-th interval.

[0019] In one embodiment, after using a piecewise function constructed based on isotonic regression to transform each prediction vector and obtaining the corresponding transformation result, the following steps are further included:

[0020] Taking the product of a preset positive constant and each prediction vector as the correction term for the corresponding prediction vector;

[0021] Using the correction term of each prediction vector to correct the transformation result of the corresponding prediction vector to obtain the corrected transformation result;

[0022] Determining the transformation function corresponding to a classification model according to the mean square loss between each transformation result and the corresponding label vector includes:

[0023] Determining the transformation function corresponding to a classification model according to the mean square loss between each corrected transformation result and the corresponding label vector.

[0024] In one embodiment, if the weights in the weighted summation of all transformation functions are obtained by training, the training process of the weights includes:

[0025] Using the initial N weights to perform weighted summation on the N transformation functions to obtain a weighted summation function as the loss function of the to-be-calibrated classification model;

[0026] Using a preset weight training set to train the to-be-calibrated classification model and counting the number of iterations of the training of the to-be-calibrated classification model when the loss function converges;

[0027] Adjusting the initial N weights, returning to execute the steps of weighted summation and training until the counted number of iterations is the smallest, and determining the N weights corresponding to the smallest number of iterations as the trained weights.

[0028] In a second aspect, an embodiment of the present application provides a posterior calibration device for a classification model, and the posterior calibration device includes:

[0029] A vector determination module, configured to input the obtained verification training set into N pre-trained classification models respectively, to obtain a prediction vector and a label vector corresponding to each data in the verification training set under each classification model, where the N pre-trained classification models are obtained by training a to-be-calibrated classification model using a pre-training set and N test sets, and N is an integer greater than zero;

[0030] A transformation function determination module, configured to construct a transformation function corresponding to a classification model according to each prediction vector and the corresponding label vector, and determine N transformation functions;

[0031] A loss function determination module, configured to perform weighted summation on all transformation functions and determine the weighted summation function as the updated loss function of the to-be-calibrated classification model;

[0032] A posteriori calibration module, configured to retrain the classification model to be calibrated after updating the loss function by using the pre-training set until the updated loss function converges, so as to obtain a posteriori calibrated classification model.

[0033] In one embodiment, the transformation function determination module includes:

[0034] A transformation unit, configured to use a piecewise function constructed based on isotonic regression to transform each prediction vector for any classification model, so as to obtain a corresponding transformation result;

[0035] A function determination unit, configured to determine a transformation function corresponding to one classification model according to the mean square loss between each transformation result and the corresponding label vector.

[0036] In one embodiment, the function determination unit includes:

[0037] A first determination subunit, configured to calculate the mean square loss between each transformation result and the corresponding label vector, perform a minimization transformation on the mean square loss, and determine the transformed function as the transformation function corresponding to one classification model.

[0038] In one embodiment, if the piecewise function is a piecewise constant, the piecewise constant includes M piecewise interval numbers, and θ j is the constant value of the j-th interval, then the first determination subunit is specifically configured to:

[0039] Calculate the mean square loss between each transformation result and the corresponding label vector, and the mean square loss is: where p i represents the prediction vector of the i-th data, f(p i ) represents the transformation result of the prediction vector of the i-th data, y i represents the label vector of the i-th data, and n is the number of all data;

[0040] Perform a minimization transformation on the mean square loss, and the obtained transformed function is:

[0041]

[0042] where a 1 ...a M+1 represents the interval boundary, a j represents the lower boundary of the j-th interval, and a j+1 represents the upper boundary of the j-th interval.

[0043] In one embodiment, the a posteriori calibration device further includes:

[0044] The correction term determination module is used to convert each prediction vector using a piecewise function constructed based on isotonic regression. After obtaining the corresponding conversion result, the product of a preset normal constant and each prediction vector is used as the correction term for the corresponding prediction vector.

[0045] The correction module is used to correct the conversion result of the corresponding prediction vector using the correction term of each prediction vector to obtain the corrected conversion result.

[0046] The function determination unit includes:

[0047] The second determination subunit is used to determine the conversion function corresponding to a classification model according to the mean square loss between each corrected conversion result and the corresponding label vector.

[0048] In an embodiment, if the weights in the weighted summation of all conversion functions are obtained through training, the training process of the weights includes:

[0049] Use the initial N weights to perform weighted summation on the N conversion functions to obtain a weighted summation function, which is used as the loss function of the to-be-calibrated classification model.

[0050] Use a preset weight training set to train the to-be-calibrated classification model, and count the number of iterations of the training of the to-be-calibrated classification model when the loss function converges.

[0051] Adjust the initial N weights, and return to perform the steps of weighted summation and training until the counted number of iterations is the smallest, and determine the N weights corresponding to the smallest number of iterations as the trained weights.

[0052] In a third aspect, an embodiment of the present application provides a computer device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the posterior calibration method described in the first aspect is implemented.

[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the posterior calibration method described in the first aspect is implemented.

[0054] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: The present application inputs the obtained verification training set into N pre-trained classification models respectively, and obtains the prediction vector and label vector corresponding to each data in the verification training set under each classification model. For any one of the classification models, according to each prediction vector and the corresponding label vector, a conversion function corresponding to one classification model is constructed, N conversion functions are determined, and all the conversion functions are weighted and summed to determine that the weighted sum function is the updated loss function of the classification model to be calibrated. The classification model to be calibrated after the updated loss function is retrained using the pre-training set until the updated loss function converges, and then the posterior calibrated classification model is obtained. By adopting the conversion functions obtained by integrating different pre-trained classification models to construct the updated loss function, the effectiveness and generalization ability of the updated loss function are improved. Therefore, there will be no situation of accuracy decline during long-term use, and it has a certain fault tolerance ability, making the robustness relatively high. The accuracy of the classification model obtained by posterior calibration using this method is relatively high. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0056] Figure 1 FIG. 9 is a schematic diagram of an application environment of a method for posterior calibration of a classification model provided in Embodiment 1 of the present application;

[0057] Figure 2 FIG. 13 is a schematic flowchart of a method for posterior calibration of a classification model provided in Embodiment 2 of the present application;

[0058] Figure 3 FIG. 17 is a schematic flowchart of a method for posterior calibration of a classification model provided in Embodiment 3 of the present application;

[0059] Figure 4 FIG. 21 is a schematic structural diagram of a device for posterior calibration of a classification model provided in Embodiment 4 of the present application;

[0060] Figure 5 FIG. 25 is a schematic structural diagram of a computer device provided in Embodiment 5 of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obscuring the description of the present application.

[0062] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0063] It should also be understood that the term "and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0064] As used in the specification of the present application and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" depending on the context.

[0065] In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0066] The reference to "one embodiment" or "some embodiments" etc. described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0068] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] It should be understood that the magnitudes of the sequence numbers of the steps in the following embodiments do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0070] To illustrate the technical solution of the present application, specific embodiments are used for illustration below.

[0071] A posterior calibration method for a classification model provided in the first embodiment of the present application can be applied in an application environment such as Figure 1 . Among them, the client communicates with the server. The client includes, but is not limited to, computer devices such as palm computers, desktop computers, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, cloud computer devices, and personal digital assistants (PDAs). The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0072] See Figure 2 , which is a schematic flowchart of a posterior calibration method for a classification model provided in the second embodiment of the present application. The above posterior calibration method is applied to the server in Figure 1 . The computer device corresponding to the server is connected to a corresponding database to obtain the corresponding data set in the database. The data set can be divided into a data set for verification, a training set for training, and a test set for use in different stages. The above computer device can also be connected to a corresponding client. The client can send the constructed classification model to be calibrated, etc. to the server and trigger corresponding tasks to enable the server to implement the posterior calibration method of the classification model. As shown in Figure 2 , the posterior calibration method of the classification model may include the following steps:

[0073] Step S201: Input the obtained verification training set into N pre-trained classification models respectively, and obtain the prediction vector and label vector corresponding to each data in the verification training set under each classification model.

[0074] In this application, the N pre-trained classification models are obtained by training the classification model to be calibrated using a pre-training set and N test sets, where N is an integer greater than zero.

[0075] Before posterior calibration starts, it is necessary to pre-train the classification model to be calibrated. Among them, the classification model to be calibrated is pre-trained using a pre-training set and a test set obtained from the corresponding database to obtain a pre-trained classification model. Then, the classification model to be calibrated is pre-trained using the same pre-training set and another set of test sets to obtain another pre-trained classification model, and so on, to obtain N pre-trained classification models according to actual needs.

[0076] For example, when N = 3, it means 3 trained classification models, and thus 3 different sets of tests are necessarily required. Specifically, if the classification model is a ResNet classification model, in this application, the classic CIFAR10 dataset can be used for posterior calibration. First, randomly select 10,000 pictures from the 60,000 pictures included in the CIFAR10 dataset as the verification training set, and the remaining 50,000 pictures are randomly sampled four times. In the first three samplings, 10,000 pictures are sampled each time as the test set, and in the last sampling, 40,000 pictures are sampled as the pre-training set. Then, the ResNet50 model is pre-trained using the pre-training set and 3 sets of test sets respectively to obtain 3 pre-trained classification models.

[0077] The pre-trained classification model already basically has the ability to predict data. Therefore, inputting the data in the verification training set into the pre-trained classification model can output the corresponding prediction vector (prediction result), and the label vector is the annotation of the corresponding data. For the training set, the label vector is a known quantity. Among them, the data type of the data in the training set can include but is not limited to the pictures in the above example.

[0078] Step S202: For any one of the classification models, construct a conversion function corresponding to one classification model according to each prediction vector and the corresponding label vector, and determine N conversion functions.

[0079] In this application, the conversion function is a function constructed based on different degrees such as the similarity and difference degree between the prediction vector and the label vector. The construction method can be set according to requirements as long as it can express the different degrees between the prediction vector and the label vector. For example, loss calculation methods such as mean square error loss and cross-entropy loss are used to calculate the different degrees between the prediction vector and the label vector.

[0080] The conversion function is an expression in the form of a function and is essentially an equation. This conversion function can be used as the loss function in a classification model to support the training of the classification model.

[0081] In step S202, for each classification model, there are prediction vectors and label vectors for each data in the validation training set. Therefore, for each classification model, a corresponding conversion function will ultimately be constructed, that is, the number of final conversion functions is N.

[0082] Optionally, for any classification model, constructing the corresponding conversion function according to each prediction vector and the corresponding label vector includes:

[0083] For any classification model, use a piecewise function constructed based on isotonic regression to transform each prediction vector to obtain the corresponding transformation result;

[0084] Determine the conversion function for a corresponding classification model according to the mean squared loss between each transformation result and the corresponding label vector.

[0085] Among them, isotonic regression, also known as monotonic regression, belongs to regression analysis. It is a regression model that performs non-parametric estimation on given data within a monotonic function space. Isotonic regression conceptually seeks a set of non-decreasing piecewise continuous linear functions, that is, piecewise functions, to make its estimated value as close as possible to the actual value.

[0086] The applications of isotonic regression include multidimensional scaling, calibration of classifiers under skewed data, and other situations where the dependent variable is restricted to monotonic changes. Isotonic regression can calibrate classifiers with sufficient sample support.

[0087] For the prediction vectors and label vectors obtained by the classification model, find the corresponding piecewise function to minimize the sum of squared differences between the two. Among them, use the piecewise function to transform the prediction vector to obtain the corresponding transformation result.

[0088] The construction of the piecewise function can be set according to requirements. This piecewise function can be a constant or a non-constant. A constant means that the transformation result is a constant value, and a non-constant means that the transformation result is an equation.

[0089] The mean squared loss can refer to the loss function obtained by calculating the loss based on the mean squared error. For each set of data of a classification model, a loss function can be calculated, and based on this loss function, the conversion function corresponding to this classification model can be obtained. Among them, a set of data includes a transformation result and the corresponding label vector.

[0090] Optionally, determining the conversion function corresponding to a classification model according to the mean square loss between each conversion result and the corresponding label vector includes:

[0091] Calculating the mean square loss between each conversion result and the corresponding label vector, performing a minimization transformation on the mean square loss, and determining the transformed function as the conversion function corresponding to a classification model.

[0092] Among them, for the mean square loss obtained in the above steps, that is, the loss function, a minimization transformation is performed to obtain the conversion function. After obtaining the loss function, a minimization factor is added to the loss function to construct the transformed function.

[0093] Optionally, if the piecewise function is piecewise constant, and the piecewise constant includes M piecewise interval numbers, θ j being the constant value in the j-th interval, then calculating the mean square loss between each conversion result and the corresponding label vector, and the minimization transformation of the mean square loss includes:

[0094] Calculating the mean square loss between each conversion result and the corresponding label vector, and the mean square loss is: where p i represents the prediction vector of the i-th data, f(p i ) represents the conversion result of the prediction vector of the i-th data, y i represents the label vector of the i-th data, and n is the number of all data;

[0095] Performing a minimization transformation on the mean square loss, and the obtained transformed function is:

[0096]

[0097] where a 1 ...a M+1 represents the interval boundary, a j represents the lower boundary of the j-th interval, a j+1 represents the upper boundary of the j-th interval.

[0098] Among them, taking the piecewise function as a constant as an example, the piecewise function includes M piecewise interval numbers, specifically θ 1 , θ 2 ,..., θ j ,..., θ M , θ j being the constant value in the j-th interval, if a 1 , a 2 ,..., a j ,..., a M+1 represents the interval boundary, then a 1 is the lower boundary of θ 1 and a2 is θ 1 the upper boundary, and so on.

[0099] After minimizing the conversion, the obtained converted function is the conversion function, which can be used as the loss function of the classification model to guide the training of the classification model.

[0100] Optionally, after using a piecewise function constructed based on isotonic regression to convert each prediction vector and obtaining the corresponding conversion result, it further includes:

[0101] Taking the product of a preset positive constant and each prediction vector as the correction term for the corresponding prediction vector;

[0102] Using the correction term of each prediction vector to correct the conversion result of the corresponding prediction vector to obtain the corrected conversion result;

[0103] Determining the conversion function corresponding to a classification model according to the mean square loss between each conversion result and the corresponding label vector includes:

[0104] Determining the conversion function corresponding to a classification model according to the mean square loss between each corrected conversion result and the corresponding label vector.

[0105] Among them, the function learned through the above steps is only isotonic. To ensure strict isotonicity, the above conversion result needs to be corrected, that is, g(p i ) = f(p i ) + εp i , where g(p i ) is the corrected conversion result, ε is a very small positive number. Through this correction, it can be ensured that when p i < p j , g(p i ) < g(p j ).

[0106] Step S203: Perform weighted summation on all conversion functions to determine that the weighted-summed function is the updated loss function of the classification model to be calibrated.

[0107] In this application, all conversion functions are weighted and summed based on an ensemble method to obtain an ensemble result, that is, an ensemble function. This ensemble function has better effectiveness and generalization ability, thereby improving the accuracy of posterior calibration of the classification model.

[0108] In this step, it is necessary to obtain the preset weights for each conversion function, and then perform weighted summation. Among them, the preset weights can be set according to manual experience. Of course, the preset weights can also be obtained through training.

[0109] If the pre-set weights are set according to manual experience, before weighted summation of all conversion functions, a corresponding set of weights needs to be selected according to the number of all conversion functions.

[0110] Certainly, in one embodiment, when N = 1, it indicates that there is only one conversion function. At this time, other conversion functions can be regarded as zero and weighted summation is performed with this one conversion function. Further, when N = 1, it indicates that there is only one conversion function, and the corresponding weight is automatically adjusted to 1, and the weight value is 1, that is, this one conversion function is directly used as the updated loss function of the classification model to be calibrated.

[0111] In actual use, due to the very wide range of data distributions targeted, using only the data of one test set for posterior calibration, the expression ability of the obtained calibration function is still limited. By selecting different test sets for parameter optimization, different conversion functions are obtained, and then the results of multiple conversion functions are weighted and integrated to obtain the true posterior distribution.

[0112] Step S204: Retrain the classification model to be calibrated with the updated loss function using the pre-training set until the updated loss function converges, and then obtain the classification model with good posterior calibration.

[0113] In this application, the loss function of the classification model to be calibrated is modified to the updated loss function, and the training of the classification model to be calibrated is started.

[0114] The pre-training set in the above step S201 is used in the training process to ensure that the updated loss function can converge as soon as possible, thereby improving the training efficiency.

[0115] The proposed classification model has greatly improved the prediction performance of the neural network, especially the classification model with a residual structure. However, as the depth and width of the neural network increase, the confidence of the model prediction decreases. Restricted by various conditions, only a small part of the data can be provided during network training, and the model is also modeled according to the distribution of this part of the data. However, due to the diverse data forms, when the data distribution shifts, the model often gives overly confident but incorrect predictions. Using the above posterior calibration method can improve the overconfidence of modern neural networks and give reasonable confidence scores to improve the user's trust in the classification model.

[0116] In the embodiments of the present application, the obtained verification training set is respectively input into N pre-trained classification models, and the prediction vectors and label vectors corresponding to each data in the verification training set under each classification model are obtained. For any one classification model, according to each prediction vector and the corresponding label vector, a conversion function corresponding to one classification model is constructed, N conversion functions are determined, and all the conversion functions are weighted and summed to determine that the weighted sum function is the updated loss function of the classification model to be calibrated. The classification model to be calibrated after updating the loss function is retrained using the pre-training set until the updated loss function converges, and then the posterior calibrated classification model is obtained. By adopting the conversion functions obtained by integrating different pre-trained classification models to construct the updated loss function, the effectiveness and generalization ability of the updated loss function are improved. Therefore, there will be no situation of accuracy decline during long-term use, and it has a certain fault tolerance ability, making the robustness relatively high. The accuracy of the classification model obtained by posterior calibration using this method is relatively high.

[0117] See Figure 3 , which is a schematic flowchart of a method for posterior calibration of a classification model provided in Embodiment 3 of the present application. As Figure 3 shown, specifically, it is the training process of the parts that need to be trained, such as the network, model, and weights, under a certain limited condition for the posterior calibration method.

[0118] In the present application, if the weights in the weighted sum of all conversion functions are obtained through training, the training of the weights and the training of the classification model to be calibrated can be jointly trained during the process of posterior calibration, or the weights can be trained separately.

[0119] If the separate training method is adopted, the training process of the weights includes:

[0120] Step S301: Use the initial N weights to perform weighted summation on the N conversion functions to obtain a weighted sum function, which is used as the loss function of the classification model to be calibrated.

[0121] In the present application, after obtaining the N conversion functions, the training of the weights is started. The training of the weights can be carried out simultaneously with the steps in Embodiment 2 above and needs to be completed before step S203. At this time, step S203 above can wait for the training result of the weights. Of course, in one implementation manner, the training of the weights can be started in advance, and then the posterior calibration method is started. The above initial N weights can be randomly set to improve the randomness of training and can improve the training efficiency to a certain extent.

[0122] Step S302: Use the preset weight training set to train the classification model to be calibrated, and count the number of iterations of the training of the classification model to be calibrated when the loss function converges.

[0123] In this application, the weight training set is not substantially different from the training set in the second embodiment above. Both use the same type of data set, and the amount of data in the weight training set can be adjusted to an appropriate size to ensure the efficiency and accuracy of training.

[0124] When using the weight training set to train the classification model to be calibrated, if the loss function does not converge after a long time or the number of iterations exceeds the threshold, it indicates that the above initial weights cannot be used, and corresponding weights need to be randomly generated again, that is, step S301 is executed again.

[0125] If the loss function converges, the number of training iterations needs to be counted for subsequent judgment.

[0126] Step S303: Adjust the initial N weights, and return to execute the steps of weighted summation and training until the counted number of iterations is the smallest. Determine the N weights corresponding to the smallest number of iterations as the trained weights.

[0127] In this application, after one training is completed, the weights are adjusted and the process returns to execute step S301 until the weights corresponding to the smallest counted number of iterations are obtained.

[0128] The way to adjust the weights can adopt the gradient descent method to ensure that the training can quickly find the weights with the smallest number of iterations.

[0129] The counted smallest number of iterations can refer to the smallest number among all iterations within a certain time or a certain number of iterations. The weights corresponding to this smallest number are the trained weights. Using the trained weights to execute the posterior calibration method can obtain a relatively accurate classification model.

[0130] Before performing step S203 of posterior calibration in the embodiments of the present application, the weights are first trained to obtain appropriate weighted summation weights, avoiding the low accuracy caused by manual setting according to experience. Then, the obtained validation training set is input into the pre-trained N classification models respectively, and the prediction vector and label vector corresponding to each data in the validation training set under each classification model are obtained. For any one of the classification models, according to each prediction vector and the corresponding label vector, a transformation function corresponding to one classification model is constructed, N transformation functions are determined, and all the transformation functions are weighted and summed to determine that the weighted and summed function is the updated loss function of the classification model to be calibrated. The classification model to be calibrated after the updated loss function is retrained using the pre-training set until the updated loss function converges, and then the posterior-calibrated classification model is obtained. By adopting the transformation functions obtained by integrating different pre-trained classification models and using the trained weights to construct the updated loss function, the effectiveness and generalization ability of the updated loss function are improved. Therefore, there will be no situation of accuracy decline during long-term use, and it has a certain fault tolerance ability, making the robustness relatively high. The accuracy of the classification model obtained by posterior calibration using this method is relatively high.

[0131] A posterior calibration method for the classification model corresponding to the above embodiments Figure 4 shows a structural block diagram of a posterior calibration device for a classification model provided in Embodiment 4 of the present application. The above posterior calibration device is applied to Figure 1 the server in. The computer device corresponding to the server is connected to the corresponding database to obtain the corresponding data set in the database. The data set can be divided into a data set for verification use, a training set for training use, and a test set to realize the use in different stages. The above computer device can also be connected to the corresponding client, and the client can send the classification model to be calibrated constructed, etc. to the server and trigger the corresponding task to enable the server to implement the posterior calibration method of the classification model. For the sake of convenience of description, only the parts related to the embodiments of the present application are shown.

[0132] See Figure 4 , the posterior calibration device includes:

[0133] A vector determination module 41, configured to input the obtained validation training set into the pre-trained N classification models respectively, and obtain the prediction vector and label vector corresponding to each data in the validation training set under each classification model. The pre-trained N classification models are obtained by training the classification model to be calibrated using the pre-training set and N test sets, and N is an integer greater than zero;

[0134] A transformation function determination module 42, configured to, for any one of the classification models, construct a transformation function corresponding to one classification model according to each prediction vector and the corresponding label vector, and determine N transformation functions;

[0135] A loss function determination module 43, configured to perform a weighted sum on all transformation functions, and determine the function after the weighted sum as the updated loss function of the classification model to be calibrated;

[0136] A posterior calibration module 44, configured to retrain the classification model to be calibrated after updating the loss function using a pre-training set, and obtain a posterior calibrated classification model after the updated loss function converges.

[0137] Optionally, the above transformation function determination module 42 includes:

[0138] A transformation unit, configured to, for any classification model, use a piecewise function constructed based on isotonic regression to transform each prediction vector to obtain a corresponding transformation result;

[0139] A function determination unit, configured to determine the transformation function corresponding to one classification model according to the mean square loss between each transformation result and the corresponding label vector.

[0140] Optionally, the above function determination unit includes:

[0141] A first determination subunit, configured to calculate the mean square loss between each transformation result and the corresponding label vector, perform a minimization transformation on the mean square loss, and determine the transformed function as the transformation function corresponding to one classification model.

[0142] Optionally, if the piecewise function is a piecewise constant, the piecewise constant includes M piecewise interval numbers, and θ j is the constant value of the j-th interval, then the above first determination subunit is specifically configured to:

[0143] Calculate the mean square loss between each transformation result and the corresponding label vector, and the mean square loss is: where, p i represents the prediction vector of the i-th data, f(p i ) represents the transformation result of the prediction vector of the i-th data, y i represents the label vector of the i-th data, and n is the number of all data;

[0144] Perform a minimization transformation on the mean square loss, and the obtained transformed function is:

[0145]

[0146] where, a 1 ...a M+1 represents the interval boundary, a j represents the lower boundary of the j-th interval, and a j+1 represents the upper boundary of the j-th interval.

[0147] Optionally, the above posterior calibration device further includes:

[0148] A correction term determination module, which is used to use a piecewise function constructed based on isotonic regression to convert each prediction vector. After obtaining the corresponding conversion result, the product of a preset normal constant and each prediction vector is used as the correction term of the corresponding prediction vector;

[0149] A correction module, which is used to use the correction term of each prediction vector to correct the conversion result of the corresponding prediction vector to obtain a corrected conversion result;

[0150] The above function determination unit includes:

[0151] A second determination subunit, which is used to determine the conversion function corresponding to a classification model according to the mean square loss between each corrected conversion result and the corresponding label vector.

[0152] Optionally, if the weights in the weighted sum of all conversion functions are obtained by training, the training process of the weights includes:

[0153] Using the initial N weights to perform a weighted sum of the N conversion functions to obtain a weighted sum function as the loss function of the classification model to be calibrated;

[0154] Using a preset weight training set to train the classification model to be calibrated and counting the number of iterations of the training of the classification model to be calibrated when the loss function converges;

[0155] Adjust the initial N weights, and return to perform the steps of weighted sum and training until the counted number of iterations is the smallest, and determine the N weights corresponding to the smallest number of iterations as the trained weights.

[0156] It should be noted that the information interaction, execution process, etc. between the above modules, due to being based on the same concept as the method embodiment of the present application, for the specific functions and technical effects brought, reference can be specifically made to the method embodiment part, and details are not described herein again.

[0157] Figure 5 This is a schematic structural diagram of a computer device provided in Embodiment 5 of the present application. As Figure 5 shown, the computer device in this embodiment includes: at least one processor ( Figure 5 only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, it implements the steps in any of the above method embodiments of the posterior calibration method of the classification model.

[0158] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that Figure 5The above are merely examples of computer devices and do not constitute limitations thereto. A computer device may include more or fewer components than shown in the figures, or combine certain components, or have different components. For example, it may also include a network interface, a display screen, an input device, etc.

[0159] The so-called processor may be a CPU, or it may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0160] The memory includes a readable storage medium, an internal memory, etc. Among them, the internal memory may be the memory of the computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be the hard disk of the computer device, or in other embodiments, it may also be an external storage device of the computer device. For example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the memory may also include both the internal storage unit of the computer device and the external storage device. The memory is used to store the operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. The memory may also be used to temporarily store data that has been output or will be output.

[0161] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above device can refer to the corresponding process in the foregoing method embodiment and will not be elaborated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0162] To implement all or part of the processes in the above method embodiments of this application, it can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can be made to execute the steps in the above method embodiments.

[0163] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0164] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0165] In the embodiments provided in this application, it should be understood that the disclosed device / computer equipment and method can be implemented in other ways. For example, the device / computer equipment embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0166] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit it; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A posterior calibration method for a classification model, characterized in that, the posterior calibration method includes: Inputting the obtained validation training set into N pre-trained classification models respectively, to obtain the prediction vector and label vector corresponding to each data in the validation training set under each classification model. The N pre-trained classification models are obtained by training the classification model to be calibrated using a pre-training set and N test sets. N is an integer greater than zero, and the data in the training set and test sets are all pictures; For any one of the classification models, use a piecewise function constructed based on isotonic regression to transform each prediction vector to obtain the corresponding transformation result; According to the mean square loss between each transformation result and the corresponding label vector, determine the transformation function corresponding to one classification model, and determine N transformation functions; The determining the transformation function corresponding to one classification model according to the mean square loss between each transformation result and the corresponding label vector includes: Calculating the mean square loss between each transformation result and the corresponding label vector, performing a minimization transformation on the mean square loss, and determining the transformed function as the transformation function corresponding to one classification model; If the piecewise function is piecewise constant and the piecewise constant includes M piecewise interval numbers, is the constant value of the j interval, then calculate the mean square loss between each conversion result and the corresponding label vector, and the minimization conversion of the mean square loss includes: Calculate the mean squared loss between each conversion result and the corresponding label vector, where the mean squared loss is: , where represents the prediction vector of the i -th data, represents the conversion result of the prediction vector of the i -th data, represents the label vector of the i -th data, n is the number of all data; The minimization transformation of the mean square loss results in a transformed function of: Among them, represents the interval boundary, represents the lower boundary of the j-th interval, represents the upper boundary of the j-th interval; Performing a weighted sum on all transformation functions, and determining the weighted sum function as the updated loss function of the classification model to be calibrated; Using the pre-training set to retrain the classification model to be calibrated after updating the loss function until the updated loss function converges, to obtain a posterior calibrated classification model.

2. The posterior calibration method according to claim 1, characterized in that, After using the piecewise function constructed based on isotonic regression to transform each prediction vector to obtain the corresponding transformation result, it further includes: Taking the product of a preset normal constant and each prediction vector as the correction term of the corresponding prediction vector; Using the correction term of each prediction vector to correct the transformation result of the corresponding prediction vector to obtain the corrected transformation result; The determining the transformation function corresponding to one classification model according to the mean square loss between each transformation result and the corresponding label vector includes: Determining the transformation function corresponding to one classification model according to the mean square loss between each corrected transformation result and the corresponding label vector.

3. The posterior calibration method according to claim 1 or 2, characterized in that, If the weights in the weighted sum of all transformation functions are obtained by training, the training process of the weights includes: Performing a weighted sum on N transformation functions using N initial weights to obtain a weighted sum function as the loss function of the classification model to be calibrated; Using a preset weight training set to train the classification model to be calibrated, and counting the number of iterations of training the classification model to be calibrated when the loss function converges; Adjusting the N initial weights, and returning to execute the steps of weighted sum and training until the counted number of iterations is the smallest, and determining the N weights corresponding to the smallest number of iterations as the trained weights.

4. A posterior calibration device for a classification model, characterized in that, the posterior calibration device includes: A vector determination module, configured to input the obtained verification training set into N pre-trained classification models respectively, and obtain a prediction vector and a label vector corresponding to each data in the verification training set under each classification model. The N pre-trained classification models are obtained by training a classification model to be calibrated using a pre-training set and N test sets. N is an integer greater than zero, and the data in the training set and the test set are all pictures. A conversion function determination module, where the conversion function determination module includes: A conversion unit, configured to use a piecewise function constructed based on isotonic regression to convert each prediction vector for any one of the classification models to obtain a corresponding conversion result. A function determination unit, configured to determine a conversion function corresponding to one classification model according to the mean square loss between each conversion result and the corresponding label vector, and determine N conversion functions. The function determination unit includes: A first determination subunit, configured to calculate the mean square loss between each conversion result and the corresponding label vector, perform a minimization conversion on the mean square loss, and determine the converted function as the conversion function corresponding to one classification model. If the piecewise function is piecewise constant and the piecewise constant includes M piecewise interval numbers, is the constant value of the j interval, then the first determination subunit is specifically configured to: Calculate the mean squared loss between each conversion result and the corresponding label vector, where the mean squared loss is: , where represents the predicted vector of the i -th data, represents the conversion result of the predicted vector of the i -th data, represents the label vector of the i -th data, n is the number of all data; Performing a minimization conversion on the mean square loss, the obtained converted function is: Among them, represents the interval boundary, represents the lower boundary of the j-th interval, represents the upper boundary of the j-th interval; A loss function determination module, configured to perform a weighted sum on all conversion functions, and determine the function after the weighted sum as the updated loss function of the classification model to be calibrated. A posterior calibration module, configured to use the pre-training set to retrain the classification model to be calibrated after updating the loss function until the updated loss function converges, and obtain a posterior calibrated classification model.

5. A computer device, characterized in that the computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the posterior calibration method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the posterior calibration method according to any one of claims 1 to 3 is implemented.

Images